Microsoft wanted to highlight its focus on Artificial Intelligence (AI) in creating technologies that align with the daily lives of users. Although it is early to judge the results, one of its most recent developments is Copilot Vision.
This is an experimental feature that allows your Copilot assistant to “see” what the user is viewing in their browser and offer help based on that.
This advancement of Microsoft Copilot’s connected system is designed to enrich the user experience by having AI interact with web content in real time.
In this article, we’ll explore what we can expect from Copilot Vision and how it fits into Microsoft’s broader vision for the role of AI.
What is Copilot Vision?
Copilot Vision is a feature designed to make Microsoft’s Copilot AI assistant able to “see” and understand on-screen content, whether text, images or graphics, and respond accordingly.
This visual capability allows you to interpret web pages in the Microsoft Edge browser, so the user only needs to activate the function by pressing the “Vision” icon for the assistant to begin analyzing the visible content.
Instead of describing what they are seeing or asking detailed questions about the context, the user can interact directly with the information on the screen.
This function is ideal for answering questions, making recommendations or guiding the user in their decisions quickly and without interruptions.
How could Copilot Vision be used?
Imagine, for example, that someone is planning a trip and is reviewing several destination options on a web page.
Instead of having to leave the page to search for additional information, the user can activate Copilot Vision and directly request recommendations or comparisons about the destinations they are viewing.
You might also look up substitutions for ingredients in a recipe or even get tips on how to arrange furniture when decorating a new apartment. The idea is to facilitate navigation and make decisions without the need to perform additional searches.
Main features of Copilot Vision
Microsoft has included several key features to make Copilot Vision stand out in terms of usability and privacy.
- Real-time interaction: Copilot Vision can analyze text and images on a web page to answer questions or suggest next actions based on what it sees. Its purpose is to help users without interrupting their workflow.
- Support for select websites: As an experimental feature, Copilot Vision is only available on a limited list of popular sites. This allows Microsoft to limit potential risks while ensuring user safety.
- Emphasis on privacy: Copilot Vision is a completely optional feature, and the user decides when to activate and deactivate it. Each session is ephemeral, meaning that the information Copilot Vision analyzes is discarded at the end of the session and is not saved or used to train the AI model.
- No access to paid or sensitive content: The tool cannot interact with restricted or paid-protected content, such as articles under paywall. This ensures that you do not infringe copyrights or share unauthorized content.
Security and privacy in Copilot Vision
Microsoft says it has placed special emphasis on security and privacy with this feature. Copilot Vision only processes data during the active session, without storing user information or using it to improve or train its AI models.
Additionally, as part of its commitment to responsible AI, Microsoft has implemented a series of multi-stage security assessments.
These include testing with human evaluators and adversarial techniques to detect potential risks, ensuring that Copilot Vision remains within the confines of a safe and controlled environment.
Microsoft’s vision: A personalized AI companion
Beyond visual capabilities, Microsoft has defined a clear mission for its Copilot technology: create an accessible and personalized AI “companion” that allows users to feel supported in their daily lives.
In the words of Mustafa Suleyman, Executive Vice President and CEO of Microsoft AI, the goal is for Copilot to be a close presence, aligned with the interests and needs of each user and acting only with their permission.
Instead of being just another tool, Copilot is designed to evolve and adapt to the tastes and routines of each user, providing personalized help and advice that can range from managing email to organizing events or making complex decisions.
Additional features in the Microsoft ecosystem
Today, Microsoft has integrated Copilot into various platforms, such as Edge, Bing, and Windows, making it easy for users to access its assistant from different devices and environments.
In the Edge browser, Copilot can summarize web pages, answer questions and translate text instantly, allowing users to find information and resolve questions with a simple interaction.
Additionally, with the launch of Copilot on mobile, WhatsApp and the web, Microsoft is expanding its reach to create ubiquitous and accessible AI.
Microsoft’s experimental laboratory, Copilot Labs, also plays a key role in this development strategy.
This controlled environment allows users to try out new features, such as “Think Deeper,” a tool that allows Copilot to provide more detailed answers** and analyze complex questions in depth.
This makes Copilot a resource for everyday life problems that may require more detailed thought.
An AI for “human well-being”
Microsoft has stressed that the creation of Copilot is not only a response to competition in the field of AI, but a commitment to improving the technological experience in a humanized way.
With technology that “understands” and “sees” what the user needs, Microsoft’s purpose is to reduce information stress and simplify the lives of its users.
In short, Microsoft Copilot Vision represents a new approach to the way we interact with technology, integrating vision and language into a single experience that ultimately aims to assist us in every important moment of our lives.
This post is also available in: