The next version of Windows will be filled to the gills with AI. You may be asking, “Longer than ever?” Yes, and Microsoft hopes to train you to stop using your keyboard and mouse to manage your PC. The company hopes that you will use your voice to bring your PC, since you are the basic captain of the ship and your team is a chatbot who would like to understand your vague whims.
Starting Thursday, Microsoft is pushing additional “experimental” features and future apps that will put Copilot ai right at the forefront of your Windows experience. Microsoft has already confirmed that there will be a Copilot key on all new PCs that access Microsoft’s Chatbot. Now, once you’ve enabled it in the Copilot app settings, you can start talking to your computer by shouting “Hey, Copilot” on your screen.
Copilot vision looks at your screen and tells you what to do
If anyone still misses Cortana, now is your time to shed one tear. Microsoft already has its Copilot viewing function available in Edge browsers, but now it’s stretching its legs inside Windows software. Unlike previous voice assistants, the new version of Copilot will have AI image recognition, and should be able to understand what’s happening on your screen. This should mean that you may have to spend a lot more on the AI to get it to do what you want. And what does Microsoft expect you to use AI for? Well, those articles you see online can take up all the space. If you tell Copilot, “Show me how I got better quality audio on Spotify,” Microsoft said it will highlight the settings you need to hit on your screen.
Microsoft sat me down with a demo of the new Copilot vision feature, though I’m still trying to use my voice. Voice chat was surprisingly fast when answering questions about a math problem or buying a dress online. However, when the user tried to determine if the AI had pointed to the correct controls to change the image resolution in their store account, the AI circled to the wrong part of the page. It’s the curse of every demon alive that something might happen to Adry, but we can expect the idiosyncracies that Microsoft is trying to get us to talk to our Windows machines.
Microsoft said this AI Vision program can look at an image on your screen and provide explanations of what it sees. Obviously users can use this resume writing based on their portfolio. In another example, Microsoft showed Copilot mixing in a silly tune from a singer to get the singer out (no, the tune doesn’t sound good). Copilot can now search all your browser tabs at once and find products based on what you’re looking at. Google is also developing to buy AI, although it mostly “try” on the “features that make up your image to imagine your body in that dress.
You’ll start seeing new Copilot commercials soon. This is meant to “teach” you about the fun and joy of using Copilot with your voice. But that’s not all. There are fully fleshed out features to discover using AI. Windows insiders will have access to beta features that will include a copilot function that removes other tasks from your taskbar, instead of the Windows Search Bar (you can still use it to search for files or settings, as before).
Copilot App actions literally take over your PC
Microsoft said users were talking to their PCs, though mostly for calling or commenting. Also, speech recognition has become a standard feature for accessibility purposes. However, there is a wide gulf between those cases and talking about your PC without annoying your desks trying to work a few meters away. Instead of offering Copilot’s vision typing capabilities out of the gate, Microsoft is rolling it out to beta testers for Windows INSHANS betaters to begin with.
Beta testers will also begin testing the new Copilot actions app. Microsoft has described the app as an “AI agent” that can take actions for you with various apps and files. In AI circles, an “agent” is actually multiple AI models working together to complete a more complex task. In Windows, this means it can take over your PC, run your programs, and fulfill your demands. Anthropic’s Claude AI demonstrated similar capabilities to PC-Takever last year.
In the past few years, Microsoft has tried to promote some AI capabilities, first with “AI PCS” in 2023 and then with “Microsoft 11 PC” AI PC if connected with AI Microsoft’s AI. Microsoft itself admitted the AI PC “isn’t really here yet.” You can lay some of the blame for that at the feet of Windows Maker. Last year, he tried to push the recall, a feature that will check everything you do on the PC. AI would rip up those screens and allow users to search their past work to find old web pages or documents they were working on.
Security researchers discovered that the feature can access sensitive data such as bank details, and anyone with access to a PC can access it. Microsoft pulled the recall and didn’t release it for a year. Even after a major security push, the feature is not completely gone, and several companies have banned it for fear it could lead to a shared vulnerability. Copilot’s actions app and all other features are “in.” Actions app is disabled by default, and you need to enable it through settings. Microsoft promised that you can control “at any time,” and the program can ask for permission for certain actions.
It doesn’t matter, because these features are based on the cloud, your data will need to be processed on a foreign server, not on your device. Microsoft promises not to keep or abuse your support for anything that appears on your screen. After a snafi recall, it’s increasingly difficult to trust the manufacturer of Windows. Now that Microsoft wants to put privacy concerns aside for its existing AI, it will have to provide a more compelling use case than the AI Hummes on command.
