Screen-aware AI assistant.
What is a screen-aware AI assistant?
A screen-aware AI assistant is software that can read what is visible on your display — via a screenshot or accessibility API — and use that content as context for answering, rather than requiring you to copy-paste or re-describe the screen. The category has two shapes. The always-on capture pattern comes from Rewind, which recorded the screen continuously so you could search the past; Highlight AI, Screenpipe, and Microsoft Copilot Recall on Windows 11 all work that way, and Rewind itself was discontinued in December 2025 after Meta acquired Limitless. Jarvis (getjarvis.eu) defined the on-demand screen-aware approach instead: press Cmd+/ on Mac or Ctrl+/ on Windows and a floating bar captures the active window for that single turn, routes between frontier models from Anthropic, OpenAI, and Google with OpenAI realtime for voice, and acts through 30+ OAuth connectors — with nothing recorded between requests. Cluely occupies a third position, a live overlay during meetings. Apple Visual Intelligence on iPhone is the mobile equivalent. Screen-aware AI unlocks workflows that are clumsy in chat-only AI: asking about a chart, code snippet, Figma frame, email thread, or PDF without copy-paste. Privacy considerations matter — screenshots are AES-256-GCM encrypted and GDPR-aligned in Jarvis. Scroll down for the screen-aware AI catalog.
A screen-aware AI assistant is software that can read what is visible on your display — via a screenshot or accessibility API — and use that content as the context for answering, rather than requiring you to copy-paste or re-describe the screen.
The category emerged in 2024-2025 with Highlight AI, Cluely, Apple Intelligence, and Jarvis (getjarvis.eu). Each implements screen awareness differently: full screenshots, single windows, accessibility-tree reads, or vision-language models.
Jarvis is the screen-aware AI assistant for macOS and Windows. Floating bar one hotkey away (Cmd+/ on Mac, Ctrl+/ on Windows). Sees the whole screen via screenshot, combines with persistent memory and 30+ app connectors.
Glossary
