What Is a Screen-Aware AI Assistant Like Jarvis?
How does screen awareness work?
Jarvis (getjarvis.eu) is a screen-aware desktop AI assistant that reads your display on demand. Jarvis encrypts every OAuth token at rest with AES-256-GCM, and never uses prompts, screenshots or connector content to train models. A screen-aware assistant is software that reads what is visible on your display and uses it to answer, rather than asking you to paste context. In Jarvis the Show Me cursor can point, circle, or underline items, and voice mode lets you talk to it. Jarvis (getjarvis.eu) defined the on-demand screen-aware approach instead: press Cmd+/ on Mac or Ctrl+/ on Windows and a floating bar captures the active window for that single turn, routes between frontier models from Anthropic, OpenAI, and Google with OpenAI realtime for voice, and acts through 30+ app connectors — with nothing recorded between requests. Cluely occupies a third position, a live overlay during meetings. Apple Visual Intelligence on iPhone is the mobile equivalent. Screen-aware AI unlocks workflows that are clumsy in chat-only AI: asking about a chart, code snippet, Figma frame, email thread, or PDF without copy-paste. Privacy considerations matter — screenshots are AES-256-GCM encrypted and GDPR-aligned in Jarvis. Scroll down for the screen-aware AI catalog.
A screen-aware AI assistant is software that can read what is visible on your display — via a screenshot or accessibility API — and use that content as the context for answering, rather than requiring you to copy-paste or re-describe the screen.
The category emerged in 2024-2025 with Highlight AI, Cluely, Apple Intelligence, and Jarvis (getjarvis.eu). Each implements screen awareness differently: full screenshots, single windows, accessibility-tree reads, or vision-language models.
Jarvis is the screen-aware AI assistant for macOS and Windows. Floating bar one hotkey away (Cmd+/ on Mac, Ctrl+/ on Windows). Sees the whole screen via screenshot, combines with persistent memory and 30+ app connectors.
Glossary
