GUI automation agents have a deployment problem: they steal your cursor. When an agent clicks through a desktop app, your mouse pointer jumps around, keyboard focus switches windows, and you lose control of your own machine. This is not a minor annoyance. It makes production GUI automation unusable for anyone who needs to keep working while agents run.
Cua is an open-source project that solves this by creating isolated macOS sessions where agents can control applications in the background. Your cursor stays put. Your keyboard keeps typing. The agent operates in parallel, invisible to your active session.
This article explains how that isolation works, what trade-offs it introduces, and whether it solves your problem.
The Session-Stealing Problem
Most GUI automation tools (Selenium, Playwright for desktop, accessibility API wrappers) assume they own the entire user session. When an agent sends a click event, macOS routes it through the window server to the frontmost application. The cursor moves. Focus changes. If you try to type an email while the agent is clicking through a form, your keystrokes land in the wrong window.
This happens because:
- Single active session: macOS maintains one primary window server context per logged-in user.
- Event routing: Input events (mouse, keyboard) go to the focused window in that context.
- Accessibility APIs: Tools like
CGEventPostinject events into the active session, not a separate sandbox.
The result is that agents and human users fight for control. You cannot run an agent in production if it hijacks the machine every time it executes a task.
How Cua Creates Background Sessions
Cua uses macOS’s ability to spawn additional login sessions that run in parallel to the active graphical session. These sessions are background sessions (no physical display attached) but fully functional: they have their own window server, input event queue, and application lifecycle.
Session Isolation Mechanics
-
Spawn a new user session: Cua launches a separate macOS login session using
launchctlor similar session management APIs. This session runs under the same user account but in a distinct window server context. -
Route automation commands: When an agent needs to click a button, Cua sends the event to the background session’s window server, not the active one. The cursor in the background session moves, but your visible cursor does not.
-
Application launch in isolation: Apps started by the agent run in the background session. They render UI, respond to events, and maintain state, but they do not appear on your screen or steal focus.
-
Screen capture for verification: Agents need visual feedback to confirm actions. Cua captures screenshots from the background session’s virtual framebuffer and feeds them to the agent’s vision model.
Key Technical Boundaries
| Component | Active Session | Background Session |
|---|---|---|
| Window server context | User’s primary display | Isolated virtual display |
| Input event queue | Physical keyboard/mouse | Programmatic event injection |
| Application focus | Visible windows | Hidden windows in background context |
| Screen capture | User’s display buffer | Virtual framebuffer for agent |
| Accessibility API scope | Active session (most APIs) | Background session (most APIs)* |
*Note: Some macOS system APIs (security prompts, certain system preferences) force themselves into the active session regardless of window server context. Most application-level accessibility APIs respect the session boundary.
The isolation is not perfect. Some macOS APIs (like certain system preferences or security prompts) force themselves into the active session. But for most applications, the boundary holds.
Example: Spawning a Background Session
# Create a new launchd session for automation
# Note: UID 501 is example-only. Real deployments require user-specific UID
# and proper plist configuration with virtual display and event permissions.
sudo launchctl bootstrap gui/501 /Library/LaunchDaemons/com.cua.background.plist
# The plist defines a session with:
# - Isolated window server context
# - Virtual display configuration (CGDisplayCreateVirtualDisplay)
# - Event injection permissions (com.apple.security.automation.apple-events)
# - Accessibility entitlements (com.apple.security.app-sandbox false)
#
# Full plist structure: https://github.com/trycua/cua/blob/main/config/background-session.plistThe session runs under the same UID but in a separate window server domain. Applications launched in this context believe they have a display and keyboard, but their events never reach your active session.
State Synchronization and Failure Modes
Running GUI automation in a background session introduces new failure modes that do not exist in foreground automation.
Timing Assumptions Break
Many GUI automation scripts assume visual feedback is instant. Click a button, wait 100ms, check if a dialog appeared. In a background session, rendering is decoupled from the agent’s perception. The app may render a dialog, but the agent’s screenshot capture might lag or miss transient UI states.
Solution: Cua uses polling with exponential backoff. Instead of fixed delays, the agent repeatedly captures screenshots and checks for expected UI changes until a timeout. Typical parameters: 100ms initial delay, 2x exponential multiplier per retry, 5-second maximum timeout.
Focus-Dependent UI Elements
Some applications behave differently when they are not the frontmost window. Dropdown menus might not render. Hover states might not trigger. Keyboard shortcuts might not fire.
Workaround: Cua forces the target application to believe it has focus within its session. This requires manipulating the window server’s focus state, which is fragile and version-dependent.
Missing Visual Feedback
Display-less sessions do not have a physical display. Some apps detect this and disable rendering optimizations, or they refuse to launch entirely. Others render to an offscreen buffer but skip animations or transitions.
Solution: Cua creates a virtual display using macOS’s offscreen rendering APIs. This tricks most apps into behaving as if a real monitor is attached.
State Leakage Between Sessions
If an application uses shared resources (files, databases, network sockets), actions in the background session can interfere with the active session. For example, if the agent opens a document in the background and the user opens the same document in the foreground, file locking conflicts arise.
Approach: Cua does not solve this at the infrastructure level. Agents must be aware of shared state and coordinate access, either through file locks, database transactions, or task queues.
Deployment Shape
Cua is designed to run on the same macOS machine where the agent executes. It is not a remote desktop protocol or a cloud sandbox. The background session runs locally, which means:
- Low latency: No network round-trip for input events or screen captures.
- Full API access: The agent can use all macOS APIs, not just what a remote protocol exposes.
- Single-machine scaling: You cannot distribute background sessions across multiple machines without additional orchestration.
For multi-agent deployments, you would run multiple macOS VMs (on bare metal or in a cloud provider that supports nested virtualization) and deploy Cua on each VM.
Security Boundaries
Background sessions do not provide security isolation. The agent runs under the same user account as the active session, so it has the same file system permissions, keychain access, and network privileges.
If an agent is compromised, it can:
- Read files in the user’s home directory.
- Access credentials stored in the macOS keychain.
- Send network requests as the user.
Strategies:
- Run Cua under a separate user account with restricted permissions.
- Use macOS’s sandbox entitlements to limit file system and network access.
- Monitor agent actions with an audit log that records every input event and screen capture.
Cua does not currently enforce any of these strategies out of the box. You must configure them manually.
Observability
Debugging GUI automation in a background session is harder than debugging foreground automation because you cannot see what the agent sees in real time.
Cua provides:
- Screen capture logs: Every screenshot the agent captures is saved with a timestamp.
- Event logs: Every input event (click, keypress, scroll) is recorded with coordinates and target application.
- Application state snapshots: Periodic dumps of the accessibility tree for the target application.
These logs are essential for post-mortem analysis when an agent fails. Without them, you have no way to reconstruct what went wrong.
Technical Verdict
Use Cua if:
- You must automate macOS desktop applications that lack APIs or CLIs, and session hijacking blocks production deployment.
- Your agents can tolerate 50-200ms rendering latency from virtual display capture and you can handle timing races with polling logic (100ms initial, 2x backoff, 5s timeout).
- You accept the operational cost of managing background sessions, virtual displays, and focus state manipulation across macOS versions.
- Your security model allows agents to run under the same user account with full file system and keychain access, or you can configure separate restricted accounts.
Avoid Cua if:
- The target application exposes a stable API, CLI, or AppleScript interface. Programmatic control is always simpler and more reliable than GUI automation.
- You need cross-platform automation. Cua’s session isolation is macOS-specific. Linux and Windows require entirely different approaches (X11 virtual displays, Windows Terminal Services).
- Your agents require sub-100ms visual feedback loops or depend on real-time animations and transitions. Background rendering adds overhead that breaks tight timing assumptions.
- You need strong security isolation between the agent and user data. Background sessions share the same privilege boundary as the active session.
Trade-off summary: Cua trades operational complexity (session management, virtual displays, focus hacks) and rendering latency (50-200ms) for the ability to run GUI automation without disrupting human users. The security boundary is weak (same user account), so treat agents as fully trusted or isolate them in separate VMs. If you can avoid GUI automation entirely, do that. If you cannot, Cua is the most practical solution for macOS production deployments today.
The project was released last weekend and the failure modes are not fully documented. Expect to spend time debugging focus issues, timing races, and application-specific quirks. But the core idea is correct: agents need their own session, not yours.
Source Links
- Primary source: Cua on GitHub
- Discussion: Hacker News thread (192 points, 43 comments)