feat(sdk): add local desktop with long polling

[leone-hai]

Summary

Adds a new [desktop] optional extra that lets users run a local sidecar on their machine and have H agents control it remotely via mouse, keyboard, and screenshots.

New public API:

• LocalDesktopClient / LocalDesktopClientConfig — polls AgP for incoming desktop commands and executes them locally using pyautogui
• session_id_from_environment_id(environment_id, api_key) — derives the deterministic routing UUID that ties the sidecar to an agent session

Usage:

# On the machine to control:
config = LocalDesktopClientConfig(environment_id="my-mac", api_key="...")
await LocalDesktopClient(config).run()

# To start an agent session pointing at that machine:
session_id = session_id_from_environment_id("my-mac", api_key)
client.run_session(
    agent="h/holo-desktop",
    environments=[{"kind": "desktop", "id": "h/local-desktop",
                   "backend": "local", "session_id": session_id}],
    messages="Take a screenshot",
)

Install: pip install "hai-agents[desktop]"

---

Note

High Risk
Remote agents can drive real input, filesystem, and shell on the host when the sidecar runs with a valid API key—high blast radius despite being an intentional feature.

Overview
Adds a [desktop] optional extra (pyautogui, pynput) and a new local_desktop module so a machine can run an async sidecar that long-polls AgP for commands and executes them locally.

Public API: LocalDesktopClient / LocalDesktopClientConfig, plus session_id_from_environment_id (UUID5 from environment_id + api_key) exported from hai_agents for routing agent sessions to the same machine without manual ID coordination.

The client ensures/creates an interactive trajectory, polls GET .../commands, dispatches named driver methods (mouse, keyboard, screenshots, file I/O, run_command with detach on POSIX/Windows), serializes results (e.g. PNG as base64), and posts back with retries and reconnect backoff. Tests stub pyautogui in conftest for headless CI; lockfile picks up desktop transitive deps and minor dev bumps (ruff, rich).

^{Reviewed by Cursor Bugbot for commit 080e475. Bugbot is set up for automated code reviews on this repo. Configure here.}

[leone-hai]

• feat(sdk): add local desktop with long polling #104 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 080e475. Configure here.}

[cursor]

Failed results may rerun commands

Medium Severity

_post_result logs and returns after exhausting retries without surfacing failure to _process_commands or the poll loop. The sidecar then keeps polling, so the server may redeliver the same command and local actions like clicks or run_command can run more than once.

Additional Locations (1)

• src/hai_agents/local_desktop.py#L431-L445

 

^{Reviewed by Cursor Bugbot for commit 080e475. Configure here.}

[cursor]

pynput required but unused

Low Severity

The desktop extra and _LocalDesktopDriver docs/error text require pynput, but the module only imports and uses pyautogui. Installations pull native pynput dependencies without any runtime use in this codepath.

 

^{Reviewed by Cursor Bugbot for commit 080e475. Configure here.}