Skip to Content

Desktop Agent

Allow AI agents to interact with personal desktops.

Overview

The Desktop Agent runs on user machines and enables AI agents to remotely control desktops through the isA platform.

Features

  • Remote Desktop Control - AI agents execute commands on your desktop
  • Ownership & Visibility - Control who can access (private, shared, friends, org)
  • 10 Built-in Tools - File ops, bash, clipboard, screenshots, and more
  • WebSocket Connection - Real-time bidirectional communication
  • Heartbeat Monitoring - Automatic health checks and reconnection

Installation

Via pip

cd os_services/desktop_os pip install -e .

Via Install Script

curl -fsSL https://your-domain.com/install.sh | bash

Usage

Basic

# Run with your user ID (required) isa-desktop --owner YOUR_USER_ID --server http://pool-manager:8090

With Options

# Share desktop with others isa-desktop --owner YOUR_USER_ID --visibility shared # Run in sandbox mode (restrict to home directory) isa-desktop --owner YOUR_USER_ID --sandbox # Custom name and tier isa-desktop --owner YOUR_USER_ID --name "My Workstation" --tier workstation

Command Line Options

OptionDescriptionDefault
--ownerYour user ID (required)-
--serverPool Manager URLhttp://localhost:8090
--visibilityAccess controlprivate
--sandboxRestrict file operationsfalse
--nameCustom desktop namehostname
--tierDesktop tierstandard

Visibility Levels

LevelDescription
privateOnly owner can access
sharedAnyone with link can access
friendsOwner’s friends can access
organizationOrganization members can access

Tier Levels

TierDescription
basicLimited capabilities
standardNormal capabilities
premiumEnhanced capabilities
workstationFull workstation features

Available Tools

1. read_file

Read file contents from desktop.

{ "tool": "read_file", "params": {"path": "/home/user/document.txt"} }

2. write_file

Write content to files.

{ "tool": "write_file", "params": { "path": "/home/user/output.txt", "content": "Hello, World!" } }

3. list_directory

List directory contents.

{ "tool": "list_directory", "params": {"path": "/home/user"} }

4. execute_bash

Run shell commands.

{ "tool": "execute_bash", "params": {"command": "ls -la"} }

5. get_clipboard

Get clipboard contents.

{ "tool": "get_clipboard", "params": {} }

6. set_clipboard

Set clipboard contents.

{ "tool": "set_clipboard", "params": {"content": "Text to copy"} }

7. screenshot

Capture screen.

{ "tool": "screenshot", "params": {"region": "full"} # or specific coordinates }

8. open_application

Launch applications.

{ "tool": "open_application", "params": {"app": "code"} # VS Code }

9. open_url

Open URLs in browser.

{ "tool": "open_url", "params": {"url": "https://example.com"} }

10. get_system_info

Get system information.

{ "tool": "get_system_info", "params": {} }

Architecture

User/Agent → APISIX → Thin Agent Proxy → Pool Manager → Desktop Agent

The Desktop Agent:

  1. Registers with Pool Manager on startup
  2. Connects via WebSocket for real-time commands
  3. Sends periodic heartbeats
  4. Executes commands and returns results

Security Considerations

Sandbox Mode

When --sandbox is enabled:

  • File operations restricted to home directory
  • No access to system files
  • Prevents accidental system damage

Visibility Control

  • Choose appropriate visibility level
  • private recommended for sensitive work
  • shared only for controlled environments

Integration

From AI Agent

import httpx async def execute_on_desktop(desktop_id: str, tool: str, params: dict): async with httpx.AsyncClient() as client: response = await client.post( f"http://pool-manager:8090/pools/desktop/{desktop_id}/execute", json={ "action": tool, "params": params } ) return response.json() # Example: Read a file result = await execute_on_desktop( "desktop-abc123", "read_file", {"path": "/home/user/notes.txt"} )

From MCP Tool

@mcp.tool() async def desktop_bash(desktop_id: str, command: str): """Execute bash command on desktop""" return await execute_on_desktop( desktop_id, "execute_bash", {"command": command} )

Troubleshooting

Connection Issues

  • Check Pool Manager is running
  • Verify network connectivity
  • Check firewall settings

Permission Denied

  • Ensure correct ownership
  • Check visibility settings
  • Verify sandbox mode restrictions

Next Steps