Hardware & Embodied AI Execution Guide

This guide explains how to connect autonomous AI agents to physical robotics, dexterous hands, and industrial actuators using the Robot Abstraction Layer (RAL), Model Context Protocol (MCP), and the isA Agent SDK 0.6.0.

1. Architecture & Design Principles

The isA Embodied AI platform bridges generative LLM reasoning and real-time physical actuation. Physical hardware requires microsecond-level timing and strict safety boundaries that differ fundamentally from cloud API workflows.

Core Principle: Local Real-Time at the Edge, Control Plane in the Cloud

• Edge / Device Side (RAL): High-frequency closed-loop control (100 Hz+), sensor feedback verification, physical limit clamping, and emergency safe-stop mechanisms remain strictly on the physical device.
• Cloud Control Plane (isagent.io): Session lease governance, durable command dispatch, audit logging, OTA firmware deployment, and telemetry persistence run on Google Cloud Platform.

Cloud-to-Edge Data Flow

┌────────────────────────────────────────────────────────┐
│            Autonomous Agent (Agent SDK 0.6.0)          │
│   Reasoning Node ──► Tool Call (execute_skill)        │
└───────────────────────────┬────────────────────────────┘
                            │ HTTPS / MCP Tool Invocation
                            ▼
┌────────────────────────────────────────────────────────┐
│             GCP Platform & Cloud Gateway               │
│   • North Gateway: wss://isagent.io/mqtt               │
│   • user-auth: Mutual TLS device token verification    │
│   • user-device: Durable command ledger & deduplication│
└───────────────────────────┬────────────────────────────┘
                            │ WSS / MQTTS (Sub-only Topic)
                            ▼
┌────────────────────────────────────────────────────────┐
│         Edge Hardware Supervisor / Reference Device    │
│   • Bounded Lease Watchdog (60s session expiry)        │
│   • Zero Automatic Physical Retries                    │
└───────────────────────────┬────────────────────────────┘
                            │ Process Call / CAN FD
                            ▼
┌────────────────────────────────────────────────────────┐
│             Robot Abstraction Layer (RAL)              │
│   • RP Hand Adapter (Travel: 0..10000, Current: 0..3000)│
│   • Feedback-Confirmed Receipts (completion_verified)  │
│   • Physical Actuators (Thumb, Index, Middle, etc.)    │
└────────────────────────────────────────────────────────┘

2. Robot Abstraction Layer (RAL) & Units

Physical devices must never suffer silent unit conversion errors (e.g. interpreting degrees as radians, or millimeters as meters). The isA Robot Abstraction Layer enforces native unit contracts per hardware family.

Supported Protocol Models & Native Units

Important: The RP Hand adapter explicitly exposes native integer travel units. Raw current feedback is neither mA nor joint torque, and is strictly quarantined from SI joint loads.

Explicit Lifecycle State Machine

Connecting to physical actuators has physical side-effects. The lifecycle requires deliberate explicit consent:

connect(acknowledge_side_effects=True)
                 │
                 ▼
         device.actuator().enable()
                 │
                 ▼
   skills.execute(home) ──► completion_verified == True
                 │
                 ▼
       Execute Move / Grasp Skills
                 │
                 ▼
        motion().safe_stop() ──► disable()
                 │
                 ▼
            disconnect()

1. Connect: Validates transport and registers lifecycle hooks.
2. Enable: Arms actuator power electronics.
3. Home: Performs mechanical calibration; all subsequent motion commands require a verified homed receipt.
4. Execute: Dispatches commands under isolated, single-owner mutual exclusion.
5. Safe Stop & Disconnect: Proves actuator zero-velocity and power cutoff before releasing transport.

3. Unified Skill MCP Tools

Agents interact with hardware through standardized Model Context Protocol (MCP) tools. This abstraction allows any foundation model to inspect capabilities and execute validated physical skills without custom driver code.

Canonical Tool Vocabulary

Feedback-Confirmed Receipts

An operation state of accepted or running does not imply physical arrival. Only a terminal receipt with completion_verified=true confirms that fresh sensor feedback matches the commanded target within tolerance.

Terminal Receipt States:

• succeeded: Actuator feedback matches target within tolerance (completion_verified=true).
• failed: Hardware fault or execution error.
• cancelled: Operator or client aborted motion; safe stop confirmed.
• timed_out: Feedback deadline elapsed; stop and disable executed.
• unsafe: Unresponsive worker or unverified safe stop; device is quarantined.

4. Agent SDK 0.6.0 Integration

Version 0.6.0 of the isA Agent SDK introduces typed hardware execution clients in both Python and TypeScript, with native Zod 4 parity and cryptographic artifact validation.

Zero Automatic Physical Retries Barrier

Safety Rule: When a network transport error or HTTP 500 occurs during physical command dispatch, the SDK never automatically retries the mutation. Replaying a physical command over an unknown edge state can cause mechanical collision or motor burnout. Retries must be explicit and operator-governed.

Python Example (isa-agent-sdk>=0.6.0)

import asyncio
from isa_agent_sdk.clients.device_client import DeviceClient
from isa_robot.runtime import RobotRuntimeConfig, create_robot_device
from isa_robot.ral import SkillRequest
 
async def run_embodied_agent():
    # 1. Initialize local RAL device driver
    device = create_robot_device(RobotRuntimeConfig(backend="rphand-mock"))
    
    try:
        # 2. Connect with explicit side-effect consent and home actuators
        await device.connect(acknowledge_side_effects=True)
        await device.actuator().enable()
        
        home_receipt = await device.skills().execute(
            SkillRequest(command_id="init-home-001", skill="home")
        )
        assert home_receipt.completion_verified, "Homing must be verified"
        
        # 3. Execute dexterous grasp skill
        grasp_receipt = await device.skills().execute(
            SkillRequest(
                command_id="task-grasp-002",
                skill="grasp",
                parameters={"synergy": "cylindrical", "current_limit": 1500}
            )
        )
        
        print(f"Grasp State: {grasp_receipt.state}")
        print(f"Sensor Confirmed: {grasp_receipt.completion_verified}")
        
    finally:
        # 4. Mandatory safe-stop and cleanup
        if device.is_connected:
            cleanup = await device.motion().safe_stop()
            assert cleanup.safe, f"Stop failed: {cleanup.errors}"
            await device.disconnect()
 
asyncio.run(run_embodied_agent())

TypeScript / Node.js Example (@isa-agent/sdk>=0.6.0)

import { createDeviceClient } from '@isa-agent/sdk';
 
async function main() {
  // Connect via GCP Gateway WSS endpoint with active token
  const client = await createDeviceClient({
    endpoint: 'wss://isagent.io/mqtt',
    deviceId: 'rphand-prod-01',
    token: process.env.ISA_DEVICE_TOKEN!,
  });
 
  // Execute motion skill with process idempotency
  const receipt = await client.skills.execute({
    commandId: 'cmd-move-101',
    skill: 'move',
    parameters: {
      positions: { thumb_flexion: 5000, index: 8000 },
      speed: 15000,
    },
    timeoutSeconds: 5,
  });
 
  if (receipt.state === 'succeeded' && receipt.completionVerified) {
    console.log('Actuator arrived at target positions successfully');
  } else {
    console.error('Execution incomplete:', receipt.lastError);
  }
}
 
main().catch(console.error);

5. Cloud Gateway & Edge Communication

Security and Transport Specifications

• Edge Gateway: Cloud communication terminates on Google Cloud Platform via wss://isagent.io/mqtt (port 8084 internally) and internal MQTTS (port 8883). Plaintext MQTT (1883) is disabled.
• Least-Privilege ACL: The user-auth service enforces strict per-device ACLs:
  • Device Username and Client ID must match the verified device_id.
  • The device is granted permission only to subscribe to its designated command topic: isa/devices/{device_id}/commands.
  • Command parameters are capped at 240 KiB JSON payload, strictly inside the 256 KiB packet budget.
• Lease Expiry: Device MQTT sessions expire automatically within 60 seconds, requiring periodic lease renewal to prevent orphaned sessions.

6. Durable OTA Firmware Updates

Firmware distribution is protected by cryptographic trust chains:

1. Dual Digest Verification: Artifacts carry both raw payload SHA-256 and release authorization signatures from Google Cloud KMS.
2. Transaction Isolation: Updates are written to isolated staging partitions before flashing.
3. Atomic Rollback: If power is lost or checksum verification fails, the bootloader automatically reverts to the preceding verified firmware release.

7. Supported Hardware Matrix