Long-Running Tasks

This guide covers running agents that may take minutes to hours, and how to keep them reliable.

1) Use Durable Execution + Checkpointing

For long tasks, enable collaborative execution with checkpoints so work can resume after interruptions.

from isa_agent_sdk import query, ISAAgentOptions, ExecutionMode
 
options = ISAAgentOptions(
    execution_mode=ExecutionMode.COLLABORATIVE,
    checkpoint_frequency=5,  # save every 5 steps
    session_id="long_task_001",
    user_id="user_123",
)
 
async for msg in query("Run a long workflow...", options=options):
    if msg.is_checkpoint:
        # Optionally show progress or persist state
        print(f"checkpoint: {msg.session_id}")
    elif msg.is_text:
        print(msg.content, end="")

To resume:

from isa_agent_sdk import resume
 
async for msg in resume(session_id="long_task_001", resume_value={"continue": True}):
    print(msg.content, end="" if msg.is_text else "\n")

2) Use Background Jobs for Hours-Long Tasks

There are two distinct background-execution paths — pick based on what you’re running:

If you need a prompt job to survive a process restart today, checkpoint it yourself (see Section 1) rather than relying on the backstop for durability.

TaskWorker (tool-list jobs)

Location:

• isa_agent_sdk/services/background_jobs/

Key components:

• nats_task_queue.py (queue)
• redis_state_manager.py (state + progress)
• task_worker.py (worker execution — runs tool-list jobs only, not arbitrary prompts)

Prerequisites

• NATS JetStream running (default port 4222)
• Redis running (default port 6379)
• REDIS_PASSWORD set if your Redis requires auth
• Worker process started

Start a Worker

python -m isa_agent_sdk.services.background_jobs.task_worker --name worker-1

If your Redis requires auth:

export REDIS_PASSWORD=staging_redis_2024
python -m isa_agent_sdk.services.background_jobs.task_worker --name worker-1

If you want the worker to only process newly enqueued tasks:

python -m isa_agent_sdk.services.background_jobs.task_worker --name worker-1 --delivery-policy new

If you need a shared task namespace across enqueuers and workers:

export BACKGROUND_JOBS_USER_ID=agent-service

Enqueue a Task

from isa_agent_sdk.services.background_jobs import enqueue_task, TaskDefinition, ToolCallInfo
 
task = TaskDefinition(
    job_id="job_123",
    session_id="sess_456",
    user_id="user_789",
    tools=[
        ToolCallInfo(
            tool_name="web_crawl",
            tool_args={"url": "https://example.com"},
            tool_call_id="call_1",
        )
    ],
    priority="high",
)
 
sequence = await enqueue_task(task)
print(sequence)

Poll Status / Result

from isa_agent_sdk.services.background_jobs import get_task_status, get_task_result
 
status = await get_task_status("job_123")
print(status.status, status.progress_percent)
 
result = await get_task_result("job_123")
print(result.successful_tools, result.total_tools)

PROMPT Jobs (in-process backstop)

For an open-ended prompt rather than a fixed tool list, the runtime’s in-process backstop executes the job without a NATS/Redis round-trip. Requests are scoped with X-User-Id:

POST /v1/autonomous/background-jobs
X-User-Id: user_789

{
  "prompt": "Summarize this week's error logs and email me the summary",
  "session_id": "sess_456"
}

Poll the same job with GET /v1/autonomous/background-jobs/{job_id} — the response shape matches get_task_status / get_task_result above. Because this path isn’t durable yet, a job in flight when the serving process restarts does not resume automatically.

The Jobs Panel

Both job types surface in the Jobs panel in the product header — start, monitor, stop, and view results for a running job without leaving the chat. See the Console developer tools overview for where this sits relative to Playground and the other panels.

run_in_background (chat tool)

End users don’t need the REST API directly — asking the agent to do something “in the background” (e.g. “kick this off in the background and let me know when it’s done”) invokes the run_in_background chat tool, which files a PROMPT job under the hood and returns immediately with a job reference. The user can check progress by asking the agent again or opening the Jobs panel; results post back into the conversation once the job completes.

3) Environment Overrides (Single Source of Truth)

Models are configured in ModelConfig.from_env() and then reused across agent settings.

Important env vars:

• AI_MODEL / DEFAULT_LLM
• AI_PROVIDER / DEFAULT_LLM_PROVIDER
• REASON_MODEL / REASON_LLM
• REASON_MODEL_PROVIDER / REASON_LLM_PROVIDER
• RESPONSE_MODEL / RESPONSE_LLM
• RESPONSE_MODEL_PROVIDER / RESPONSE_LLM_PROVIDER
• BACKGROUND_JOBS_USER_ID (shared Redis/NATS namespace for background jobs)

ReasonNode uses settings.reason_model, so REASON_MODEL is the critical override for reasoning.

4) Recommended Practices

• Use explicit session_id for long tasks
• Use checkpointing for resumability
• Use TaskWorker (tool-list jobs) when you need durability across restarts
• Use the in-process backstop (PROMPT jobs) only for jobs you can afford to re-run if the process restarts mid-flight
• Use polling, SSE, or the Jobs panel to surface progress

5) Troubleshooting

• If tool-list jobs never run: confirm NATS + Redis are healthy and the worker is running
• If tool-list jobs stay in QUEUED: check worker logs for NATS/Redis auth or Consul discovery errors
• If tool-list jobs stay in QUEUED and worker is running: ensure BACKGROUND_JOBS_USER_ID matches between enqueuer and worker
• If a PROMPT job disappears after a deploy/restart: expected — the in-process backstop has no durable resume yet; re-submit it
• If ReasonNode uses wrong model: check REASON_MODEL env var
• If responses are slow: check RESPONSE_MODEL env var