Reliability

Reliability patterns added and hardened in ISA MCP.

Key Improvements (2026-04-08)

• Health/readiness semantics with healthy vs degraded states
• Graceful semantic-to-lexical search fallback
• Vector ID overflow detection for Qdrant point IDs
• Exponential-backoff retry for transient Qdrant failures
• Cache key versioning for safe schema evolution
• Atomic delete patterns to avoid count/delete race conditions
• Transaction-wrapped migrations with rollback-ready structure

Health and Degraded Modes

GET /health distinguishes ready, degraded, and unready states instead of collapsing all dependency issues into a single healthy response.

The health payload includes a dedicated search block:

{
  "search": {
    "status": "ok",
    "mode": "semantic"
  }
}

When the model dependency is unavailable, search degrades instead of hard-failing:

{
  "search": {
    "status": "degraded",
    "mode": "lexical",
    "reason": "ISA Model health check failed: ReadTimeout"
  }
}

Operationally:

• treat search.mode = semantic as the normal steady state
• treat search.mode = lexical as serviceable but degraded
• treat search.status = error as unready and page on it

Standard Retry and Rate-Limit Headers

Structured resilience errors are translated into standard HTTP headers so clients can back off automatically.

HTTP 429 responses may include:

• Retry-After
• X-RateLimit-Limit
• X-RateLimit-Remaining
• X-RateLimit-Reset

HTTP 502 responses raised from an open circuit breaker include:

• Retry-After

Vector Store Safety

Overflow Detection

Vector IDs use type offsets. Capacity is bounded per type to avoid collisions.

• max items per type: 1,000,000
• warning threshold: 900,000

When limits are exceeded, writes fail early with explicit errors.

Retry Strategy

Qdrant writes/deletes are retried on transient failures:

• max retries: 3
• base delay: 0.5s
• exponential backoff progression

Cache Versioning

Cache keys include an explicit schema version prefix:

mcp:cache:v{CACHE_VERSION}:...

When schema changes, bumping CACHE_VERSION safely invalidates stale payload shapes.

Database Consistency

Atomic Deletes

COUNT + DELETE race windows were removed via CTE-style atomic delete-and-count queries.

Migration Safety

Migrations use:

• BEGIN/COMMIT transactional boundaries
• backup-first patterns for destructive changes
• idempotent DDL guards where possible

Health Failure Persistence

The aggregator persists health failure counts and error timestamps in the server registry so that failure state survives process restarts. Each server record carries two fields:

How it works

A background health monitor runs on a 30-second interval. For each connected server:

1. A session-level health check is issued.
2. On success, health_failures is reset to 0 in both in-memory state and the registry.
3. On failure, health_failures is incremented and last_error_at is updated.
4. Once health_failures reaches the threshold (default 3), the server status transitions to DEGRADED.

Because failures are written to the registry (PostgreSQL or in-memory store), a restarted aggregator can pick up where it left off instead of assuming all servers are healthy.

Health check response

The single-server health check returns:

{
  "server_id": "uuid",
  "server_name": "my-server",
  "status": "connected",
  "is_healthy": true,
  "consecutive_failures": 0,
  "last_check": "2026-03-12T10:00:00Z",
  "health_check_url": "http://localhost:8080/health"
}

Automatic recovery

The reconnect_unhealthy method iterates over all servers in DEGRADED or ERROR status and attempts reconnection. On a successful reconnect, health_failures is reset to 0.

Connection Event Tracking

Every connection lifecycle transition is recorded as a ConnectionEvent in a per-server ring buffer (default capacity: 100 events per server). Events are stored newest-first and are available through MCP tools for diagnostics.

Event types

Event structure

Each event contains:

{
  "event_type": "connect",
  "timestamp": "2026-03-12T10:00:00Z",
  "server_id": "uuid",
  "details": "Connected successfully"
}

Aggregator Observability Tools

The following MCP tools expose health and connection state at runtime. They are registered under AggregatorTools and available to any connected client.

aggregator_health

Run a health check across all connected servers or a single server.

Returns a list of health status objects (see health check response above).

get_aggregator_state

Dashboard-style snapshot of the entire aggregation layer.

Returns:

{
  "total_servers": 5,
  "connected_servers": 3,
  "disconnected_servers": 1,
  "error_servers": 1,
  "total_tools": 42,
  "servers": [
    { "id": "uuid", "name": "my-server", "status": "connected", "tool_count": 12 }
  ]
}

get_server_connection_info

Detailed connection metadata for a single server including transport type, uptime, session ID, last event ID, and the 10 most recent events.

Returns:

{
  "server_id": "uuid",
  "transport_type": "stdio",
  "connected_at": "2026-03-12T09:00:00Z",
  "uptime_seconds": 3600,
  "session_id": "abc123",
  "last_event_id": "evt-99",
  "recent_events": [
    { "event_type": "connect", "timestamp": "2026-03-12T09:00:00Z", "details": "Connected successfully" }
  ]
}

get_connection_history

Full connection event history for a server, useful for post-incident analysis.

Returns an array of event objects, newest first.

reconnect_server

Force-disconnect and reconnect a specific server, rediscovering its tools. Use this when a server is misbehaving or after its configuration changes.

Returns:

{
  "success": true,
  "server_id": "uuid",
  "status": "connected",
  "tools_discovered": 12
}

On reconnect, health failures are reset to 0 and a reconnect event is recorded in the connection history.

Related

• Security
• Multi-Tenant