isA Model Service

Unified AI model platform for inference, training, and generic ML prediction.

Installation

pip install isa-model

Quick Start

from isa_model import AsyncISAModel
 
async with AsyncISAModel(base_url="http://localhost:8082") as client:
    response = await client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": "Hello!"}]
    )
    print(response.choices[0].message.content)

Latest Updates (2026-05-17)

• Sleep intelligence APIs added under /api/v1/sleep.
• Sleep evaluation endpoint added: POST /api/v1/sleep/evaluate.
• Sleep recommendation endpoint added: POST /api/v1/sleep/recommend.
• Commercial sleep model products registered: com_sleep_quality, com_sleep_stage, and com_health_risk.
• Commercial model evidence gate added to validate serving behavior before release.

Earlier Updates (2026-02-14)

• Generic ML prediction API added with dedicated training and inference routes.
• Prediction endpoint pagination added (limit / offset) for model and prediction lists.
• Tenant isolation enforced for prediction model storage and retrieval.
• Prediction reliability/security hardening:
  • training data key validation (ds, y)
  • frequency / period bounds
  • sanitized error persistence
  • deleted-model safety checks during background training
  • HMAC integrity checks for saved model artifacts
  • tenant-scoped LRU model caching and eviction limits
• Auth integration improved by validating API keys through the auth service.
• Optional dependency guards added for Lightning algorithm modules.

Core Capabilities

Inference

• OpenAI-compatible chat/completions interface
• Multi-provider routing (OpenAI, Anthropic, Ollama, Cerebras, YYDS, Replicate, OpenRouter, Anthropic Sub)
• Streaming, tool-calling, and structured outputs
• Multi-modal support (text, vision, audio, embedding, image, video)
• Role-based and capability-based model resolution with cost-aware ranking

Voice & Realtime

• WebSocket bidirectional proxy for real-time voice conversations
• Session management with 15-minute TTL and reconnect window
• Voice privacy filter with PII redaction (SSN, phone, email, credit card)
• Configurable retention policies (STRICT, STANDARD, DEBUG)
• Turn detection with server-side VAD
• Per-session metrics (latency, audio chunks, interruptions)

Caching

• L1 exact-match cache (Redis)
• L2 semantic cache (Qdrant)
• Cache stats and invalidation endpoints

Training

• Lightning training pipeline endpoints for trace-based optimization
• Algorithm support for closed_loop, apo, grpo, and custom

Generic ML Prediction

• Model lifecycle for prediction models (create/train/list/get/delete)
• Inference endpoint for forecasts
• Tenant-scoped prediction history with pagination

Sleep Intelligence

• Sleep-quality evaluation
• Sleep-stage recommendation
• Health-risk assessment products for commercial serving flows

API Endpoints

Base URL: http://localhost:8082

Inference & Cache

Voice & Realtime

Training

Prediction (ML)

Sleep

Prediction Example

# 1) Create + train a model (prophet)
curl -X POST http://localhost:8082/api/v1/models/training/ml/prediction/models \
  -H "Content-Type: application/json" \
  -d '{
    "name": "daily-sales-forecast",
    "model_type": "prophet",
    "data": [
      {"ds": "2026-01-01", "y": 120},
      {"ds": "2026-01-02", "y": 125}
    ]
  }'
 
# 2) Run prediction
curl -X POST http://localhost:8082/api/v1/models/inference/ml/prediction/models/<MODEL_ID>/predict \
  -H "Content-Type: application/json" \
  -d '{
    "periods": 14,
    "frequency": "D",
    "include_history": false
  }'

Intelligent Model Routing

The invoke endpoint supports role-based and capability-based model selection with optional cost constraints:

# Role-based resolution — automatically selects best model for the role
curl -X POST http://localhost:8082/api/v1/invoke/v2 \
  -H "Content-Type: application/json" \
  -d '{
    "role": "reasoning",
    "messages": [{"role": "user", "content": "Solve this logic puzzle"}],
    "preferred_max_cost_per_token": 0.00005
  }'

Role defaults:

Resolution priority: explicit model/provider → role lookup → capability filter → service type defaults.

Latency-Aware Ranking

When multiple candidate models match a request (e.g., several models share the same capability), the router ranks them using a weighted composite score that balances cost and recent latency performance.

Composite score formula:

score = (1 - latency_weight) * normalized_cost + latency_weight * normalized_latency

• normalized_cost — per-token output cost scaled to [0, 1] across candidates.
• normalized_latency — provider P50 latency scaled to [0, 1] across candidates.
• The candidate with the lowest composite score wins.

When no latency data is available (cold start or new provider), the ranker falls back to cost-only sorting automatically.

Example request with latency preference:

curl -X POST http://localhost:8082/api/v1/invoke/v2 \
  -H "Content-Type: application/json" \
  -d '{
    "role": "coding",
    "messages": [{"role": "user", "content": "Refactor this function"}],
    "preferred_max_cost_per_token": 0.0001,
    "preferred_max_latency_ms": 500
  }'

Per-Provider Latency Tracking

The Model service maintains a sliding-window latency tracker for every provider. Each completed inference request records its response latency, and the tracker computes percentile statistics on demand.

Tracked percentiles:

How data flows:

1. Every inference call records latency_ms for the provider that served it.
2. Measurements are stored in a per-provider sliding window (default: last 1,000 requests).
3. The latency ranker reads P50 from the tracker when scoring candidates.
4. A NATS telemetry event (isa.model.selection) is published for each selection decision, including the observed latency and chosen strategy. NATS failures never block inference.

Querying latency stats:

Provider latency percentiles are exposed through the health/stats endpoint and can also be queried programmatically:

from isa_model.serving.api.provider_latency import get_latency_tracker
 
tracker = get_latency_tracker()
 
# Get percentiles for a single provider
openai_stats = tracker.get_percentiles("openai")
print(f"OpenAI P50={openai_stats.p50_ms}ms  P95={openai_stats.p95_ms}ms  P99={openai_stats.p99_ms}ms")
 
# Get percentiles for all tracked providers
all_stats = tracker.get_all_percentiles()
for provider, stats in all_stats.items():
    print(f"{provider}: P50={stats.p50_ms}ms ({stats.sample_count} samples)")

Selection Statistics

The Model service also collects aggregate selection statistics that reveal routing patterns over time:

These statistics are available via the /api/v1/models/selection/stats endpoint.

Tuning Guide

Control latency-aware routing behavior with these environment variables:

Tuning recommendations:

Window size guidance:

• 100 — Reacts quickly to provider degradation; noisier percentiles.
• 1,000 (default) — Stable percentiles with reasonable responsiveness.
• 5,000+ — Very smooth percentiles; slow to reflect sudden latency spikes.

For production deployments, monitor P95 and P99 through the Grafana SLO dashboard (deployment/monitoring/grafana-slo-dashboard.json) and adjust the weight if a provider consistently exceeds latency SLOs.

Voice Session Example

# 1) Create a realtime voice session
curl -X POST http://localhost:8082/realtime/sessions \
  -H "Content-Type: application/json" \
  -H "x-user-id: user-123" \
  -d '{"model": "gpt-4o-realtime-preview", "voice": "alloy"}'
 
# Response: { "session_id": "...", "websocket_url": "ws://...", "expires_at": "..." }
 
# 2) Connect via WebSocket for bidirectional audio streaming
# 3) Interrupt (barge-in) during a response
curl -X POST http://localhost:8082/realtime/sessions/{session_id}/interrupt

Privacy modes:

Documentation

Getting Started

• Quick Start - First steps
• Providers - Provider configuration

Features

• LLM Services - Text generation and routing
• Tool Calling - Function calling
• Caching - Cache architecture and tuning

Environment Variables