NeuroNetes extends Kubernetes to provide agent-native orchestration for LLM workloads. This document describes the system architecture, design decisions, and component interactions.
The control plane manages the lifecycle of agent workloads through custom controllers.
Model Controller
AgentClass Controller
AgentPool Controller
ToolBinding Controller
GPU Topology Scheduler
┌─────────────────────────────────────────┐
│ Scheduling Pipeline │
├─────────────────────────────────────────┤
│ 1. Filter Nodes │
│ - GPU availability │
│ - MIG partition matching │
│ - Topology requirements │
│ - Data locality │
├─────────────────────────────────────────┤
│ 2. Score Nodes │
│ - Cost efficiency │
│ - SLO headroom │
│ - Model cache presence │
│ - Network topology │
├─────────────────────────────────────────┤
│ 3. Gang Scheduling │
│ - Multi-GPU coordination │
│ - All-or-nothing placement │
│ - NUMA awareness │
├─────────────────────────────────────────┤
│ 4. Binding │
│ - Reserve GPU resources │
│ - Update node state │
│ - Trigger model loading │
└─────────────────────────────────────────┘
Token-Aware HPA
Metrics Collection → Metric Evaluation → Scaling Decision
↓ ↓ ↓
- tokens_in_queue - Compare to target - Scale up/down
- ttft_p95 - Apply smoothing - Respect limits
- concurrent_sessions - Check stability - Execute action
- tokens_per_second window
Cost/SLA Policy Engine
The data plane handles request routing and traffic management.
Request → Session Key Extraction → Affinity Check → Route
↓ ↓ ↓ ↓
Headers conversation_id Hash Table Target Pod
user_id Cache TTL Fallback
custom_key Load Balance
Features:
Connection → Protocol Detection → Stream Handler → Backpressure
↓ ↓ ↓ ↓
WebSocket gRPC/HTTP2 Token Streaming Flow Control
SSE HTTP/1.1 Cancellation Rate Limiting
Chunking QoS
Capabilities:
Queue/Topic → Consumer Group → Agent Pool → Response Queue
↓ ↓ ↓ ↓
NATS Partition Processing Acknowledgment
Kafka Assignment Timeout Dead Letter
SQS Load Balance Retry Monitoring
Features:
The runtime layer optimizes agent execution and resource utilization.
Pool State → Prewarm Strategy → Instance Management → Readiness
↓ ↓ ↓ ↓
Target Size Model Loading Container Start Health Check
Current Size Weight Caching GPU Allocation Ready/NotReady
Scaling Lazy Init Memory Map Serve Traffic
Optimization strategies:
Running Agent → Checkpoint → Storage → Restore → Running Agent
↓ ↓ ↓ ↓ ↓
Memory State CRIU/CRAC S3/PV Fast Load Resume
Model Weights Metadata Compression Decompress Context
Context Incremental Dedup Memory Map Generation
Techniques:
KV Cache (Short-term Memory)
Agent ←→ Sidecar ←→ Redis/Memcached
↓
Per-session storage
Encrypted at rest
TTL-based expiration
Key: conversation_id
Value: compressed JSON
Vector Cache (RAG)
Agent ←→ Sidecar ←→ Vector DB Shard
↓
Node-local shard
Embedding cache
LRU eviction
Co-located scheduling
Standard metrics exported:
# Time-To-First-Token
neuronetes_ttft_seconds{agent_class, model, route}
# Tokens per second throughput
neuronetes_tokens_per_second{agent_class, model, direction}
# Input/output token counts
neuronetes_tokens_total{agent_class, model, direction}
# Tool invocation latency
neuronetes_tool_latency_seconds{tool_name, agent_class}
# Retrieval metrics
neuronetes_retrieval_misses_total{agent_class}
neuronetes_retrieval_latency_seconds{agent_class}
# Safety blocks
neuronetes_safety_blocks_total{guardrail_type, action}
# Cost tracking
neuronetes_cost_usd{agent_class, model}
Traffic → Route Matching → Split Logic → Variants → Compare
↓ ↓ ↓ ↓ ↓
Request Intent/Prompt % to A/B System Quality
Headers Classification Canary Prompt Metrics
Rules Tools Rollback
Features:
Request → Admission → Guardrails → Decision → Response
↓ ↓ ↓ ↓ ↓
Payload Webhook PII Detect Allow Forward
Headers Policy Safety Block or
Jailbreak Redact Reject
Guardrail types:
1. Client Request
↓
2. Ingress Controller
↓
3. Guardrail Admission (pre-processing)
↓
4. Session Router (affinity check)
↓
5. Agent Pod
↓
6. Sidecar Cache (context fetch)
↓
7. Model Inference
↓
8. Tool Invocations (if any)
↓
9. Token Streaming
↓
10. Metrics Collection
↓
11. Sidecar Cache (state save)
↓
12. Response to Client
1. Message Published to Queue
↓
2. ToolBinding Consumer
↓
3. Load Balancing across Pool
↓
4. Agent Pod Processing
↓
5. Model Inference
↓
6. Tool Invocations
↓
7. Result Publishing
↓
8. Message Acknowledgment
↓
9. Metrics Collection
1. Metrics Aggregator
↓
2. Token-Aware HPA
↓
3. Scaling Decision
↓
4. AgentPool Controller
↓
5. Warm Pool Activation (if available)
↓ OR
Pod Creation
↓
6. Scheduler Selection
↓
7. GPU Allocation
↓
8. Model Loading
↓
9. Readiness Check
↓
10. Traffic Routing Update
Standard Kubernetes scheduler doesn’t understand:
Custom schedulers enable:
CPU/memory-based HPA fails because:
Token-based metrics enable:
Benefits:
Agent workloads need:
Session affinity provides:
Challenges:
Solutions:
Agent memory contains:
Protection:
Layers:
Techniques:
Target: < 500ms TTFT from cold
Optimization strategies:
Target: > 80% GPU utilization
Optimizations:
Add custom autoscaling metrics:
type CustomMetricProvider interface {
GetMetricValue(ctx context.Context, pool *AgentPool) (float64, error)
GetMetricName() string
}
Extend scheduling:
type CustomSchedulerPlugin interface {
Filter(ctx context.Context, pod *Pod, node *Node) bool
Score(ctx context.Context, pod *Pod, node *Node) int64
}
Add guardrails:
type GuardrailPlugin interface {
Check(ctx context.Context, request *Request) (*GuardrailResult, error)
GetType() string
}