Kubernetes, reimagined for AI agents
Deploy and scale LLM-based AI agents as easily as web services. NeuroNetes handles GPU scheduling, cold starts, session management, and token-aware autoscaling—so you can focus on building.
You've built an amazing AI agent. Now you need to deploy it. Here's what you're facing:
Loading a 70B model takes 45+ seconds. Your users aren't waiting that long. Scaling from zero means losing customers.
GPUs sitting at 50% utilization while you pay full price. No easy way to share expensive hardware across workloads.
Session affinity? Build it yourself. Token-aware scaling? Write custom metrics. Model versioning? Good luck.
NeuroNetes extends Kubernetes with everything you need to run AI agents in production. Define your models, set your SLOs, and let NeuroNetes handle the rest.
Warm pools keep models preloaded. Go from zero to serving in 2.3 seconds instead of 45+. Snapshot/restore makes scaling feel instant.
Topology-aware bin-packing, MIG partitioning, and gang scheduling for multi-GPU jobs. Get 85%+ GPU utilization without manual tuning.
Scale on what matters: tokens/sec, TTFT, queue depth, context length. Not just CPU and memory. Your agents scale based on actual demand.
Conversations stick to the same backend. Graceful handoff during scaling. No lost context, ever.
First-class CRDs for models with versioning, quantization profiles, and sharding. Declarative model management finally.
TTFT, tokens/sec, tool-call latency, cost per request. Pre-built Grafana dashboards included.
Balance SLOs, $/token, and carbon footprint. Spot instances with guardrails. 40% average cost reduction.
Tool permissions, rate limits, PII detection, and audit trails. Security by default.
NATS, Kafka, SQS integration out of the box. Scale on queue depth. Message-driven architecture that works.
Specify weights, quantization, and sharding with a simple YAML. NeuroNetes handles the rest.
Set your SLOs, guardrails, and tool permissions. This is your agent's blueprint.
Set replica limits and let NeuroNetes autoscale based on real demand.
# That's it. Your agent is now production-ready.
kubectl apply -f model.yaml
kubectl apply -f agentclass.yaml
kubectl apply -f agentpool.yaml
# Check status
kubectl get agentpools
# NAME READY REPLICAS TTFT-P95
# chat-pool True 5 142ms
┌─────────────────────────────────────────────────────────────┐
│ Control Plane │
├─────────────────────────────────────────────────────────────┤
│ ┌────────────┐ ┌────────────┐ ┌────────────┐ │
│ │ Model │ │ AgentClass │ │ AgentPool │ CRDs │
│ │ Controller │ │ Controller │ │ Controller │ │
│ └────────────┘ └────────────┘ └────────────┘ │
│ │
│ ┌────────────┐ ┌────────────┐ ┌────────────┐ │
│ │ Token │ │ GPU │ │ Cost/SLA │ Schedulers │
│ │ HPA │ │ Topology │ │ Policy │ │
│ └────────────┘ └────────────┘ └────────────┘ │
└─────────────────────────────────────────────────────────────┘
│
┌─────────────────────────────────────────────────────────────┐
│ Data Plane │
├─────────────────────────────────────────────────────────────┤
│ ┌────────────┐ ┌────────────┐ ┌────────────┐ │
│ │ Sticky │ │ Stream │ │ Message │ │
│ │ Session │ │ Aware │ │ Broker │ │
│ │ Router │ │ Ingress │ │ Class │ │
│ └────────────┘ └────────────┘ └────────────┘ │
└─────────────────────────────────────────────────────────────┘
│
┌─────────────────────────────────────────────────────────────┐
│ Runtime Layer │
├─────────────────────────────────────────────────────────────┤
│ ┌────────────┐ ┌────────────┐ ┌────────────┐ │
│ │ Warm Pool │ │ Snapshot/ │ │ Sidecar │ │
│ │ Controller │ │ Restore │ │ Caches │ │
│ └────────────┘ └────────────┘ └────────────┘ │
└─────────────────────────────────────────────────────────────┘
Deploy a conversational AI agent with session management, HTTP endpoints, and cost optimization. Perfect for customer support or virtual assistants.
kubectl apply -f - <
RAG-powered code assistant with multiple tools, queue-based task processing, and session-based state management.
toolPermissions:
- name: code-search
rateLimit: 100/min
- name: file-read
rateLimit: 50/min
guardrails:
- type: pii-detection
action: redact
slo:
ttft: 500ms
tokensPerSecond: 50
Complete retrieval-augmented generation setup with vector store, embedding model, and data locality optimization.
scheduling:
dataLocality:
vectorStoreAffinity:
- weaviate
cacheAffinity:
- redis
costOptimization:
enabled: true
spotEnabled: true
Pre-built dashboards for complete observability
# Import ready-to-use Grafana dashboard kubectl create configmap neuronetes-dashboard \ --from-file=config/grafana/neuronetes-dashboard.json \ -n monitoring # Access at http://localhost:3000/d/neuronetes-agents # Default credentials: admin/admin
Deploy NeuroNetes on Amazon EKS with GPU-enabled EC2 instances
# Configure variables export CLUSTER_NAME=neuronetes-prod export REGION=us-west-2 export K8S_VERSION=1.28 # Create cluster with GPU node groups eksctl create cluster \ --name $CLUSTER_NAME \ --region $REGION \ --version $K8S_VERSION \ --node-type m5.xlarge \ --nodes 2 \ --nodes-min 2 \ --nodes-max 5 \ --managed \ --with-oidc # Add GPU node group (P4d instances with A100 GPUs) eksctl create nodegroup \ --cluster $CLUSTER_NAME \ --region $REGION \ --name gpu-nodes \ --node-type p4d.24xlarge \ --nodes 1 \ --nodes-min 0 \ --nodes-max 10 \ --node-labels workload=ai-agent \ --managed
# Add NVIDIA Helm repository helm repo add nvidia https://helm.ngc.nvidia.com/nvidia helm repo update # Install NVIDIA GPU Operator helm install gpu-operator nvidia/gpu-operator \ --namespace gpu-operator \ --create-namespace \ --set operator.defaultRuntime=containerd
# Install CRDs kubectl apply -f https://raw.githubusercontent.com/Bowenislandsong/NeuroNetes/main/config/crd/ # Deploy controllers kubectl apply -f https://raw.githubusercontent.com/Bowenislandsong/NeuroNetes/main/config/deploy/ # Verify installation kubectl get pods -n neuronetes-system
See why teams are choosing NeuroNetes over building custom solutions or using generic ML serving platforms.
Join the growing community of teams shipping AI agents to production with NeuroNetes.