Compare
Niu, LiteLLM and OpenRouter
From your first model call to ongoing application operations: Niu combines Provider management, workspace budgets, transparent usage and billing, and text/video generation management. Compare it with LiteLLM for self-hosted gateway and SDK workflows, and OpenRouter for hosted model access.
Performance benchmark
Niu, New API and LiteLLM Proxy
Archived runs: Niu v0.2.4, New API 45fbb14 and LiteLLM Proxy v1.104.0.
Apple M4, 24 GB · PostgreSQL 17.11 · local zero-delay mock upstream. Historical synthetic results, not current-release or production capacity claims. OpenRouter was not tested.
| Workload / metric | Niu v0.2.4 | New API 45fbb14 | LiteLLM Proxy v1.104.0 |
|---|---|---|---|
| c = 64 throughput (/v1/chat/completions) | 3,238.7 RPS p50: 18.94 ms · p99: 31.54 ms | 865.6 RPS (peak) p50: 71.15 ms · p99: 135.73 ms | 302.2 RPS p50: 193.26 ms · p99: 338.74 ms |
| Anthropic /v1/messages (c = 64) | 2,518.8 RPS (p99: 60.61 ms) | 136.2 RPS (p99: 1,475.53 ms) | 232.2 RPS (p99: 397.88 ms) |
| Concurrent streaming throughput (c = 16) | 235.9 streams/s 9,436 tokens/s · Stream p50: 62.17 ms | 100.9 streams/s 4,036 tokens/s · Stream p50: 150.33 ms | 83.0 streams/s 3,320 tokens/s · Stream p50: 192.65 ms |
| Concurrent Inter-Token Latency (ITL p50, c = 16) | 1.34 ms | 3.55 ms (2.65x slower) | 4.62 ms (3.45x slower) |
| 16 agents × 30 turns (codex /v1/chat/completions) | 1,686.6 ms wall Turn p50: 54.05 ms · Turn p99: 72.50 ms | 4,891.9 ms wall Turn p50: 132.79 ms · Turn p99: 510.17 ms | 5,803.9 ms wall Turn p50: 180.75 ms · Turn p99: 328.23 ms |
| 16 agents × 30 turns (claude_code /v1/messages) | 1,746.0 ms (turn p50: 55.02 ms) | 4,864.9 ms (turn p50: 141.96 ms) | 3,353.3 ms (turn p50: 103.01 ms) |
| Idle resident memory (RSS) | 18.16 MB | 37.42 MB (2.06x Niu) | 158.00 MB (8.70x Niu) |
| Peak resident memory under c = 64 load (RSS) | 26.40 MB | 98.50 MB (3.73x Niu) | 194.60 MB (7.37x Niu) |
Different configurations: Niu used PostgreSQL admission and batched completion; New API used token caching and batched logs; LiteLLM used an in-memory master key without database spend logs. PostgreSQL synchronous_commit was off.
| Capability | Niu | LiteLLM | OpenRouter |
|---|---|---|---|
| From experimentation to ongoing operation | Provider management, company/workspace/key budgets, request accounting and text/video generation management | Self-hosted Proxy and Python SDK with virtual keys, spend tracking and an admin UI | Hosted model catalog, Provider routing and API-key limits |
| Video generation management | Create and track supported video jobs, restore saved inputs and history, and preview/download available results with workspace-key authorization | Supported video endpoints depend on the configured model and Provider | Hosted video APIs and video logs; model availability depends on its catalog |
| Application keys | Workspace keys with model grants, expiry, revocation and rotation | Virtual keys with model access, budgets and expiry | API keys with expiry and management APIs |
| Spending controls | Company funds plus independent workspace/key caps; outstanding reservations count | Key, user and team budgets; configurable budget duration | USD key limit; optional daily/weekly/monthly reset |
| Pricing | Customer tariffs are separate from Provider offers; unresolved charges stay unknown | Model cost tracking with configurable pricing | Catalog charges paid with credits; BYOK is also supported |
| Request history | PostgreSQL records: model, key, status, reported tokens, timing and customer charge | Spend logs and configurable logging integrations | Generation logs: model, Provider, key, usage, cost and timing |
| Inspecting retries | Each recorded attempt remains distinguishable from the request | Retry and fallback configuration; logging depends on configured integrations | Upstream Requests tab and Provider response waterfall |
| Chat endpoint | POST /v1/chat/completions | POST /v1/chat/completions | POST /api/v1/chat/completions |
| Model names | Aliases mapped to configured upstream models | Model aliases in model_list map to upstream deployments | Catalog model slugs |
| Upstream credentials | Provider businesses with separate credentials and model mappings | Provider credentials configured per deployment | OpenRouter credits or your Provider keys (BYOK) |
| Routing | Configured pools with priorities and weights | Load balancing across deployments with configurable strategies | Provider order; price, latency or throughput sorting |
| Retries and fallbacks | Bounded Chat failover after a qualified OpenRouter 401; no generic replay of uncertain requests | Configurable retries, cooldowns, timeouts and fallbacks | Provider fallbacks; allow_fallbacks can disable them |
| Function tools | Opt-in route support; the caller executes requested tools | Tool calling for supported models; caller executes tools | Routes to tool-capable Providers; caller handles execution |
| Where it runs | Your own gateway deployment | Self-hosted Proxy or Python SDK in your application | OpenRouter-hosted service |
| Database | PostgreSQL is required | PostgreSQL required for Proxy virtual-key management | No gateway database to operate |
| Source code | Public core under MIT | Open-source core; review separate enterprise licensing terms | Hosted service; client SDKs are public |
Go deeper