Niu vs LiteLLM
Compare two self-hosted gateways by operating model. Niu focuses on Provider management, workspace budgets and customer-charge accounting. LiteLLM offers a Proxy and Python SDK with broad Provider integrations, virtual keys, spend tracking and configurable routing. Choose against the capabilities you need today.
| Feature | Niu | LiteLLM |
|---|---|---|
| Access & billing | ||
| From experimentation to ongoing operation | Provider management, company/workspace/key budgets, request accounting and text/video generation management | Self-hosted Proxy and Python SDK with virtual keys, spend tracking and an admin UI |
| Application keys | Workspace keys with model grants, expiry, revocation and rotation | Virtual keys with model access, budgets and expiry |
| Spending controls | Company funds plus independent workspace/key caps; outstanding reservations count | Key, user and team budgets; configurable budget duration |
| Pricing | Customer tariffs are separate from Provider offers; unresolved charges stay unknown | Model cost tracking with configurable pricing |
| Model access | ||
| Video generation management | Create and track supported video jobs, restore saved inputs and history, and preview/download available results with workspace-key authorization | Supported video endpoints depend on the configured model and Provider |
| Chat endpoint | POST /v1/chat/completions | POST /v1/chat/completions |
| Model names | Aliases mapped to configured upstream models | Model aliases in model_list map to upstream deployments |
| Upstream credentials | Provider businesses with separate credentials and model mappings | Provider credentials configured per deployment |
| Routing | Configured pools with priorities and weights | Load balancing across deployments with configurable strategies |
| Retries and fallbacks | Bounded Chat failover after a qualified OpenRouter 401; no generic replay of uncertain requests | Configurable retries, cooldowns, timeouts and fallbacks |
| Function tools | Opt-in route support; the caller executes requested tools | Tool calling for supported models; caller executes tools |
| Observability | ||
| Request history | PostgreSQL records: model, key, status, reported tokens, timing and customer charge | Spend logs and configurable logging integrations |
| Inspecting retries | Each recorded attempt remains distinguishable from the request | Retry and fallback configuration; logging depends on configured integrations |
| Setup | ||
| Where it runs | Your own gateway deployment | Self-hosted Proxy or Python SDK in your application |
| Database | PostgreSQL is required | PostgreSQL required for Proxy virtual-key management |
| Source code | Public core under MIT | Open-source core; review separate enterprise licensing terms |
Performance benchmark
Niu, New API and LiteLLM Proxy
Archived runs: Niu v0.2.4, New API 45fbb14 and LiteLLM Proxy v1.104.0.
Apple M4, 24 GB · PostgreSQL 17.11 · local zero-delay mock upstream. Historical synthetic results, not current-release or production capacity claims. OpenRouter was not tested.
| Workload / metric | Niu v0.2.4 | New API 45fbb14 | LiteLLM Proxy v1.104.0 |
|---|---|---|---|
| c = 64 throughput (/v1/chat/completions) | 3,238.7 RPS p50: 18.94 ms · p99: 31.54 ms | 865.6 RPS (peak) p50: 71.15 ms · p99: 135.73 ms | 302.2 RPS p50: 193.26 ms · p99: 338.74 ms |
| Anthropic /v1/messages (c = 64) | 2,518.8 RPS (p99: 60.61 ms) | 136.2 RPS (p99: 1,475.53 ms) | 232.2 RPS (p99: 397.88 ms) |
| Concurrent streaming throughput (c = 16) | 235.9 streams/s 9,436 tokens/s · Stream p50: 62.17 ms | 100.9 streams/s 4,036 tokens/s · Stream p50: 150.33 ms | 83.0 streams/s 3,320 tokens/s · Stream p50: 192.65 ms |
| Concurrent Inter-Token Latency (ITL p50, c = 16) | 1.34 ms | 3.55 ms (2.65x slower) | 4.62 ms (3.45x slower) |
| 16 agents × 30 turns (codex /v1/chat/completions) | 1,686.6 ms wall Turn p50: 54.05 ms · Turn p99: 72.50 ms | 4,891.9 ms wall Turn p50: 132.79 ms · Turn p99: 510.17 ms | 5,803.9 ms wall Turn p50: 180.75 ms · Turn p99: 328.23 ms |
| 16 agents × 30 turns (claude_code /v1/messages) | 1,746.0 ms (turn p50: 55.02 ms) | 4,864.9 ms (turn p50: 141.96 ms) | 3,353.3 ms (turn p50: 103.01 ms) |
| Idle resident memory (RSS) | 18.16 MB | 37.42 MB (2.06x Niu) | 158.00 MB (8.70x Niu) |
| Peak resident memory under c = 64 load (RSS) | 26.40 MB | 98.50 MB (3.73x Niu) | 194.60 MB (7.37x Niu) |
Different configurations: Niu used PostgreSQL admission and batched completion; New API used token caching and batched logs; LiteLLM used an in-memory master key without database spend logs. PostgreSQL synchronous_commit was off.
Why choose Niu?
| Your requirement | Niu | LiteLLM | Fits best |
|---|---|---|---|
| Use a Python SDK inside an existing application | Gateway accessed through HTTP APIs | Python SDK alongside the self-hosted Proxy | Choose LiteLLM for the Python SDK |
| Configure retries and fallbacks across deployments | Bounded Chat failover after a qualified OpenRouter 401 | Configurable retries, cooldowns and fallbacks | Choose LiteLLM when broader routing controls are required |
| Separate customer charges from Provider purchase rates | Customer tariffs independent of Provider offers; persisted customer-charge records | Model cost tracking and configurable pricing; validate your billing requirements | Evaluate Niu for this customer-billing workflow |
| Manage company funds across workspaces and application keys | Company funds with independent workspace/key caps and pending reservations | Key, user and team budgets | Compare the budget hierarchy against your organization |