Dashboard
Why NiuPerformance · Reliability · Economy

Make every
token count.

The Token Platform for developers and teams. Access models with performance and reliability, and keep token usage and spending under control.

Open dashboardGet startedStar on GitHub

Compare gatewaysBrowse modelsWhy Niu?

requests / second
3,239
faster than LiteLLM
10.7×
p99 tail latency
31.5 ms
idle RAM footprint
18.2 MB
Default workspace / Logs

Logs

1,428 requests

Inspect every workspace operation, upstream route-pool attempt, phase latency, and settled customer tariff.

Request volume · Last 24 hours99.8% completed · 2 bounded failovers · 1 uncertain stream
All statusesPool: claude-sonnet-4
All time
Model & timeStatusCharge & latency
A
claude-sonnet-414:22:08 · Anthropic
Completed$0.01801,180 ms
OR
claude-sonnet-414:22:07 · OpenRouter
Failed · HTTP 401Not charged3.2 ms
DS
deepseek-r114:19:41 · OpenRouter
UncertainUnresolved4,820 ms
Gateway · Workspace Dashboard · Model Catalog · Docs — served from one endpointOpen workspace dashboard
Performance

Fast model access

A Rust gateway with streaming support and measured proxy performance. Connect applications and coding tools through compatible API endpoints.

Reliability

See what happened to every request

Model routing, bounded failover and attempt-level logs help you understand failures. Unresolved usage stays explicit.

Economy

Keep token spending visible

Review model prices, token usage and request charges. Set workspace and API-key budgets before usage grows.

API + Chat + Video

One workspace for your model usage

Use APIs in your applications, compare models in Chat, and revisit saved conversations and supported video results.

10× PERFORMANCE · MEASURED BENCHMARKS

3,239 RPS in Rust.
10.7× LiteLLM. 3.7× New API.

Why Rust? Because your gateway should never be the bottleneck. Measured with ai-gateway-bench on Apple M4 with PostgreSQL 17.11, Niu sustains 3,239 RPS with full database accounting and 18.2 MB idle RAM.

  1. 01
    3,238.7 RPS peak throughput

    10.7x LiteLLM Proxy and 3.7x New API peak at c = 64 (/v1/chat/completions), with 31.54 ms p99 tail latency.

  2. 02
    235.9 concurrent SSE streams/s

    9,436 tokens/s at c = 16 with 1.34 ms median inter-token latency (ITL) and 100% SSE chunk preservation.

  3. 03
    18.2 MB idle / 26.4 MB peak RSS

    7.4x lower peak RAM and 10.7x lower proxy compute cost ($0.014 / 1M requests) while persisting ACID PostgreSQL records.

AI-GATEWAY-BENCH · 0MS MOCK UPSTREAM (36,649 RPS DIRECT BASELINE)Apple M4 (10-core, 24 GB) · PostgreSQL 17.11 · 0% errors
CLOSED-LOOP LOAD PROBE · 32 WARMUP + 2,000 REQUESTS PER TIER

High-concurrency /v1/chat/completions & /v1/messages (c = 1 to 64)

Isolates gateway routing, workspace key verification, protocol handling, and PostgreSQL accounting overhead against a deterministic 0ms mock upstream (36,649 RPS direct baseline).

Durability barACID PostgreSQL admission & completion on Niu · 0% errors

Niu v0.2.4

Rust · Axum · Tokio

3.7x – 10.7x faster
PEAK THROUGHPUT AT C = 643,238.7 RPSp50: 18.94 ms · p95: 26.11 ms · p99: 31.54 ms
c = 16 RPS / p99
1,040.0 / 22.26 ms
c = 1 RPS / p50
233.6 / 4.00 ms
/v1/messages (c=64)
2,518.8 / 60.6 ms
PostgreSQL work
ACID + batch settle

New API

Go · Gin · GORM (45fbb14)

26.7% of Niu
PEAK THROUGHPUT AT C = 64865.6 RPSp50: 71.15 ms · p95: 111.82 ms · p99: 135.73 ms
c = 16 RPS / p99
612.4 / 408.08 ms
c = 1 RPS / p50
187.9 / 4.36 ms
/v1/messages (c=16)
164.5 / 473.9 ms
PostgreSQL work
Cached + SQL logs

LiteLLM Proxy

Python · FastAPI (v1.104.0)

9.3% of Niu
PEAK THROUGHPUT AT C = 64302.2 RPSp50: 193.26 ms · p95: 318.24 ms · p99: 338.74 ms
c = 16 RPS / p99
328.0 / 163.16 ms
c = 1 RPS / p50
136.5 / 6.41 ms
/v1/messages (c=64)
232.2 / 397.9 ms
PostgreSQL work
None (stateless)
10.7xhigher peak RPS than LiteLLM Proxy

Full ACID PostgreSQL admission and completion batching without sacrificing tail latency.

Niu records pre-dispatch operations and attempts in a single SQL CTE and coalesces up to 64 post-dispatch completions per batch transaction, keeping c = 64 p99 latency at 31.54 ms while LiteLLM Proxy saturates its Python event loop at ~302 RPS.

Fairness note: Niu v0.2.4 ran with full ACID PostgreSQL pre-dispatch admission & batched completion; New API (45fbb14) ran with MEMORY_CACHE_ENABLED=true; LiteLLM Proxy (v1.104.0) ran in stateless master_key mode with DB logs disabled. Always qualify your own deployment and workload.

Full benchmark report & methodology Side-by-side matrix

YOUR TOKEN WORKSPACE

Access, use and manage models.
Keep usage in one workspace.

Connect your applications, try models in Chat, and manage keys, budgets and request history together.

01

Model routing & bounded failover

Use one model entry point across configured routes. Inspect each attempt and understand when a fallback occurred.

Models & routing
02

Request logs & usage evidence

Inspect request status, token counts, timing and charges without stitching together separate dashboards.

Request evidence
03

Clear usage & spending

Review the price of a model and the charge for each request. Keep missing or unresolved billing evidence visible.

Usage & spending
04

Workspace & API-key budgets

Set scoped API keys and spending caps for your applications and team, with in-flight budget reservations.

Access & limits
05

Drop-in for SDKs & coding agents

Native OpenAI (/v1/chat/completions, /v1/responses) and Anthropic (/v1/messages) compatibility. Change one Base URL to connect Claude Code, Codex, or your app.

API & SDK
06

Saved Chat & Video workspaces

Try models in Chat and manage supported video jobs. Saved sessions and authorized results stay available across visits.

Generations

DROP-IN COMPATIBILITY

One endpoint for SDKs
and coding agents.

Point any OpenAI-compatible client, Claude Code, or Codex at your Niu gateway using a scoped workspace API key.

  • 01
    Choose a model for your application

    Browse available models, check capabilities and prices, and choose the model that fits your workload.

  • 02
    Issue workspace API keys with model & budget limits

    Create a workspace API key, limit its model access, and set a spending cap for your application.

  • 03
    Change one Base URL—Niu handles failover and billing

    Clients stream responses with zero code changes while Niu handles automatic failover, token metering, and real-time billing.

API quickstart TypeScript SDK
POST /v1/chat/completionsTLS / Localhost · Bearer Workspace Key
curl http://localhost:2555/v1/chat/completions \
  -H "Authorization: Bearer $NIU_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4",
    "messages": [{"role": "user", "content": "Check Q3 p99."}],
    "stream": true
  }'
Recorded in PostgreSQL: Operation → Attempt #1 (Anthropic 200 OK) → 40 SSE tok → Settled Customer Charge ($0.018)

FULL-STACK OPEN SOURCE · SELF-HOSTED

Full-stack open source.
Deploy with a single PostgreSQL database.

The gateway, dashboard, model catalog, and docs are MIT-licensed with zero black-box dependencies or license servers—your data stays 100% yours.

READY TO DEPLOY TODAY

Included in the open-source stack

  • High-speed Rust gateway (/v1/chat/completions, /v1/responses, /v1/messages)
  • Smart Model Pools with conditional routing, failover, and strategy audit
  • Company balance, workspace budgets, and per-key spend protection
  • Request logs, token usage and customer-facing charges
  • Public Model Catalog, saved multi-model Chat, and Seedance Video studio
Single-container deployment guide
CONTINUOUS PRODUCTION QUALIFICATION

Transparent engineering roadmap

  • Continuous protocol & edge-case verification across upstream Providers
  • Commercial payment gateways and automated Provider settlement workflows
  • Async video generation lifecycle and media retention policies
  • Multi-node high availability, crash recovery, and zero-downtime upgrades
Release acceptance matrix

FROM THE NIU BLOG

Engineering notes from
the gateway team.

Model access, token usage and reliable API integration in practice.

All articles
Product2 min read

Why Niu? A Token Management Platform for Developers and Teams

Performance, reliability and economy for people using model APIs. Access models, manage token spending and understand every request.

Read article
Comparisons2 min read

Niu, LiteLLM and OpenRouter: choosing your model API platform

Hosted model access or a gateway you operate? Compare operating responsibilities, workspace budgets and request evidence before choosing.

Read article

FREQUENTLY ASKED QUESTIONS

Key questions before
you run in production.

What developers and teams need to know before connecting their applications.

Read the docs Ask on GitHub
01Why NiuWho is Niu for?

Niu is a token management platform for developers and teams using model APIs. Access models, manage workspace keys and budgets, and inspect usage and request charges from one workspace.

Browse models
02EconomyHow can I control token spending?

Check the listed model price, review actual token usage and charges, and set workspace and key-level budgets. Prices and availability depend on the configured offers; Niu does not promise every model is cheaper.

Usage & budgets
03Billing accuracyWhat happens if an upstream stream drops halfway through?

No mystery charges and no fake $0 bills. Niu records pre-dispatch state in PostgreSQL and never blindly replays a request that may have already run upstream. If a stream drops without usage evidence, the call is explicitly flagged as unresolved for review.

Request evidence contract
04Full-stack open sourceIs the full stack truly open source for self-hosting?

Yes. The Rust gateway, admin console, customer workspace dashboard, model catalog, and docs are all MIT-licensed and run together against a single PostgreSQL database—with zero license servers or private dependencies.

Single-container deployment
05IntegrationCan I use Niu with my existing applications?

Use a scoped workspace API key with the documented OpenAI-compatible and Anthropic-compatible endpoints. Check the model capabilities and supported protocol before switching your application.

API quickstart
06Production readinessWhich capabilities can I use?

Niu includes model access, workspace API keys and budgets, request logs, and saved Chat and supported Video workflows. Availability depends on configured models and permissions. Consult the release matrix for qualification limits.

Release acceptance matrix

PERFORMANCE · RELIABILITY · ECONOMY

Put your tokens to work.
Keep usage under control.

Browse models, create a workspace API key and start calling. You can also self-host the open-source stack.

Open dashboardGet startedStar on GitHub