Specific AI gateway jobs done with APISIX plugins: balancing and failing over between LLM providers, token budgets, prompt and content guards, self-hosted model routing, LLM metrics and MCP.
Spreading LLM traffic across providers and keeping it flowing when one fails: weighted, hash and semantic balancing, priorities, fallback on 429 and 5xx, retry limits, health checks and unique instance names required in 3.19.
Capping LLM spend per team or customer with ai-rate-limiting: limiting total, prompt or completion tokens, per-instance and per-consumer budgets, shared counters in Redis and shifting traffic when a quota runs out.
Filtering prompts and responses at the gateway: ai-prompt-guard, AWS and Aliyun content moderation, Lakera Guard, the 3.18 changes to moderation scope and fail modes, and where gateway checks stop and model-level controls begin.
Putting APISIX in front of models you run with vLLM or Ollama: OpenAI-compatible upstreams in ai-proxy, mixing self-hosted and cloud providers, authentication and quotas per consumer, and keeping prompts inside your network.
Measuring LLM usage at the gateway: token and model logging, apisix_llm_latency with time to first token since 3.18, dashboards that break on the new type label, per-consumer cost reporting and log retention.
What APISIX offers for Model Context Protocol traffic today: the deprecated mcp-bridge plugin, the openapi-to-mcp plugin, authentication and rate limits for agent clients, and what to confirm before relying on either.
Tell us what you're working on, whether it's a deployment, an audit, a security test or a cyber range. You'll speak with an engineer who can help you scope it.