AI gateway RFP questions to ask before you choose a partner
You are about to choose a partner to run a self-hosted AI gateway, such as LiteLLM or Apache APISIX, for many teams at once. This section sets out how to run that evaluation, what a good answer contains and the red flags that should slow a decision down. Send every question in writing, ask for a reference architecture and a written scope before any conversation about price, and require each answer to name the product version and the configuration file or setting it applies to. An answer that says "we configure routing" without naming the model_list entries, router strategy or priority order it will produce tells you little about what you will receive.
Ask each candidate to show, in writing, how keys, budgets and logs are set up. A strong answer names the database holding virtual keys and spend (LiteLLM documents PostgreSQL as its primary supported database), the team and key hierarchy, the rate limit and budget fields used, and where prompt logs land, how long they are kept and who can read them. Ask whether any feature you need, such as SSO beyond the free user limit, SCIM, audit logs or team-level guardrails, requires a LiteLLM Enterprise license, and ask the partner to price that license separately from its own services. Confirm the answer against the LiteLLM Enterprise documentation and the license file, because feature boundaries change between releases.
DeployOpen does not publish a price list. Quotes follow from a written scope that lists deliverables, responsibilities and timelines, so compare candidates on that scope rather than on headline day rates. Use the same question set for every candidate, score the written answers against the same red flags, and keep the reference architecture and scope as the basis of the contract. Vendors that will not commit to a written scope or that cannot name the configuration they will deliver should be treated as unfinished proposals.
Questions by area and what a good answer contains
How should a partner configure LiteLLM routing and failover across providers?
A strong answer names the routing strategy and the failure behavior, not just the provider list. LiteLLM's router supports simple-shuffle (which its documentation recommends), usage-based, latency-based, least-busy and cost-based routing. Ask the partner which strategy they would start with and why, how retries are set, how cooldowns are triggered after repeated failures, and how priority ordering defines the failover sequence. Ask for session affinity settings if conversations must stay on one model. Request a written test plan that forces a provider error and shows which model answered, and ask how the routing configuration is version-controlled and reviewed before each change.
How do virtual keys and budgets map to teams and applications in LiteLLM?
Virtual keys are the unit that carries budgets, rate limits and model restrictions, and they need a PostgreSQL database. LiteLLM supports per-key, per-user and per-team budgets, TPM and RPM limits, temporary budget increases and restricting each key to specific models. Ask the partner to show the mapping in a written design: one team entity per cost center, keys per application, who may request a budget increase and who approves it. Ask how spend is attributed when a key is shared, and how the partner will confirm that a budget limit actually blocks a call in a test.
Where are prompt logs stored, how long are they kept, and who owns retention?
A good answer names the storage location, the retention period, the deletion method and a named owner for each. Ask where request and response logs are written, whether they land in PostgreSQL, a separate logging backend or your own SIEM, and whether prompt bodies are stored at all or only metadata. Ask for the retention value as a configured setting, not a verbal promise. For high-risk AI systems, Article 19(1) of the EU AI Act requires providers to keep automatically generated logs for at least six months unless other law provides otherwise, so ask the partner to confirm which retention rule applies in your case with your legal team.
Which LiteLLM features require an enterprise license, and which ones does the open source edition cover?
The answer should list each feature you need and mark it as open source or enterprise. LiteLLM's code is MIT licensed except for the enterprise directory, which uses the license in enterprise/LICENSE. From v1.76.0, SSO for the admin UI is free for up to 5 users, and an Enterprise license is needed beyond that. Enterprise adds SCIM, JWT authentication with identity provider tokens, audit logs, organization and team RBAC, IP allowlisting, secret manager integrations and team-level guardrails and logging. Ask the partner to state which of these they would need for your user count and to confirm in writing which files they will deploy, so no enterprise code is used without a license. Ask LiteLLM directly for current pricing.
How are guardrails applied to requests and responses at the gateway?
A good answer states where each guardrail runs (before the provider call, after the response, or both), which checks it performs, and what happens on a match: block, redact or log only. Ask whether guardrails are configured per key, per team or globally, and whether team-level guardrails require the enterprise edition. Ask which detection tools they will use for prompt injection, personal data and unsafe output, and whether those run inside your infrastructure. Request a test set of prompts with expected outcomes, and ask who approves changes to guardrail rules.
How does the gateway scale across multiple instances, and which components need Redis and PostgreSQL?
PostgreSQL is the primary supported database, and LiteLLM's production guidance requires Redis 7.0 or newer once more than one proxy instance runs, because Redis holds shared rate limits, router state and cache. Ask the partner to confirm the Redis topology, how it is made highly available, and how failover works if Redis is lost. Ask how many instances they expect for your traffic and how it was sized, using the guidance of 1 vCPU and 4 GiB memory per worker as a starting point. LiteLLM recommends one Uvicorn worker per pod on Kubernetes and horizontal scaling, and batches spend updates to the database every 60 seconds at scale. Ask how the partner will load test before go-live.
How is the admin UI secured, and how is the master key managed?
Ask the partner to name the identity provider for admin UI sign-in and the exact configuration path. LiteLLM supports SSO with Google, Microsoft Entra ID, Okta and generic OIDC providers, subject to the licensing limit described above. The LITELLM_MASTER_KEY must start with sk- and should live in a secret manager, not in a configuration file. Ask who holds the key, how it is rotated, and whether break-glass access is logged. Ask how the partner will restrict admin access by network and role, and what evidence they will hand over for an access review.
Answers that should worry you
Partners rarely lie outright about an AI gateway. They give vague or overconfident answers on the points that carry the most risk: where prompts are stored, which features cost money, how guardrails are enforced, and how the gateway behaves under load. The cards below describe the answers to be wary of and what to ask for instead.
No clear answer on prompt logs
A partner who cannot say which database or object store holds prompt and response logs, who can read them, or how long they are kept is not ready to run a gateway in a regulated setting. Ask for a written data map showing where each log field lands, the retention period set in configuration, the deletion procedure, and which personnel can view raw prompts. Logs can contain personal data and confidential material, so this answer needs to be specific before any pilot starts.
All LiteLLM features presented as free
LiteLLM is MIT licensed outside its enterprise directory, but some capabilities sit under the enterprise license. A partner who says SSO, audit logs, SCIM or team-level guardrails are free without checking the enterprise directory may leave you with a license gap after go-live. Ask the partner to list each required feature and its license source, and confirm the current free SSO user limit against the vendor's terms before you commit.
Guardrails treated as a model setting
A partner who describes guardrails as a switch on the model provider, or as a prompt instruction telling the model to behave, has not separated the controls. Input filtering, PII redaction, blocked providers and output checks should run at the gateway, where every request passes through regardless of which model answers. Ask where each guardrail executes, what happens when it fails, and whether a request can bypass it by calling a provider directly.
Uptime promised without a tested design
Claims of high availability or fixed throughput without a described architecture are not evidence. A multi-instance LiteLLM deployment needs shared state, and LiteLLM's production guide names Redis 7.0 or newer once more than one proxy instance runs for shared rate limits, router state and cache. Ask for the topology, the failover test results from a staging environment, and the load profile used to size workers, and treat any uptime figure without these as marketing.
Certification or audit outcomes offered
DeployOpen and most implementation partners prepare systems, controls and evidence for assessments, but independent auditors and certification bodies issue reports and attestations. A partner who says the gateway is certified, or guarantees that an audit will pass, is overstating its role. Ask which controls the gateway supports with evidence, who performs the assessment, and what the partner will not claim in writing.
Routing and failover without a test plan
Routing strategies such as simple-shuffle, latency-based and cost-based routing, along with retries, cooldowns and priority ordering, each change which provider sees regulated data and how often a request fails over. A partner who names the strategy but cannot show how fallback behaves for a sensitive workload has not thought through data residency. Ask which providers each route may reach, how fallback is restricted by data classification, and how failover is tested before production use.
How prompt log retention questions connect to the EU AI Act
Log storage and retention belong in the RFP because a gateway that sits in front of every model call may end up recording the evidence an AI system is required to produce. Article 12(1) of the EU AI Act requires high-risk AI systems to technically allow for the automatic recording of events (logs) over the lifetime of the system. Article 19(1) requires providers to keep those logs for at least six months unless other law provides otherwise, as set out by the EU AI Act Service Desk. Whether your organization is the provider or the deployer of a given system, and which retention period applies to it, is a question for legal review, not for the gateway vendor to answer.
Ask the implementation partner to show where the gateway writes prompt and response records, which component holds them (typically the PostgreSQL database that LiteLLM uses for virtual keys and spend, or a separate log sink you configure), and how long each store keeps them. Then ask who owns those logs, where they physically sit, and which roles can read raw prompts. Ask how deletion requests are handled, including how a deletion reaches backups and downstream observability tools, and how a deletion is itself recorded. Confirm the answers against your own environment, because logging settings, retention jobs and access controls are configured per deployment rather than supplied by default.
Treat the retention decision as a design input, not a cleanup task. A retention period that is too short may leave you without evidence the legal review requires, while one that is too long keeps personal data and confidential prompts longer than necessary. Ask the partner to document the retention settings, the access model for log readers, and the procedure for legal holds, so that the answers can be tested during onboarding. Keep the legal review in the scope agreed before implementation starts, so that the gateway configuration reflects the obligations that actually apply.
How the questions change by evaluation track
A platform team building a shared gateway needs answers about topology, failure behavior and day-two operations. The questions focus on how the proxy scales across instances, how shared state is kept consistent, and how routing and failover behave when a provider degrades. LiteLLM's production guidance names PostgreSQL as the primary supported database and calls for Redis 7.0 or newer once more than one proxy instance runs, because rate limits, router state and cache are shared. APISIX offers traditional, decoupled and standalone deployment modes, so the partner should explain which one fits the cluster and why.
- Scaling model: Ask how replicas are run on Kubernetes, including the recommended worker count per pod and how spend writes are batched at volume.
- Routing and failover: Ask which router strategy is configured, how retries and cooldowns are tuned, and how priority ordering is tested against a forced provider outage.
- Shared state: Request the PostgreSQL and Redis topology diagram, including backup, failover and where
LITELLM_MASTER_KEYis stored. - Configuration change control: Ask how routes, models and keys are versioned, promoted between environments and rolled back.
- Upgrade policy: Confirm which minor versions receive support and how a version upgrade is tested before production.
Request a load test report and the architecture diagram for the target environment before comparing vendors on features.
A finance team needs spend to map cleanly to cost centers, and it needs the mapping to hold up under monthly review. The questions focus on how keys, users and teams are structured, what budget controls exist, and whether usage data can be exported in a form the ledger can reconcile. LiteLLM virtual keys require PostgreSQL and support per-key, per-user and per-team budgets, TPM and RPM limits, temporary budget increases and model restrictions. Those controls only help chargeback if the team hierarchy is maintained as a governed record rather than a spreadsheet.
- Key hierarchy: Ask how keys are issued per application and team, who approves a new key, and how keys are retired when a project closes.
- Budget enforcement: Ask what happens when a budget is reached, whether requests stop or fall back to a cheaper model, and who approves a temporary increase.
- Spend reporting: Request a sample budget report per virtual key and per team, with the export format and the refresh interval.
- Price tables: Ask who maintains per-model cost data when a provider changes prices, and how the partner tests that change.
- Reconciliation: Ask how gateway totals are matched against provider invoices, including the lag between a call and its recorded spend.
The finance track is complete only when a sample export has been reconciled by the finance team using invoice data from a test period.
A regulated buyer needs to show who called which model, with what data, and what the gateway allowed or blocked. The questions focus on log content, storage location, retention, access control and export for auditors. EU AI Act Article 12(1) requires high-risk AI systems to technically allow automatic recording of events over their lifetime, and Article 19(1) requires providers to keep those logs for at least six months unless other law provides otherwise. Retention in a gateway is therefore a policy decision that the buyer must set and the partner must implement, not a default to accept.
- Log content: Ask which fields are written per request, such as caller identity, model, timestamp, policy decision and token counts, and whether prompt bodies are stored at all.
- Storage and retention: Ask where logs are kept, who can read them, how long they persist, and how deletion is evidenced.
- Log export: Request a sample export from a test environment and confirm the format an auditor can read without vendor tooling.
- Admin access: Ask how SSO, role-based access and audit logs of administrator actions are configured, and which of these need an enterprise license.
- Guardrail evidence: Ask how blocked requests are recorded and how the partner tests that a policy rule actually fires.
Confirm with your own compliance lead which retention period applies to your system before accepting any default in a proposal.
How to run the evaluation in four steps
The sequence moves from written answers to a demonstration on a real configuration of the gateway, then to scored comparison and a fixed-scope proposal. Each step produces a record that the next step depends on, so the buying decision rests on evidence rather than sales claims.
Send the written question set to each candidate
The buying team sends the same written questions to every candidate partner, covering routing and failover, virtual keys and budgets, prompt log storage and retention, LiteLLM license boundaries, guardrail placement and horizontal scaling. Ask each partner to answer in writing, cite the exact product version they would deploy, and name the license that applies to each feature they rely on. The step is complete when every candidate has returned written answers to the same questions, and the security lead has confirmed the answers are specific enough to test.
Request a demonstration on a non-production instance
Ask each shortlisted partner to show the gateway running on a non-production instance in a environment that resembles your own, using the same provider endpoints and identity setup you plan to use. The demonstration should include a failover test when a provider returns errors, a budget cap reached by a test virtual key, a log entry being written and then read back from the database, and a guardrail blocking a sample request. The step closes when the engineering lead has recorded what was observed, what was configured by hand, and what the partner could not show.
Score answers with a shared rubric
Before the demonstrations, the evaluation group agrees on a rubric with weighted criteria for security, operability, licensing exposure, log retention, scaling approach and the quality of the written answers. Each reviewer scores each partner independently, then the group reconciles differences and records the reason for every score that moved by more than one point. The output is a single scoring sheet signed by the CISO or delegate, showing which claims were verified in the demonstration and which remain open.
Resolve open items with the shortlisted partners
Send each shortlisted partner the list of unverified claims and licensing questions from the scoring sheet, and ask for written responses. Typical open items include which LiteLLM features need an Enterprise license for your SSO and audit requirements, where the prompt logs sit for each deployment mode, and how the Redis and PostgreSQL tiers are sized for your expected traffic. The step is complete when each open item has a written answer or has been moved to a named risk owner.
Ask for a fixed-scope written proposal
Request a fixed-scope proposal from the top candidate that lists the platforms, integrations, identity connections, log storage design, backup and upgrade responsibilities, test evidence and acceptance criteria. The proposal should separate one-time implementation work from recurring support and license costs, and state which licenses the client buys directly. The step closes when the proposal is accepted in writing by the sponsoring executive and the procurement lead, with the scope, deliverables and acceptance criteria signed off before implementation starts.
How to score partner responses
This table gives each evaluation area a 0 to 3 scale for evidence quality, specificity of named settings, licensing clarity and operational ownership. Score each cell only when the partner has supplied a written example from its proposal or a demonstration, and use the scores to compare answers side by side, not as a final verdict.
| Area | Evidence quality | Specificity of named settings | Licensing clarity | Operational ownership |
|---|---|---|---|---|
| Routing and failover | 0 means a general claim; 3 means a documented routing strategy, retry count, cooldown rule and priority order tested in a demo | 0 means no setting names; 3 means named router strategy, such as simple-shuffle or latency-based, with the failover order written out | 0 means silent on features; 3 means each routing feature is marked as open source or enterprise | 0 means unclear who tunes routing; 3 means a named owner for changes and a documented review cycle |
| Keys and budget mapping to teams | 0 means a verbal answer; 3 means a sample key hierarchy from virtual keys, per team and per user, shown in the admin UI | 0 means generic budgets; 3 means named limits such as TPM, RPM, per-key budgets and model restrictions | 0 means unclear whether team RBAC or SSO requires a paid license; 3 means the license boundary is stated per feature | 0 means no owner for team onboarding; 3 means a written process for issuing, rotating and revoking keys |
| Prompt log storage and retention | 0 means no location named; 3 means a stated database, storage location, access path and retention rule, with a sample log record | 0 means retention described as configurable only; 3 means the setting name, default and the owner who sets the period | 0 means unknown whether audit logs are in the open source or enterprise tier; 3 means the tier is stated in writing | 0 means no plan for backups or log access reviews; 3 means a named role for log review and a backup schedule |
| Enterprise license features | 0 means a list without sources; 3 means each feature is traced to current documentation with a date | 0 means feature names only; 3 means SSO, SCIM, audit logs, RBAC and secret manager integrations listed separately | 0 means the partner treats the whole product as one license; 3 means open source and enterprise components are separated, including the enterprise directory | 0 means the license purchase is not in scope; 3 means the partner owns license procurement and renewal tracking |
| Guardrails and PII controls | 0 means a product name only; 3 means a described control point, input and output checks, and a tested block example | 0 means no rules named; 3 means specific rule types, action on match and how exceptions are approved | 0 means third-party guardrail license terms are not addressed; 3 means each guardrail component's license is listed | 0 means no testing owner; 3 means a named owner and a test cycle that includes prompt injection and data leakage cases |
| Scaling across instances | 0 means no architecture described; 3 means a diagram with shared PostgreSQL and Redis, worker count and horizontal scaling rules | 0 means no sizing inputs; 3 means stated resource sizing per worker, number of replicas and the state store used | 0 means no mention of licensed components needed for multi-instance use; 3 means Redis and database licensing or hosting terms are stated | 0 means no load test or monitoring plan; 3 means a named load test plan and the metrics the operator will watch |
| Admin access and SSO | 0 means generic SSO claim; 3 means the identity provider, protocol and admin role mapping are named | 0 means no settings named; 3 means the SSO configuration steps and the user limit that applies are stated | 0 means the free user limit is not mentioned; 3 means the partner states the user threshold and the license it triggers | 0 means no admin account review; 3 means a documented admin access review with a named owner |
| Evidence for compliance reviews | 0 means no artifacts offered; 3 means sample log exports, configuration files and change records are offered for review | 0 means evidence described in general terms; 3 means specific artifacts named, such as configuration exports and access logs | 0 means no statement on whether evidence is sufficient for the assessor; 3 means the partner states it prepares evidence and does not issue attestations | 0 means no retention of evidence after go-live; 3 means a defined evidence pack refreshed on a named schedule |
What drives the cost of the gateway work
The software line is the smallest part of the bill for most gateway projects. LiteLLM's core is under the MIT license, and content in its enterprise directory uses a separate license, so the enterprise license is an optional line item you choose, not a default cost. Features that typically sit behind it include SCIM provisioning, audit logs, organization and team RBAC, IP allowlisting and secret manager integrations. The LiteLLM admin UI SSO is free for up to 5 users, and an Enterprise license is needed beyond that, so the number of admin users changes the license count. Ask the vendor to confirm the current terms for each feature you need, since licensing can change between releases.
The infrastructure and the partner's fixed-scope work are mostly one-time costs, while operations recur. PostgreSQL is the primary supported database for LiteLLM and holds virtual keys, budgets and spend data. Redis 7.0 or newer becomes required once more than one proxy instance runs, for shared rate limits, router state and cache. Cost rises with the number of teams and virtual keys, the number of model providers to integrate and test, the number of environments (development, staging, production, and any disaster recovery copy), the volume of logged prompts and their retention period, and the depth of identity integration with your SSO and directory. Each of these adds configuration, testing and storage, so each should appear as a named assumption in the scope.
When you request a quote, ask for itemized licenses, assumptions and exclusions as separate sections. Licenses should list every enterprise feature, the user or instance basis for it and the renewal terms. Assumptions should state the number of teams, providers, environments, expected request volume, log retention period and identity sources. Exclusions should name what the partner will not do, such as provider contract negotiation, model evaluation, guardrail tuning, or ongoing on-call support. Also ask for the recurring operations scope: upgrades across minor versions, backup and restore testing, and security patch handling, with the frequency and who performs each task. Compare the quote with your own internal effort so the recurring cost is visible before the first deployment.
Build, partner or hosted service: what to compare
This table compares an in-house build, a partner-delivered self-hosted gateway and a hosted gateway plan across log control, licensing exposure, operating responsibility and exit path. Read each row as a question to put to your own team, your partner or your vendor contract, because the answers depend on the deployment and the agreement you sign.
| Dimension | In-house build | Partner-delivered self-hosted | Hosted gateway plan |
|---|---|---|---|
| Where prompt logs are stored | Your storage, designed and retained by your team | Your infrastructure, with retention set in the written scope | Vendor-hosted, confirm region, retention and deletion terms in the contract |
| Licensing exposure | Open-source components plus any enterprise features you enable, tracked by your legal team | Same components, with license review of each enterprise feature before rollout | Governed by the vendor's subscription terms and usage metering |
| Enterprise features needed for SSO, SCIM and audit logs | Build or license them yourself, confirm each feature's license status | Confirm which features need a paid license before the scope is signed | Confirm which features are included in the plan you are buying |
| Operating responsibility for uptime and upgrades | Your platform team, including PostgreSQL, Redis and gateway upgrades | Shared, with responsibilities for patching, backups and on-call written into the scope | Vendor operates the service, confirm the support terms and incident commitments |
| Scaling across instances | Your team designs shared state, rate limits and cache | Partner designs and tests the shared state layer as part of the scope | Vendor controls the scaling, confirm limits and how they change |
| Exit path to another gateway or self-hosted models | Strongest control, since configuration and data stay with you | Strong if the scope includes exporting config, keys and logs in documented formats | Depends on export options in the contract, test an export before you commit |
| Evidence you can hand to an auditor | You generate and store all records | Partner documents controls and evidence, the auditor decides what is sufficient | Ask for logs and audit records you can export independently |
Questions buyers ask before hiring outside help
The first question most buyers ask is what the work needs from their own team. For an AI gateway, the answer depends on who will own the model provider accounts, the identity provider used for the admin UI, the PostgreSQL and Redis services behind the gateway, and the approval process for which models each team may call. A partner should list these inputs in the written scope before work starts, and your team should confirm early who will hold each credential, who approves changes to routing rules and who receives the logs. If your team cannot name those owners, the schedule will slip regardless of who does the build.
Buyers also ask whether they can run the gateway themselves once it is live. That is usually realistic for a team that already operates Kubernetes or similar container platforms and PostgreSQL, but it depends on whether your team can keep the LiteLLM or APISIX configuration in version control, rotate the master key held in your secret manager and test failover before an incident forces the question. Ask the partner which runbooks, configuration exports and handover sessions are included, and whether your staff will make routing and budget changes during the rollout or only watch.
On timing, DeployOpen works on fixed-scope engagements, so the schedule is set in the written scope and not in a generic estimate. The main drivers are the number of providers and self-hosted models to connect, how many identity and secret manager integrations are required, whether guardrails and log retention rules need approval from legal or compliance, and how many teams need virtual keys at launch. Ongoing support, upgrades and continuous testing are available as retainers. DeployOpen prepares systems, controls and evidence for compliance assessments, but it does not issue audit reports, attestations or certifications. Ask for that boundary to be written into the scope so your assessor's requests are matched to what was delivered.
How DeployOpen scopes, delivers and supports AI gateway work
DeployOpen turns the questions in an AI gateway RFP into a written scope for a self-hosted gateway inside your own cloud accounts, data centers or air-gapped network. We test each answer against the platform you choose, then document, hand over and support the result.
How we scope
- Written scope lists which routing, failover and budget behaviors are configured, tested and documented for your teams.
- We confirm which LiteLLM features your requirements need and which depend on an Enterprise license.
- Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts.
How we build
- We deploy the gateway with PostgreSQL for virtual keys and spend tracking, and Redis when more than one instance runs.
- We configure key, team and budget mappings, model allowlists and guardrail placement, then test failover paths end to end.
- We secure the admin interface with your identity provider and keep master keys in your secret manager.
How we hand over
- Deliverables include architecture notes, configuration records and a runbook for routing changes, key rotation and budget updates.
- We document where prompt logs are stored, who can read them and how long your retention policy keeps them.
- Your staff receive walkthroughs of the admin console, alerting and scaling steps before the handover is closed.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers, covering version updates to the gateway.
- We retest routing, budgets and guardrails after each upgrade or change to providers and model endpoints.
- Compliance evidence, such as log samples and configuration exports, is refreshed for your assessors on a schedule you set.
