How a self-hosted AI gateway supports DORA ICT third-party risk for LLM APIs
A bank, insurer or investment firm whose staff and applications call hosted LLM APIs has a third-party register problem before it has an AI problem. A self-hosted AI gateway, running in the entity's own cloud account or data center, gives the ICT risk function one place to inventory every model provider, enforce an approved list of endpoints and evidence exit planning under Articles 28(1), 28(3), 28(7) and 28(8) of Regulation (EU) 2022/2554. DORA does not certify anything. DeployOpen prepares the systems, controls and evidence that an entity's own assessors and supervisors will review, but it does not issue audit reports or attestations. The gateway is where the evidence is generated. Each provider and model alias is defined once in the gateway configuration, and applications reach them only through the gateway's OpenAI-compatible endpoint, so the list of providers in use can be exported and matched against the register of information that Article 28(3) requires. Virtual keys can be restricted to specific models, which turns the approved-provider decision into a configuration state that can be reviewed and version controlled. Per-team budgets and request logs record which provider received which traffic and when, which is the input a supervisor will ask for when a provider is questioned. Exit planning under Article 28(8) is the harder part, and the gateway helps only if the exit path is built and tested. Confirm with the provider team which open-weight models will run on self-hosted serving infrastructure, and route a representative set of non-sensitive workloads to them through the gateway's router so failover is exercised rather than assumed. Record the switch time, the quality differences and the data flows as part of the documented exit strategy. Confirm with legal and your competent authority's expectations which contract terms must be present under Article 28(7), since the gateway produces evidence but does not write those clauses.
Mapping DORA Article 28 requirements to gateway configuration
This table maps four DORA Article 28 obligations to what the financial entity must do and how to configure LiteLLM or APISIX to produce the supporting controls and records. Read each row as a contribution to the entity's ICT third-party framework, not as proof of compliance, since the gateway covers only the model-access layer. The main takeaway is that the gateway makes provider inventory, access limits and failover testable, while contracts, due diligence and exit governance remain separate work.
| Requirement | What the function must do | How to configure LiteLLM or APISIX |
|---|---|---|
| Article 28(1): ICT risk framework integration | Manage each hosted model provider as an ICT third-party service within the ICT risk framework, with the entity remaining fully responsible. | Register each provider as a named model group with an owner and risk tier in config. Use virtual keys per team so every provider call traces to an accountable business owner. Confirm the risk review workflow with your own GRC tooling. |
| Article 28(3): Register of information | Maintain a complete, current record of all contractual arrangements for ICT services, provided on request and reported yearly for new arrangements. | Export the model group list, provider endpoints and key ownership on a schedule from the LiteLLM database or the APISIX configuration store. Tag each route with the contract reference. Confirm with your legal team which fields the register template requires. |
| Article 28(7): Termination conditions | Ensure the entity can act on triggers such as significant breaches or conditions that prevent effective supervision, including by stopping traffic to a provider. | Use a provider kill switch: disable the model group or route in config and confirm that virtual keys cannot reach it. Test that a disabled provider stops receiving requests within the change window your runbook sets. Contract termination rights still sit in the agreement. |
| Article 28(8): Exit strategy and transition | Document and test an exit strategy and transition plan for services supporting critical or important functions. | Define a fallback order from the hosted provider to a self-hosted vLLM or Ollama endpoint using priority failover. Run scheduled failover tests against the self-hosted model and record latency and output checks. Confirm model quality parity with the business owner before relying on the path. |
Which open-source gateway and model platforms fit this function
The gateway is the control point where DORA-relevant decisions become enforceable: which providers are reachable, who holds keys, and what gets logged. The options below cover the gateway itself, the self-hosted model targets that make an exit path real, and the trace store that keeps records in your own storage.
LiteLLM as the policy gateway
LiteLLM sits in front of more than 100 model APIs using the OpenAI request format, so one allowlist and one logging path cover every provider. The core is MIT licensed, but the enterprise directory uses a separate license, so confirm which features you need (SSO beyond the free tier, SCIM, audit logs, team-level guardrails) before designing around them. Multi-instance deployments need PostgreSQL for virtual keys and Redis 7.0 or newer for shared rate limits and router state.
Virtual keys and provider allowlists
Virtual keys tie each team or application to a set of permitted models, budgets and rate limits, which is the mechanism for enforcing approved providers in code rather than in policy documents. Restrict each key to named models and test that a request to an unapproved provider fails. Store the master key in a secret manager, not in configuration files.
Routing, cooldowns and failover
The router supports simple-shuffle, usage-based, latency-based, least-busy and cost-based strategies, with retries, cooldowns after repeated failures and priority ordering for failover. For DORA, failover is a resilience control only if the fallback provider is itself on the register and approved for the same data. Document which routes can fail over across providers and which must stop.
Apache APISIX as the edge gateway
APISIX is Apache-2.0 licensed and runs in traditional, decoupled or standalone mode. Standalone mode reads configuration from a YAML or JSON file, which suits change-controlled deployments where routing rules are reviewed like code. Its openid-connect plugin works with Keycloak, Entra ID, Okta and other identity providers, and custom plugins can be written in Lua, Java, Go, Python or WebAssembly. Confirm whether its AI plugins cover your provider-allowlist needs before relying on them.
vLLM and Ollama as exit targets
An exit path needs a destination that already works. vLLM (Apache-2.0) serves open-weight models behind an OpenAI-compatible endpoint on your own GPUs, and Ollama (MIT) runs models locally for smaller workloads. Routing a representative set of prompts to these targets during testing shows whether quality, latency and capacity are acceptable before a provider is ever removed. Model licenses are separate from runtime licenses, so check each model's terms.
Langfuse for trace records
Langfuse is MIT licensed except its ee directories, and it can store traces, prompts and evaluation results in your own database. Use it to keep the detailed trace record behind each gateway log entry, which supports incident reconstruction and exit testing. Confirm which Langfuse features sit under the ee license, and set retention in your own storage rather than relying on defaults. Avoid storing sensitive prompt content in traces without a redaction decision.
Configuring provider inventory and approved endpoints for the register
The first configuration area is a complete, current list of every model provider and endpoint that the gateway can reach. Under DORA, the financial entity must keep a register of information covering all contractual arrangements for ICT services, so the gateway's model configuration is a practical place to start. Each provider entry should map to a contract, a legal entity, a service description and an owner, because the register asks for those details and not only for a technical endpoint. The register remains the entity's own record. The gateway supplies source data, and the compliance function, with procurement and the ICT risk function, decides what is recorded and how it is maintained.
- Named provider entries: Give every provider and model endpoint a stable identifier that links to a contract reference in procurement records.
- Export of configured models: Schedule a regular export of the configured model list so that additions and removals are visible for the yearly reporting cycle.
- Change history: Keep records of when providers were added, changed or disabled, and by whom, so the register reflects the state at each reporting date.
- Ownership fields: Record the accountable business owner and technical owner for each endpoint alongside the gateway configuration.
- Gap reconciliation: Compare gateway entries with the procurement and SaaS inventory to find AI services in use that never passed through the gateway.
Confirm with your compliance team which fields the register template requires and which of them the gateway can supply directly, since the register format is set by the entity's supervisory guidance.
The second area restricts which models each team or application may call. An allowlist turns the approved-provider decision into a setting that the gateway enforces on every request, rather than a policy document that depends on individual developers. LiteLLM virtual keys can restrict a key to specific models, and APISIX routes can be limited by consumer and path, so the same approved set can be applied at both the key and route level. Each allowlist entry should trace back to a documented approval decision that includes the provider's risk assessment, data location and exit options.
- Per-team scoping: Restrict each virtual key to the models its team has been approved to use, and review the mapping on a fixed schedule.
- Approval reference: Store the approval record identifier with each allowlist entry so an auditor can trace a model to its decision.
- Deny by default: New providers stay blocked until they are added to an allowlist, so unreviewed endpoints cannot receive traffic.
- Application-level routes: Apply stricter allowlists to applications that process customer or regulated data than to internal experimentation.
- Denied request logs: Record rejected calls so the team can see attempts to reach unapproved providers.
Confirm with your gateway partner how allowlist changes are approved, versioned and rolled back, and whether the enforcement applies to every route or only to keys issued through the admin interface.
The third area sets spending and throughput limits per key, user and team. Budgets matter for DORA because a sudden change in usage or cost at one provider can signal concentration risk, and because the entity must be able to show how much it depends on each arrangement. LiteLLM supports per-key, per-user and per-team budgets, tokens-per-minute and requests-per-minute limits, and temporary budget increases. These settings also feed the usage data that the register and the entity's concentration analysis rely on. Budget data is operational data, so the entity should decide how long it is kept and who can change it.
- Budget per owner: Assign a monthly budget to each team or key, with the owner named in the configuration.
- Throughput limits: Set requests-per-minute and tokens-per-minute limits to contain runaway usage and to limit dependence on a single provider during peaks.
- Temporary increases: Require a recorded approval for any temporary budget increase, with an expiry date.
- Spend exports: Export spend by provider and by team to the same store that holds the register source data.
- Usage thresholds: Alert the ICT risk function when a provider's share of total traffic crosses a level the entity has set.
Ask the partner where spend records are stored, how often they are written, and what happens to budget enforcement when the database is unavailable, since LiteLLM virtual keys depend on PostgreSQL.
The fourth area separates routes to hosted provider APIs from routes to models the entity runs itself, such as those served by vLLM or Ollama. This separation supports the exit strategy that Article 28(8) expects for services supporting critical or important functions. If the gateway exposes one OpenAI-compatible endpoint to applications, the route table decides where each request goes, and the entity can test whether a self-hosted fallback works before a provider problem occurs. Routes should be labeled by deployment model, data location and contractual status so that the register and the exit plan can be produced from the same configuration.
- Labeled route groups: Tag each route as hosted external, hosted in the entity's contracted cloud or self-hosted, and record the data location for each tag.
- Tested fallback: Run scheduled failover tests from hosted routes to self-hosted routes and keep the results as exit evidence.
- Priority ordering: Use LiteLLM priority ordering or APISIX plugin logic so that regulated workloads never fall back to an unapproved external endpoint.
- Session affinity: Define when a session may move between providers, since a switch mid-conversation can change the data that a provider receives.
- Route change records: Log each change to the route table as a configuration event tied to an approval.
Confirm with your architecture team which self-hosted models can serve each regulated workload and at what quality, because a fallback that exists on paper but cannot meet the workload's needs does not support an exit plan.
Building and testing an exit path to self-hosted models
This sequence turns the exit strategy required by Article 28(8) into a tested, repeatable capability: each step produces evidence that a critical or important function can move from a hosted provider to models the institution runs itself. It also produces the records a competent authority or internal audit would ask to see.
Classify workloads by function and criticality
Start by listing every application that calls a model through the gateway, then mark which ones support a critical or important function as defined in the entity's own ICT risk framework. Risk, compliance and the business owner agree the classification, and the gateway's virtual keys are mapped to each application so every workload has an owner. Done means each workload has a documented criticality rating, a named owner and a list of acceptable fallback model types. Ask the risk committee which classification method it already uses so the result matches existing ICT risk records.
Stand up self-hosted model endpoints
The platform team deploys open-weight models on the institution's own servers or GPUs using a serving engine such as vLLM or Ollama, exposed behind the gateway as OpenAI-compatible endpoints. Each model is tested against the prompts and output formats the critical workloads use, and quality gaps are recorded rather than assumed away. Done means each fallback model answers the agreed test set, is registered in the gateway model list and has its own logging enabled. Confirm with the vendor and your infrastructure team which hardware the chosen models require, since that is not covered here.
Configure failover rules and traffic shares
Engineers define routing groups in the gateway that send a defined share of traffic for each workload to the self-hosted model, using priority ordering for failover, retries and cooldowns after repeated failures. Routing rules are version-controlled so changes are reviewed like code. Done means a change ticket records the routing configuration, the share sent to each target and the rollback step. Check whether the chosen gateway edition supports the routing and session features you need, because some may depend on license tier.
Run a documented switchover test
The test team runs a scripted switchover in a non-production environment and then in a controlled production window, moving the defined traffic share from the hosted provider to the self-hosted model and back. They measure error rates, latency, output acceptance against the test set and whether logs and budgets still attribute calls correctly. Done means the test record shows the start and end state, the observed results, any defects and who signed off on each result. Confirm with the risk committee whether it requires production testing or accepts a staging test for each workload class.
Document the transition plan and exit strategy
The exit strategy and transition plan are written for each workload that supports a critical or important function, covering triggers for moving traffic, roles, communication steps, data handling and the conditions for returning to the provider. The document is approved by the accountable owner and reviewed whenever the provider, model list or gateway configuration changes. Done means the plan references the most recent test record and names the review cycle agreed with the risk committee. The plan does not replace the contract clauses required under Article 28(7), so legal review should confirm both align.
Record results against Article 28(8) evidence
Test outputs, routing configurations, plan versions and approvals are stored in the institution's own evidence repository and linked to the matching entry in the register of information. Compliance checks that each critical or important workload has a current plan and a test result no older than the review cycle the committee set. Done means an auditor or supervisor can trace one workload from its register entry to its classification, failover configuration, test record and approved plan. Ask your compliance lead which retention period applies to these records, since that is set by internal policy and other law.
Repeat the test after material change
The switchover test is rerun whenever the hosted provider, model version, gateway release or workload changes in a way that could affect output or routing, and on the schedule the risk committee sets. Results from each run are compared with earlier records so drift in quality or latency is visible over time. Done means the repeat test produces a dated record, updates the exit plan where needed and closes any defects from the previous run. Repeatability is what makes the exit path credible, so treat a test that cannot be rerun from its written script as a finding.
Evidence supervisors and auditors ask for and which component produces it
This table maps each piece of evidence a supervisor or internal auditor is likely to request for a hosted LLM provider to the component that produces it. Read the owning component as the system of record, with the entity's GRC system holding the contract and register content the gateway cannot produce. Retention periods depend on the entity's own record-keeping rules and the authority's expectations, so confirm them before design.
| Evidence item | Owning component | Retention note |
|---|---|---|
| Register of information extract for each LLM provider | Entity GRC system, fed by gateway provider inventory exported from LiteLLM model configuration | Confirm retention with your register and supervisor reporting rules; keep dated snapshots |
| Provider contracts and termination clauses | Entity GRC system or contract repository (outside the gateway) | Follow contract and DORA record-keeping obligations; retain after termination as your legal team confirms |
| Gateway configuration exports (model routes, allowlists, fallback order) | LiteLLM proxy configuration under version control, with APISIX route configuration where used | Keep every released version; confirm retention against change management policy |
| Virtual key, budget and team mapping records | LiteLLM virtual keys stored in PostgreSQL, with change history in audit logs (Enterprise) | Confirm retention with finance and audit; key revocation history should persist |
| Request and routing logs (which provider served which request) | LiteLLM spend and request logs in PostgreSQL, or Langfuse traces where deployed | Confirm retention per data classification; prompt content may fall under GDPR minimization rules |
| Exit test records and transition plan evidence | Entity GRC system, with test outputs from the self-hosted model route in LiteLLM | Retain each test run with date, scope and result; confirm period with your exit strategy policy |
| Incident and provider failure records | Entity incident management system, using gateway error and cooldown logs from LiteLLM | Align with DORA incident reporting timelines and your internal retention schedule |
What a gateway alone does not satisfy under DORA
A gateway enforces routing, keys and logging at runtime, but DORA places duties on governance, contracts, regulator communication and exit planning that sit outside the technical stack. The cards below separate what the gateway can evidence from what the financial entity must still decide, document and test.
Board-level ownership of ICT third-party risk
Article 28(1) requires ICT third-party risk to be managed as an integral part of the ICT risk management framework, and the financial entity stays fully responsible for compliance at all times. A gateway can show who called which model, but it cannot accept accountability for that use. Check that the management body has approved the policy covering LLM providers, that named owners exist for each provider, and that the gateway's approved-provider list is a governed artifact rather than a configuration file an engineer edits.
Contract terms the gateway cannot create
Article 28(7) requires contracts with ICT providers to be terminable in defined circumstances, such as significant breaches, changes that could alter performance, evidenced weaknesses in the provider's ICT risk management, or conditions that prevent effective supervision. Routing traffic away from a provider does not change what the contract allows. Ask legal and procurement to confirm that every hosted model contract contains these termination rights, audit and access terms, and data location commitments before the provider is added to the gateway.
Notice to the supervisor for planned arrangements
Article 28(3) requires timely notice to the competent authority of planned contractual arrangements on ICT services supporting critical or important functions. A gateway that adds a new model endpoint in a day can create an arrangement faster than the governance process can notice it. Set a rule that no provider is enabled in production until it is recorded in the register of information and, where the function is critical or important, the planned arrangement has been notified.
Register of information beyond gateway logs
Article 28(3) requires a register of information covering all contractual arrangements for ICT services, reported yearly for new arrangements and provided to the authority on request. Gateway configuration can list providers and models, but the register also needs contract references, service descriptions, locations and subcontracting details that the gateway does not hold. Confirm which fields come from the gateway export, which come from contract management, and who reconciles the two before each reporting cycle.
A tested exit strategy beyond routing rules
Article 28(8) requires documented and tested exit strategies and transition plans for services that support critical or important functions. A fallback rule that sends traffic to a second hosted provider is not an exit plan if that provider depends on the same data terms or is not ready to take the workload. Test whether self-hosted models on vLLM or Ollama can serve the affected use cases at acceptable quality, document the transition steps, and record the date and result of each test.
No certification scheme under DORA
DORA sets no certification scheme for compliance, and competent authorities supervise using their Article 50 powers. Vendors may offer badges or reports that look like assurance, but none of them is a DORA certificate. The only authority-issued attestation the Regulation provides for concerns a threat-led penetration test under Article 26(7). Ask any partner or gateway vendor to explain which evidence they produce for supervisors and which evidence is internal, and do not present vendor claims as DORA approval.
What do buyers ask before a DORA-focused AI gateway project?
How much does a self-hosted AI gateway project cost?
The software is open source, so the cost sits in people, infrastructure and licenses. The main components are the gateway itself (LiteLLM or APISIX), a PostgreSQL database for virtual keys and spend records, Redis once more than one gateway instance runs, the compute the gateway runs on, and the work to integrate it with identity, logging and the applications that call it. Recurring costs include infrastructure, upgrades, monitoring and support. The biggest variable is licensing: LiteLLM's SSO is free for up to 5 users from v1.76.0, and Enterprise features such as SCIM, audit logs and team-level guardrails need an Enterprise license. Ask any quote to split one-time build work from recurring operations, and to list each license it assumes.
Which parts of the gateway project need an enterprise license?
Licensing depends on which features you need, so confirm each one before you commit. According to LiteLLM's enterprise documentation, Enterprise adds SSO and SCIM beyond the free limit, JWT authentication with identity provider tokens, audit logs, organization and team RBAC, IP allowlisting, secret manager integrations and team-level guardrails and logging. Ask the implementation partner to map each control in your DORA evidence plan to either the open-source feature or an Enterprise feature, and to state the license cost in the written scope. Apache APISIX is Apache-2.0 licensed, but any commercial add-on from another vendor, such as API7 Enterprise, is a separate decision with its own terms.
How long does a DORA-focused gateway rollout take?
The timeline depends on scope, not on the software, which installs quickly. The pace is set by the number of model providers to register, the identity integration, the approval process for providers and routing rules, the logging and retention design, and how many internal applications must be moved behind the gateway. Testing the exit path to a self-hosted model also takes time. Most engagements are fixed-scope, so the timeline is agreed in writing before work starts. Ask for a phased plan with a first milestone that delivers a complete provider inventory, and ask which dependencies sit with your teams, such as change approvals and access to your identity provider.
What does our team have to provide during the project?
Your team owns the decisions and access, and the partner does the build. Expect to provide the list of applications and teams that call model APIs, an accountable owner for each approved provider, identity provider access for SSO, the network and cloud accounts or data center capacity where the gateway runs, secret manager access for keys such as LITELLM_MASTER_KEY, and the data classification rules that decide which provider may receive which data. You will also need to name the people who approve provider changes and who receive gateway alerts. Ask the partner to list these inputs in the quote with the person or role responsible for each.
Can our own team run the gateway after the project ends?
Yes, if your team has the skills and the runbooks to operate it, and the choice is yours. The stack is open source and runs in your own infrastructure, so you hold the code and the configuration. Operating it requires Kubernetes or equivalent container operations, PostgreSQL and Redis administration, certificate and secret handling, upgrade testing and on-call coverage. LiteLLM supports the four most recent stable minor versions, so upgrades cannot wait indefinitely. Ask the partner for documented runbooks, a handover plan, and optional retainer support for upgrades and testing, and ask which operational tasks would remain with your staff.
How does the gateway support our register of information?
The gateway does not write the register, but it produces the inventory data the register needs. DORA Article 28(3) requires a register of information on all contractual arrangements for ICT services, provided on request and reported yearly for new arrangements. Each model provider reached through the gateway should appear with its contract, service and the applications that use it. The gateway can record that list from its configuration and call logs, so the register reflects what actually runs. Ask the partner how the provider inventory is exported, how it is reconciled with your contract records, and who owns the register itself, since that remains your entity's responsibility.
Does DORA allow us to use a hosted LLM provider through a gateway?
DORA does not forbid hosted providers, but it requires you to manage them as ICT third-party services. Article 28(1) keeps the financial entity fully responsible for compliance at all times. Article 28(7) requires contracts to be terminable in defined circumstances, and Article 28(8) requires documented, tested exit strategies for services supporting critical or important functions. A gateway helps by enforcing an approved provider list and keeping a tested route to self-hosted models. Confirm with your legal and compliance teams how each provider contract meets these articles, because DeployOpen prepares controls and evidence but does not give legal opinions or issue attestations.
Primary sources for DORA Article 28 and the gateway projects
Start with the regulation itself. Article 28 of DORA, Regulation (EU) 2022/2554, is the text that defines your duties for ICT third-party risk, and it is the reference your register of information, contract terms and exit plan must answer to. Read the Article 28 text on EUR-Lex before scoping the gateway, and map each paragraph to a named owner in your ICT risk management framework. Pay particular attention to Article 28(3), which governs the register, and Article 28(8), which requires documented, tested exit strategies for services supporting critical or important functions.
Next, read the European Supervisory Authorities' answer on yearly reporting. The EBA Q&A 2025_7309 on the annual report on new arrangements clarifies how new ICT service arrangements are reported, which is the step where a newly approved model provider becomes a reportable arrangement. Confirm with your supervisor how the Q&A applies to model APIs that are called through a gateway, and whether a provider behind a shared gateway endpoint is recorded as a separate arrangement.
For the technical build, use the vendor documentation as the second source of truth. The LiteLLM production best practices set out the PostgreSQL and Redis dependencies, and the LiteLLM virtual keys documentation explains how keys can be restricted to specific models, which is how approved-provider lists are enforced. If you choose Apache APISIX as the gateway, the APISIX deployment modes documentation determines where configuration lives and who can change it. Confirm in writing which features, such as SSO and audit logs, sit under a LiteLLM Enterprise license before you cost the design.
How DeployOpen works on this
DeployOpen plans, implements, documents and supports self-hosted open-source platforms inside the client's own infrastructure, so the gateway, its logs and its exit path sit under the bank's or insurer's control. We agree scope, deliverables and timelines in writing before work starts, and we prepare the evidence a DORA assessment will ask for without issuing any audit report or attestation ourselves.
How we scope
- Inventory every model provider, endpoint and application that calls an LLM API today, including shadow usage.
- Map each provider to the ICT service entries your register of information needs, with the contracting entity and function.
- Agree in writing which providers are approved, which stay on self-hosted models, and the exit conditions for each.
How we build
- Deploy LiteLLM or Apache APISIX in your cloud accounts or data centers, backed by PostgreSQL and, for multiple instances, Redis.
- Configure provider allowlists, virtual keys per team, budgets and model restrictions so unapproved endpoints cannot be reached.
- Test failover to a self-hosted model and document the tested exit path, including the switch steps and who runs them.
How we hand over
- Deliver the provider inventory, configuration records, routing rules and test results as documents your third-party risk team can review.
- Document log capture, retention settings and access controls so they can be mapped to your ICT risk and audit evidence.
- Train operators on key rotation, provider onboarding and exit runbooks, and confirm which enterprise features your license must cover.
How we support
- Offer ongoing support, upgrades and continuous testing as a retainer after handover.
- Review new provider requests against the approved list and update the register inputs when arrangements change.
- Re-run exit and failover tests on a schedule you set, and record the results for supervisory questions.
