Why an AI gateway is the control point for EU AI sovereignty
A self-hosted AI gateway gives an EU organization one place to enforce which model endpoints receive data, what gets logged and how traffic can move away from a provider. LiteLLM is the usual choice for model-level routing, with Apache APISIX or Kong as alternatives where the organization already runs an API layer and wants control at the network and API tier. Without a gateway, staff tools and product features call US-hosted model APIs directly, each with its own keys, its own logs and no shared list of approved endpoints. The gateway does not make a transfer lawful by itself. It is the technical point where the organization can block non-EU endpoints, record each call and switch routes, while the transfer assessment, processor contracts and supplementary measures remain separate work that the controller must complete.
In practice, the short answer on platforms is LiteLLM for model-level routing, with self-hosted vLLM or Ollama behind it so that EU-resident prompts can be served without leaving the organization's infrastructure. Routing rules in LiteLLM can restrict virtual keys to specific models, so a team handling personal data is limited to approved endpoints while other teams keep access to hosted models that the legal review has cleared. Reader actions include listing every model endpoint currently in use, including ones reached indirectly through SaaS tools, and deciding which of them are permitted for which data classes before any routing rule is written. Confirm with the chosen vendor and your own network team whether outbound traffic can be restricted to the gateway, since a gateway that other systems can bypass enforces nothing.
Plan for the risk that an EU-hosted processor is still subject to third-country law. The EDPB's guidance (see EDPB Guidelines 05/2021) treats complying with a third-country authority request as a transfer, so the gateway's endpoint list should record the legal owner and jurisdiction of each provider, not only its region. Logs held by the gateway are also personal data and must sit in storage the organization controls, in a PostgreSQL database it operates. Ask any implementation partner to show how a provider can be removed from the allowlist in one configuration change and how the same workload is re-pointed to a self-hosted model, and test that exit path before it is needed.
Which EU rules set the gateway requirements
This table maps each EU rule that bears on model traffic to the gateway control it implies and the evidence an assessor will ask for. Read each row as a design input: the rule on the left shapes the configuration on the right, and the evidence column shows what to keep. The main takeaway is that transfer and logging rules translate into concrete gateway settings, while the legal status of transfer mechanisms still needs review by counsel.
| Rule | What it means for the gateway build | Evidence to keep |
|---|---|---|
| GDPR Article 44 and Chapter V (transfers) | Every model endpoint that receives personal data must be classified by location and legal basis. Build an endpoint allowlist that blocks non-approved regions by default. | Approved endpoint register with region tags and the transfer mechanism for each external provider |
| EDPB Guidelines 05/2021 (remote access and cloud storage) | Support staff viewing data from outside the EEA, and storage outside the EEA, both count as transfers. Restrict admin and support access to the gateway and its log store by location. | Access records for the gateway admin UI and database, with location of administrators |
| Schrems II and the Data Privacy Framework status | Do not treat a US provider as adequate because it is listed. The Data Privacy Framework adequacy decision is under appeal (Case C-703/25 P), so route sensitive traffic to EU-hosted or self-hosted models and keep a switch path ready. | Documented transfer assessment per provider, reviewed when the appeal is decided |
| CLOUD Act exposure (18 U.S.C. 2713) | A US-headquartered provider can be required to disclose data it controls wherever it is stored. Model these providers as a disclosure risk in the gateway policy, not only as a location question. | Provider contract terms on government requests and the gateway policy that excludes restricted data classes |
| GDPR Article 48 (third-country orders) | Foreign court or administrative disclosure orders are enforceable only under an international agreement in force with the EU or a Member State. Keep this in mind when choosing hosted providers for personal data. | Legal review note on which provider requests are recognized, held with the processor record |
| EU Data Act Article 32 (non-personal data) | Providers of data processing services must take measures against unlawful third-country access to non-personal data. Log which provider handled each request so the exposure can be traced. | Provider measures documented in contracts and the per-request routing log |
| AI Act Articles 12 and 19 (automatic logging) | High-risk systems must technically allow automatic event logging over their lifetime, and providers keep logs for at least six months unless other law applies. The gateway log should record model, endpoint, team and timestamp for each call, with retention set to meet these periods. | Retention settings, log schema and a tested export of logs for a given system and period |
| Data Act switching rights (Articles 25, 29 and 50) | Customers may move to another provider or on-premises infrastructure, with switching charges banned from 12 January 2027. Keep the gateway config provider-neutral so a model alias can point to a self-hosted vLLM endpoint. | Tested failover from the hosted provider to a self-hosted model, with timed runbook results |
Where a US provider still creates transfer exposure
Using a US provider does not automatically create an international transfer, but several common setups do. Under the EDPB's *Guidelines 05/2021*, a transfer includes storing personal data in a cloud outside the EEA and remote access from a third country, even when the remote party only views data on a screen for support or administration. A gateway that is hosted in an EU region but whose logs, spend records or prompt stores sit in a US-operated SaaS control plane, or a vendor engineer who can open a prompt log from outside the EEA, can therefore fall inside Chapter V. Map every place a prompt, response, key or log record is written, and every role that can read them, before you approve a provider.
The guidelines also draw a line that matters for procurement. Using an EU processor that is subject to third-country law with extraterritorial reach is not, by itself, a transfer. Complying with a third-country authority's request for data, however, would be one. The *EDPB Recommendations 01/2020* explain why this exposure is hard to close with technical measures alone: where a provider needs access to data in the clear and the destination's public authority powers go beyond what is necessary and proportionate, the EDPB cannot envision an effective technical supplementary measure. Encryption with keys held by the provider does not resolve that case, so the decision rests on whether the provider can actually see the content.
Assess each provider's exposure before any contract is signed, and record the result in the gateway's provider inventory. Ask the provider where it stores data and logs, which entity controls access, whether support staff outside the EEA can reach customer data, and how it would respond to a government request. Then confirm the answers and the contract terms with your data protection officer and external counsel, including the standard contractual clauses and any transfer impact assessment. Where the exposure cannot be closed, the practical control is to keep regulated prompts on a self-hosted or EU-operated model behind the gateway, and to document that routing rule as part of the design.
Which open-source platforms fit EU sovereignty routing
EU sovereignty routing depends on two layers: a gateway that decides where each request may go and records what happened, and the model endpoints that receive it. The cards below cover the gateway candidates, the self-hosted model servers that keep traffic inside the organization, and the logging layer that produces records. Each card names the license or edition check to run before a decision.
LiteLLM as the routing gateway
LiteLLM is MIT licensed outside its enterprise directory and can route one OpenAI-format endpoint to self-hosted and approved external models. Its router supports priority ordering for failover, and virtual keys can restrict each key to specific models, which is the main control for blocking non-EU endpoints. Shared state for keys, budgets and rate limits needs PostgreSQL, and Redis is required once more than one proxy instance runs. Check the enterprise boundary: SSO for more than 5 users and SCIM need an Enterprise license, so confirm which sovereignty controls you need before design.
Apache APISIX as API-layer policy
APISIX is Apache-2.0 licensed and offers AI gateway features alongside its general API gateway. It suits organizations that already run APISIX for internal APIs and want one policy point for authentication, rate limiting and routing of model traffic. Confirm that the AI plugins you need, such as provider routing and logging, exist in the version you deploy, because model-aware controls are newer than the core gateway. Keep the model allowlist in one place so it is reviewed with other API policy.
Kong Gateway for shared API policy
Kong Gateway is Apache-2.0 licensed and is commonly used as an enterprise API layer with plugins for authentication, rate limits and logging. For sovereignty routing, it can enforce which upstream hosts a service may call, so model endpoints outside approved regions are never reachable from application networks. Confirm which AI-specific plugins and any licensed features your organization needs, since the open-source edition and paid editions differ. Check how logs leave the cluster before relying on them as EU AI Act records.
vLLM for self-hosted models
vLLM is an Apache-2.0 inference engine that serves open-weight models through OpenAI-compatible endpoints, so the gateway can route sensitive traffic to a model that never leaves your infrastructure. It has no paid edition, so cost comes from GPU capacity, surrounding infrastructure and the people who operate it. Its --api-key option does not cover every endpoint, so place a reverse proxy in front that allowlists only user-facing routes. Multi-node deployments are insecure by default between nodes and need an isolated network.
Ollama for local models
Ollama is MIT licensed and runs open-weight models locally, which suits departments, pilot workloads and developer machines that must keep prompts on-premises. It is simpler to operate than a GPU serving cluster, but it is designed for smaller deployments, so confirm throughput and concurrency against the workload before using it as a shared production endpoint. Register each Ollama instance in the gateway as an approved model, and confirm that its network exposure is limited to the gateway rather than open to application teams.
Langfuse for self-hosted call records
Langfuse is MIT licensed outside its ee directories and stores traces, prompts, outputs and costs in your own storage. Pairing it with the gateway gives you a second record of which model received which request and where the answer went, which supports GDPR transfer reviews and EU AI Act log questions. Confirm which features sit in the ee directories before deployment, and set retention and access controls on the trace store, since prompts often contain personal data. Arize Phoenix is a comparable option, but its Elastic License 2.0 bars offering it to third parties as a hosted or managed service, so review that terms before any shared internal service is offered to subsidiaries or partners.
Choosing EU-hosted, self-hosted or mixed model endpoints
Open-weight models run on vLLM inside your own EU data center or EU cloud accounts, behind the gateway. Prompts, completions and gateway logs stay in infrastructure you control, and no model provider receives the data. The trade-off is that your team owns capacity planning, GPU procurement or allocation, patching, model updates and on-call response.
- Data location: inference, caches and logs sit where you place them, so the transfer analysis reduces to your own cloud or hosting contracts.
- Provider exposure: the main third-party relationships are the hardware vendor and infrastructure host. Confirm whether either can access systems or data for support.
- Operating burden: vLLM ships minor releases about every two weeks, so a patch and regression cycle is a recurring task.
- Network controls: vLLM's security documentation says node-to-node traffic in multi-node deployments is insecure by default, and its
--api-keyoption does not cover every endpoint, so a reverse proxy should expose only the paths users need. - Cost driver: vLLM has no paid edition, so the spend is accelerator capacity, surrounding infrastructure and the people who run it.
Confirm in writing which hosting location, operator and staff access model apply to the GPU nodes and the storage behind the gateway.
Here the model runs on a provider's infrastructure in the EU, and your gateway routes requests to that endpoint under a contract. Operating effort is lower because the provider runs the model, but the provider becomes a processor and its own access practices matter. A contractual EU region does not by itself answer whether a non-EU parent company, support staff or subprocessor can access data from outside the EEA.
- Data location: inference and storage happen in the region named in the contract, which must be confirmed for every component including logs, fine-tuning data and backups.
- Provider exposure: check whether the provider or its parent is subject to third-country law with extraterritorial reach, such as the US CLOUD Act, and what the provider commits to when such a request arrives.
- Remote access: EDPB Guidelines 05/2021 treat remote access from a third country, even only displaying data for support, as a transfer. Ask where support and administration access originates.
- Operating burden: lower on infrastructure, but you still own gateway configuration, key management and contract monitoring.
- Exit path: the Data Act gives customers switching rights for data processing services, but your gateway should still keep a second route so a provider change is a configuration change.
Confirm in writing the processing region, the list of subprocessors, the support access locations and the provider's handling of government access requests.
The mixed pattern classifies each request by data sensitivity and routes it to a destination that matches the classification. Regulated prompts, such as those containing personal data or client records, go to self-hosted vLLM models in your own EU infrastructure. Less sensitive or public-content requests may go to an EU-hosted provider endpoint where that gives better model quality or capacity. This pattern takes the most design work, because the classification rules and the routing rules must agree.
- Classification: define which applications, teams or data categories are regulated. The gateway can enforce this only when the caller supplies the classification or the rules are tied to the key or team.
- Routing rules: LiteLLM's router supports priority ordering, cooldowns and per-model key restrictions, which can keep regulated keys on self-hosted models only.
- Failover guardrails: a fallback from self-hosted to a provider endpoint can move regulated data out of scope. Disable cross-destination fallback for regulated keys unless the legal review covers it.
- Logging: record which destination handled each call so you can show which model saw which data.
- Operating burden: both operating models run in parallel, so you carry the costs of each.
Confirm in writing that regulated routes never fall back to a non-EU endpoint and that the routing configuration is reviewed whenever a new model or provider is added.
How the rollout moves from inventory to enforced routing
The sequence turns scattered model calls into one gateway that only sends data to approved endpoints, logs every call and has been tested against an exit to a self-hosted model.
Inventory every model call and provider
Your team and DeployOpen review each application, script, SaaS feature and developer tool that calls a model, including calls made through vendor features you may not think of as AI. For each call, record the application owner, the provider or hosting location, the endpoint domain, the data it sends and whether the traffic leaves the EU. The output is a register of model endpoints with their owners and data flows. The step is done when every known caller is listed and the owners have confirmed the entries, and the open questions are assigned to named people.
Define the allowlist and data classes
Your data protection, security and legal teams agree which data classes exist (for example, public, internal, personal and special category) and which endpoints may receive each class. Each approved endpoint is recorded with its hosting location, the contract terms and the transfer analysis your counsel has completed. The output is a signed-off policy that maps each data class to permitted endpoints, along with a list of endpoints that are blocked. The step is done when every approved endpoint has a documented basis and the policy has an owner who will review it on a set schedule.
Deploy the gateway with PostgreSQL and Redis
DeployOpen deploys the chosen gateway, such as LiteLLM or Apache APISIX, inside your cloud account, data center or air-gapped network, with PostgreSQL for keys, budgets and spend records and Redis for shared state once more than one instance runs. Secrets such as the master key are held in your secret manager, and only the approved endpoints are configured as model routes. The output is a running, highly available gateway in a non-production environment with documented configuration. The step is done when the team can send test requests through the gateway and the state survives the loss of a single instance.
Enforce endpoint blocking and logging
Network egress rules and gateway configuration are set so applications can reach model providers only through the gateway, and the gateway accepts only routes on the allowlist. Requests to blocked endpoints fail closed, and every model call is logged with the caller, key, model, endpoint and timestamp, with the logging scope set for each data class. The output is a tested enforcement state where direct calls from applications fail and gateway logs show the expected records. The step is done when a deliberate attempt to reach a non-EU endpoint is blocked and recorded.
Validate retention and access controls
Your records and compliance owners confirm how long prompt and response logs are kept, where they are stored and who can read them, and DeployOpen configures the admin interface with your identity provider and role-based access. Retention periods are set against the legal requirements your counsel identifies, including the minimum log retention for high-risk systems under the EU AI Act where it applies. The output is a written retention schedule and an access review showing who can view, export or delete logs. The step is done when the settings match the schedule and a sample of log entries has been checked by someone outside the build team.
Run an exit test to a self-hosted model
DeployOpen and your team switch a representative, non-sensitive workload from its hosted provider to a self-hosted model, such as one served by vLLM or Ollama, by changing only the gateway route. The test checks response quality against an agreed set of prompts, latency, logging continuity and the budget and key settings. The output is a documented exit runbook with measured results and a list of application changes still needed. The step is done when the test has passed in a non-production environment and the owners have accepted the remaining gaps in writing.
What auditors and regulators ask for from the gateway
This table lists the evidence an assessor or data protection authority is likely to request from a self-hosted gateway, the component that produces each item and where it usually goes missing. Read it as a checklist to agree with your auditor before the gateway goes live. DeployOpen prepares this evidence for assessments but does not issue audit reports, attestations or certifications.
| Evidence item | Produced by | What to confirm or keep |
|---|---|---|
| Endpoint allowlist and change history | Gateway model and route configuration, stored in version control | Approved non-EU endpoints blocked by default, with a dated approval record for each change |
| Per-call log with timestamp, key, model and endpoint region | Gateway request logging, written to PostgreSQL or a log store you control | Confirm that endpoint region is captured as a field, not inferred later from the provider name |
| Log retention settings against AI Act Article 19(1) | Gateway log storage and retention jobs, plus the provider's own log export | Retention of at least six months where no other law applies, verified by a documented test restore |
| Admin UI access logs | Gateway admin interface and the identity provider behind its SSO | Who changed routes, budgets or guardrails, with the identity provider as the source of truth |
| Key and budget administration records | Virtual key and team configuration in the gateway database | Ownership of each key, revocation dates and the approver for budget increases |
| Transfer assessments for each external provider | Organization's privacy team, informed by the provider's documented data locations | Records covering the GDPR Chapter V analysis, supplementary measures and the review date |
| Provider contract and processor terms | Legal and procurement, held outside the gateway | Data processing terms, sub-processor lists and whether a third-country authority request clause exists |
| Guardrail and fallback configuration | Gateway guardrail and router settings, exported with each release | Proof that fallback never routes an EU-restricted request to a non-approved region |
What EU teams ask before building an EU-sovereign gateway
How much does an EU-sovereign AI gateway cost to build and run?
The software is not the main cost. LiteLLM and APISIX are open source, so the spend comes from infrastructure, GPU or accelerator capacity for any self-hosted models, PostgreSQL and Redis for shared state, enterprise licenses where you need features such as SSO beyond the free limit or SCIM, and the people who build and operate the platform. Some costs are one-time, such as design, integration and hardening. Others recur, including hosting, GPU time, license renewals, upgrades and support. LiteLLM's enterprise terms are the main licensing question: SSO is free for up to 5 users from v1.76.0, and an Enterprise license is needed beyond that. Ask any partner for a written quote that separates infrastructure, licenses, implementation and run costs, and state which models will be self-hosted and which will stay on external APIs.
How long does an EU gateway rollout take?
The timeline depends on scope, not on the gateway software alone. The main drivers are how many providers and models must be routed, whether self-hosted inference on vLLM or Ollama needs new hardware, how identity integration works with your SSO, how many teams need virtual keys and budgets, and how long your security and legal reviews take. Hardware lead times and procurement often set the critical path more than configuration does. DeployOpen plans fixed-scope engagements, so scope, deliverables and timelines are agreed in writing before work starts. Ask for a phased plan with a first milestone that routes one non-sensitive workload through the gateway, then adds guardrails, logging and the remaining teams.
What must our internal team provide during the project?
Your team owns decisions that only your organization can make. You need to name the data classes that may go to each model, approve the provider list, define who receives virtual keys and budgets, and supply identity groups for SSO and RBAC. You also need access to the target cloud account or data center, network segmentation for model servers, a secret manager for keys such as LITELLM_MASTER_KEY, and a named owner for on-call operations. For EU compliance, your data protection officer and legal team must review transfer assessments and processor terms. Confirm these responsibilities in the statement of work so that gaps do not appear late in the project.
Can we build an EU AI gateway in-house without a partner?
Yes, if you have platform engineers who can run PostgreSQL, Redis, Kubernetes or similar infrastructure, and security staff who can review model-serving exposure. The gateway itself is documented open-source software, so the question is capacity and continuity. Plan for the work after go-live: upgrades (LiteLLM supports its four most recent stable minor versions for open source and Enterprise deployments), provider changes, guardrail tuning, log retention, and incident response. Note that vLLM's security documentation says its --api-key option does not protect every endpoint, so a reverse proxy that allowlists user-facing paths is needed. An in-house team can do this, but it should estimate the recurring operating load before committing.
Does routing through an EU gateway make a data transfer lawful?
No. A gateway enforces where requests go, which supports your transfer analysis, but it does not create a lawful basis or replace a transfer assessment. Under GDPR Article 44, every transfer to a third country, including onward transfers, must meet Chapter V conditions. EDPB Guidelines 05/2021 also treat remote access from a third country, even only displaying data on a screen, as a transfer. Using an EU processor subject to third-country law is not itself a transfer, but complying with a third-country authority request would be, so the exposure needs assessment before you engage that processor. The gateway's block and allowlist rules, logs and per-team routing are useful evidence. Confirm the legal position with your counsel and data protection officer.
How do EU AI Act logging duties differ for providers and deployers?
Article 12(1) requires high-risk AI systems to technically allow automatic recording of events over their lifetime, and Article 19(1) requires providers to keep those logs for at least six months unless other law provides otherwise. Providers and deployers have different duties, so your gateway log design should not assume they are the same. A provider builds the system and must design logging into it. A deployer that uses the system must keep the logs under its control, which is where a self-hosted gateway that records each model call with timestamp, model, key, team and policy decision helps. Your legal team should confirm which role your organization holds for each use case and how long other law requires retention.
How do we test an exit from a US model provider?
Test the exit as an operating procedure, not a contract clause. Start by inventorying every model route and the data classes allowed on it. Then, in a non-production environment, change the routing configuration so a representative workload fails over to a self-hosted model or an EU-hosted provider, and measure whether the application still works, which prompts or features break, and how long the change takes. LiteLLM supports priority ordering for failover, cooldowns and retries, but you should confirm each provider's behavior in your own tests. Also check the Data Act switching rights in Articles 25, 29 and 50, which apply from 12 September 2025 and ban switching charges from 12 January 2027, and record the results for your exit plan.
Questions to ask an implementation partner about EU routing
Start with the controls that decide where a prompt can go. Ask the partner how they build the endpoint allowlist in the gateway configuration, which model deployments and provider base URLs are permitted, and how a request to a non-approved endpoint is blocked rather than logged and forgiven. Ask them to show the rejected-route test: a request addressed to a non-EU model should fail at the gateway, and the failure should appear in the logs. Ask who approves a change to the allowlist, how that approval is recorded, and whether a change can be made without a second reviewer. Also ask how failover is constrained. LiteLLM's router offers retries, cooldowns and priority ordering for failover, so the partner should explain how fallback groups are limited to EU-hosted or self-hosted deployments and how a fallback to a US region is prevented, not just discouraged.
Next, ask where the logs live and for how long. The partner should name the database that stores spend and request records (LiteLLM's production guide names PostgreSQL as the primary supported database), where prompt and response content is stored if it is captured at all, and which backups and replicas hold copies. Ask which LiteLLM features need an Enterprise license for this design. The LiteLLM enterprise documentation lists SSO and SCIM beyond the free limit, audit logs, organization and team RBAC, IP allowlisting, secret manager integrations and team-level guardrails and logging. Ask for the feature list in writing against the version you plan to run, since release-specific terms change. Also ask who holds the LITELLM_MASTER_KEY, where it is kept in a secret manager, and which personnel can reach the admin UI from which networks. Confirm where the admin plane runs, because a control plane hosted outside your jurisdiction can itself be a transfer.
Finally, ask what the partner tests and delivers for legal review. Request a documented exit test in which the routing configuration is switched to a self-hosted model endpoint and the application continues to work, with the time and steps recorded. For transfer assessments, ask whether they will prepare the data map of which systems receive personal data, the transfer analysis against the EDPB guidance on transfers, and the inventory of processors, and what they will not provide, since DeployOpen prepares evidence but independent counsel and your data protection officer make the legal determination. For DPA reviews, ask which evidence they hand over: configuration exports, access reviews, log samples and change records. Confirm the delivery format, the owner on each side and the acceptance criteria in the written scope before work starts.
How DeployOpen scopes and runs an EU-sovereign AI gateway
DeployOpen plans, implements, integrates, hardens, documents and supports self-hosted open-source platforms inside the client's own infrastructure. For an EU-sovereign AI gateway, that means agreeing in writing which endpoints are approved, deploying the gateway in your cloud account or data center, and handing over the routing, logging and transfer evidence your compliance team needs.
How we scope
- Inventory every model provider, endpoint and application that will call the gateway today
- Classify which data categories may reach EU-hosted, self-hosted or non-EU endpoints
- Agree the written scope, deliverables and timeline before any work starts, with no fixed prices in this page
How we build
- Deploy LiteLLM or another agreed gateway with PostgreSQL and Redis inside your infrastructure
- Configure allowlists that block non-EU endpoints by default and test that blocked calls fail
- Set up logging of model calls with retention settings agreed with your legal and compliance owners
How we hand over
- Document routing rules, virtual key ownership, budget settings and the failover path to self-hosted models
- Provide test records showing blocked endpoints, log capture and admin access controls
- Deliver a runbook for adding a provider so changes go through your approval process
How we support
- Offer ongoing support, upgrades and continuous testing as a retainer after handover
- Review gateway versions and provider changes so the approved endpoint list stays current
- Confirm with your auditors and counsel which evidence they need, since DeployOpen does not issue attestations
