OpenAI API alternative when you need zero data retention

Why OpenAI zero data retention may not be enough for your OpenAI API alternative

A security or compliance lead who must guarantee that prompts and completions are never stored by a third party should not treat OpenAI's Zero Data Retention as a switch to flip. Zero Data Retention and Modified Abuse Monitoring need prior approval from OpenAI, and they do not cover every endpoint, so some traffic may fall outside the approved terms. Serving open-weight models yourself is the simpler guarantee, because no third party receives the prompt at all and the claim rests on your own network controls and logs. For most teams the choice comes down to vLLM for production traffic and Ollama for small teams and developer workstations. This page covers what OpenAI retains by default, what approval changes, the endpoint gaps to check, and what self-hosting requires you to own. Cost, migration and deprecation planning are covered on sibling pages.

Start with an inventory of every application, SDK call, plugin and feature that sends data to the OpenAI API, then check each one against the approved scope in writing. Self-hosting moves the guarantee onto your side, so the serving layer needs hardening from the start. The vLLM --api-key option does not protect /metrics or /health, and the vLLM security documentation recommends a reverse proxy that allowlists only the endpoints meant for users. Ollama binds to 127.0.0.1 on port 11434 by default and its local API has no authentication, so any change to OLLAMA_HOST needs a proxy that authenticates callers. Set OLLAMA_NO_CLOUD=1 on servers that must never send requests to Ollama's cloud.

Ask the OpenAI account team to confirm, in writing, which endpoints and features the approval covers for your organization, and keep that answer with your data flow diagram. Then confirm with your legal and privacy team whether your own gateway logs, traces and prompt stores count as retention, since those systems will hold the same content. Ollama states that when it runs locally it does not see your prompts, but it serves one request at a time per model by default (OLLAMA_NUM_PARALLEL defaults to 1), which suits small teams more than shared production traffic. Choose vLLM when many users or services call the model, and plan for an operator to own GPU capacity, patching and the endpoint proxy.

What a zero retention requirement actually asks of your stack

A zero retention requirement is less about one setting than about where prompts and outputs travel and who can store them. Each card below is a control your architecture must answer for, from the model endpoint to the logs and traces that quietly capture the same content.

No third-party storage of prompts

Prompts and outputs must not be written to any system outside your control, including the model provider's stores. With a self-hosted model, the inference engine runs in your cloud account or data center, so this requirement becomes a question about your own storage. Confirm that the serving host, its disks and any caches are inside your boundary and included in your retention schedule.

A documented, complete data flow

Map every hop a request takes: application, gateway, model server, and any fallback route. A hidden fallback to a hosted API is the most common way a zero retention claim breaks, so every endpoint in the diagram must be one you configured. Ask each vendor for the exact outbound destinations its software calls at runtime.

Control over prompt-bearing logs and traces

Gateways and observability tools often record full request bodies by default. Langfuse and LangSmith store prompts in the platform they run in, so unless you self-host them inside your infrastructure, traces leave your boundary. LangSmith's self-hosted option requires an Enterprise license, and Langfuse's advanced features need an enterprise key. Decide what is logged, redact or disable body capture, and set retention on the trace store.

Gateway control plane placement

Some gateways split into a data plane you run and a control plane the vendor runs. Portkey's hybrid model keeps the gateway in your VPC but calls out over HTTPS to its own control plane, so prompt metadata and configuration may leave your network. Confirm whether the gateway you choose can run fully inside your environment, including its prompt management and logging features.

Telemetry and endpoint exposure

Inference servers expose operational endpoints beyond the chat API. vLLM's --api-key option does not protect every route, and its docs recommend a reverse proxy that allowlists only user-facing endpoints. Ollama's local API has no authentication by default and binds to localhost. Check which routes are reachable, which metrics include request detail, and whether any cloud features can be disabled.

Evidence that retention is off everywhere

Retention claims need records, not assurances. Collect configuration exports, network policy and egress rules, storage bucket lifecycle settings and gateway logging settings, then test them with known canary prompts and check the stores afterward. An independent auditor reviews this evidence; DeployOpen can prepare it, but it does not issue the attestation.

Which self-hosted option fits which serving situation

For shared production traffic, vLLM is the serving engine most teams evaluate. Its vllm serve command exposes OpenAI-compatible endpoints, including /v1/chat/completions, /v1/completions, /v1/responses and /v1/embeddings, so applications written for the OpenAI API can change their base URL instead of their code. vLLM has no paid edition, so the cost is the accelerator capacity, the surrounding infrastructure, implementation and the people who operate it.

API compatibility: Applications that call the OpenAI chat or responses endpoints need little change, but confirm that each endpoint and parameter your code uses is supported by the vLLM version you pin.
Authentication caveat: The --api-key option protects only the /v1, /v2, /inference and /cohere paths. It does not protect /metrics, /health, /pause, /resume, /abort_requests or /invocations, and the docs warn against relying on it alone.
Reverse proxy requirement: The vLLM security docs recommend a reverse proxy that allowlists only the endpoints meant for users, so administrative and metrics paths stay unreachable from application networks.
Multi-node traffic: Communication between nodes is insecure by default, with no authorization or encryption, so multi-node deployments need an isolated network.
Gaps: vLLM does not provide per-user SSO, budgets or audit trails. Those controls belong in a gateway or in your identity platform.

Before committing, have your security team test the exposed paths against the allowlist and confirm who is on call for GPU failures.

Ollama suits developers and small teams who want models running quickly on a workstation or a single server. It is MIT licensed, runs on macOS, Windows, Linux and Docker, and exposes a REST API on port 11434 with OpenAI compatibility plus Python and JavaScript SDKs. It is a runtime for individual use, so its defaults favor a single user on a trusted machine.

Local default: Ollama binds to 127.0.0.1 on port 11434, and the local API at localhost:11434 requires no authentication.
Exposing the server: Changing the bind address with OLLAMA_HOST makes the API reachable by other hosts. Put a proxy such as Nginx with authentication in front of it before doing so.
Concurrency limits: OLLAMA_NUM_PARALLEL defaults to 1 request per model, so shared use slows quickly. OLLAMA_MAX_QUEUE defaults to 512 queued requests.
Cloud features: Local inference does not send your prompts to Ollama, but cloud models use ollama.com/api with a Bearer API key. Set OLLAMA_NO_CLOUD=1 or the equivalent server configuration to turn these features off when you need a strict boundary.
Gaps: Ollama does not offer enterprise SSO, per-user budgets or the audit controls that its paid Enterprise plan adds. Those are vendor commercial features, so confirm what the open-source build provides.

Check the cloud settings on every workstation and the proxy configuration before any shared use.

How each open-source option meets the zero retention constraint

This table compares vLLM, Ollama and the LiteLLM gateway, with Langfuse or Opik only as self-hosted tracing, against the questions a zero retention review asks. Read each row as a fact about the software, then confirm the logging defaults in your own environment, because a retention guarantee depends on your configuration as much as on the product.

Requirement	vLLM	Ollama	LiteLLM gateway
License	Apache-2.0 according to its GitHub repository, with no paid edition	MIT licensed	MIT except content under its enterprise directory, which has its own license
Where prompts are processed	On the GPU hosts you run, inside your network or cloud account	On the machine running Ollama; Ollama states local runs do not see your prompts or data	In the gateway process you host, which forwards requests to the backend you configure
OpenAI-compatible endpoints	Yes: /v1/chat/completions, /v1/completions, /v1/responses and /v1/embeddings	Yes: REST API on port 11434 with OpenAI compatibility	Yes: one OpenAI-compatible endpoint across 100+ providers
Authentication	--api-key covers /v1 paths only; docs advise a reverse proxy allowlist	Local API at localhost:11434 has no authentication; put a proxy in front	Virtual keys issued by the gateway, plus budgets and rate limits
Metrics and logging	Prometheus metrics at /metrics; confirm what request content, if any, is written to logs	Confirm log settings in the server configuration; OLLAMA_NO_CLOUD=1 disables cloud features	Logs to Langfuse, Datadog, OpenTelemetry and Prometheus; choose which callbacks run, since they can store prompts
Cloud calls to a third party	None required by the server itself; confirm your deployment tooling	Cloud models call ollama.com; set OLLAMA_NO_CLOUD=1 to turn this off	Calls only to providers you configure, so disable any fallback to a hosted API you do not want
Main trade-off	Strongest fit for multi-user production, but you own GPU sizing and multi-node network isolation	Simple for single users and small teams; default concurrency is limited, so it suits lighter traffic	Adds a component to secure and operate, but centralizes keys, logging and routing controls

How to deploy a zero retention OpenAI replacement step by step

This sequence moves OpenAI API traffic onto self-hosted open-weight models so that prompts are processed only inside your network, with each control verified before the next one starts.

Inventory every endpoint and SDK call
Engineering leads list every application, service, script and SDK that calls the OpenAI API, including the endpoints used (chat completions, responses, embeddings, audio, files and batch), any Assistants or vector store features, and any background jobs. Each item is tagged with the data it sends. The output is a written inventory that shows which calls need a replacement and which can be retired. Done when every caller has an owner and every endpoint has a mapped vLLM or gateway equivalent, or a documented reason for removal.
Choose a model and size GPU capacity
The team selects an open-weight model against the tasks in the inventory, then measures memory and throughput on representative prompts and expected concurrency. Infrastructure owners confirm which accelerators are available, since vLLM runs on NVIDIA CUDA, AMD ROCm, Intel XPU and several CPU architectures. Cost depends mainly on GPU capacity, so ask hardware vendors and cloud providers for quotes against the measured load. Done when a load test on the chosen model meets the latency and quality targets written in the scope.
Deploy vLLM on Kubernetes or a single host
Platform engineers start the model with vllm serve on a single host for a pilot, or deploy it on Kubernetes using native manifests or Helm, with the model weights stored on an internal volume. Multi-node setups need the tensor and pipeline parallelism settings reviewed against the hardware layout. The model cache and container images are pulled from internal sources so nothing is fetched from a third party during runtime. Done when a test request to /v1/chat/completions returns a response from the internal address and the pods or service restart cleanly.
Expose only user endpoints through a reverse proxy
Security engineers place a reverse proxy in front of vLLM and allowlist only the /v1 paths that applications use. The vLLM security documentation notes that --api-key does not protect /invocations, /pause, /resume, /abort_requests, /metrics or /health, so those paths must never be reachable from user networks. Metrics are scraped from inside the cluster instead. Done when a request to a blocked path from the user network fails and the allowlisted calls succeed with a valid key.
Protect the API key and isolate internal nodes
Platform engineers set the --api-key option on vLLM, store the key in the organization's secrets manager, and rotate it on a defined schedule. For multi-node deployments, the vLLM security documentation states that node-to-node traffic is unauthenticated and unencrypted by default, so the nodes are placed on an isolated network with no route from application or user subnets. Done when a node port is unreachable from outside the cluster network and the key is absent from configuration files and logs.
Route applications through a self-hosted gateway
Teams deploy a self-hosted gateway such as LiteLLM, which presents one OpenAI-compatible endpoint, and point application base URLs and keys at it. Virtual keys, logging and rate limits are configured at the gateway, and the gateway's logging destination is set to an internal store. Applications are changed in configuration rather than code where possible. Done when each application in the inventory sends traffic through the gateway and the gateway log shows requests served by the internal model endpoint.
Disable OpenAI fallbacks and Ollama cloud features
Gateway owners remove every OpenAI provider entry and fallback route, and remove stored OpenAI keys from applications and the gateway. If Ollama is part of the stack, the team sets OLLAMA_NO_CLOUD=1 or the equivalent server configuration entry so cloud features stay off. Done when the gateway configuration contains no external provider routes and a test with the OpenAI host blocked produces no errors in the application.
Verify no outbound calls to third-party inference
Network engineers enforce egress rules that allow the inference servers and gateway to reach only internal services, then capture traffic during a full test run of the inventory. Security staff review the captured destinations and confirm that no prompt or response leaves the network to a third-party inference or logging host. Ask the team to document the results and the egress rules as evidence for the zero retention position. Done when a signed test report lists the destinations observed and none is an external inference service.

What changes for applications and how to keep OpenAI out of the path

Applications built on the OpenAI SDK can often be pointed at a vLLM server by changing the base URL, because vllm serve exposes OpenAI-compatible endpoints such as /v1/chat/completions and /v1/responses. The compatibility of the request shape does not guarantee the same behavior. Output formatting, tool calling, refusal patterns, token limits and handling of system messages can all differ between a hosted OpenAI model and an open-weight model, even when the call succeeds. Treat the switch as a model change: rerun every prompt that matters against the new endpoint and record where outputs, parsed JSON, function arguments or downstream decisions move. Do not assume the earlier behavior carries over.

The guarantee depends on removing OpenAI from the configuration, not just from the default path. Search environment files, secrets stores, Helm values, CI variables and SDK client initializations for OPENAI_API_KEY, OPENAI_BASE_URL and any hard-coded api.openai.com host, and delete them from workloads that should never reach the hosted API. A fallback to OpenAI is the most common way the guarantee breaks: a gateway retry rule, a try-catch that reinstantiates the OpenAI client, or a secondary provider list in a routing layer can send prompts to a third party during an outage. If a fallback is kept for availability, it is a separate decision that ends the zero retention claim for the traffic it touches. Confirm with the gateway configuration that no provider entry for OpenAI remains active, and that egress rules block the OpenAI hosts from application networks.

Build the proof into your own test suite. Confirm that requests fail closed when the vLLM endpoint is unavailable, rather than silently switching providers. Run network-level tests from the application subnet to confirm that outbound connections to OpenAI are refused. Replay a representative set of production prompts with tool calls and structured outputs, and compare results against your acceptance criteria. Check the vLLM --api-key behavior and the reverse proxy allowlist described in the security documentation, because unprotected endpoints such as /metrics and /health change what an outside party can observe. Keep the test evidence with the configuration baseline so an auditor can see both the design and its verification.

What auditors and regulators ask to see for retention claims

A retention claim is only as strong as the evidence behind it. Auditors and regulators tend to test the same few artifacts: where data flows, which endpoints are reachable, who can call them, how long logs live, and which model versions were in use at each point in time. DeployOpen prepares this evidence for your environment, but independent auditors and certification bodies are the ones who issue any opinion.

Network diagrams with trust boundaries

Auditors want a diagram that shows every hop a prompt takes, from the application through the gateway to the vLLM server and any tracing backend, with each trust boundary marked. The diagram should show which segments sit inside your VPC or data center and whether any segment leaves it. Check that the diagram matches the running network, not the design document from months earlier, and that it shows where isolated node-to-node traffic is placed in multi-node deployments.

Configuration exports with no OpenAI endpoints

Export the live routing configuration of your gateway and application settings, then search them for any api.openai.com base URL, fallback route or stored OpenAI key. This is the evidence that supports a claim that prompts never leave your environment. Record the export date and the commit or change ticket it came from, so the exported state can be tied to a specific point in time.

Authentication on the vLLM and gateway layers

Show how each endpoint is protected. The vLLM --api-key option covers the /v1 routes but not /metrics, /health or several control routes, so the evidence should show a reverse proxy that allowlists only the endpoints meant for users. Include gateway virtual key configuration and who can issue or revoke keys. Confirm the exact list of exposed paths against the current vLLM release you run.

Log and trace retention settings

Retention claims often fail on logs, not on the model. Gateway request logs, tracing data, metrics and reverse proxy access logs can each store prompt or response content. Document the retention period set for each store, whether prompt bodies are logged at all, how deletion is performed and who can read the data. Verify the settings in the running systems, since a default in a tool may differ from what your team configured.

Change records for model and software versions

Auditors ask which model weights and which serving software handled a given request on a given date. Keep a change record for each model version, vLLM release and gateway release, including the approval, the deployment time and the rollback plan. vLLM ships minor releases about every two weeks, so a dated record is what lets you show exactly which version was live during any period under review.

Access reviews for operators

Regulators will also ask who can read prompts, change routing or pull logs. Produce a current list of administrators for the gateway, the inference hosts and the log stores, with the review date and the reviewer. Pair it with the process that removes access when people leave, so the evidence covers people as well as systems.

Questions buyers ask before replacing OpenAI for zero data retention

Is OpenAI zero data retention enough to keep prompts out of a third party's storage?

Zero data retention reduces what OpenAI stores, but it is an arrangement you must request and have approved, and it does not automatically apply to every endpoint or feature. Read the terms that apply to your account and the specific endpoints your applications call, then record which requests are covered. If your policy requires that no third party ever stores prompts or outputs, a contractual control is weaker than an architecture where the model runs in your own environment. Confirm the scope with OpenAI in writing and with your legal and compliance team before you treat it as your control.

What does OpenAI approval involve, and what does it leave uncovered?

Approval is a process with OpenAI that results in account-level terms, so the protection depends on your agreement and on which endpoints and features are included. Expect to document your use case, your data categories and your application flows. Ask OpenAI which endpoints, tools and modalities fall outside the agreement, how abuse monitoring is handled under any modified arrangement, and how logs for your own troubleshooting are stored on your side. Your own application logs, caches, retrieval stores and vendor add-ons can still persist prompts even when the model provider does not.

How much does a self-hosted replacement cost, and which components drive it?

Cost is split across GPU or accelerator capacity, the inference server, the gateway, monitoring, implementation and the people who operate the system. The inference engine itself has no license fee: vLLM is Apache-2.0 licensed with no paid edition, and Ollama is MIT licensed. Recurring costs come from hardware or cloud GPU hours, storage, backups and staff time, while implementation is a one-time effort. Capacity planning depends on model size, concurrency and traffic patterns. Request a written scope and quote from DeployOpen that lists each component, its owner and what is one-time versus recurring. DeployOpen does not publish prices.

How long does implementation take, and what drives that timeline?

The timeline depends on the number of applications that must move, the model and hardware you choose, how many environments you need, and how much compliance evidence you must produce. Serving a model with vLLM and placing a gateway in front can be a short technical step, but production readiness includes authentication, network isolation, monitoring, testing of outputs against current prompts and a cutover plan. DeployOpen agrees timelines in the written scope before work starts, so ask for milestones tied to specific deliverables rather than a single date.

What does the work need from our team?

You will need named owners for the infrastructure, the application changes, the security review and the business sign-off on output quality. The infrastructure side needs access to a GPU environment or data center capacity, a Kubernetes or container platform if you use one, identity integration and a network design that keeps the model endpoints off public paths. Application teams need to point their OpenAI client libraries at a new base URL and run regression tests against their prompts. Ask the implementation partner to list these inputs in the scope so your team can plan its time.

Can our internal team do this alone?

Many teams can run a proof of concept internally, because vLLM and Ollama are open source and expose OpenAI-compatible endpoints. The harder parts are production hardening: vLLM's security documentation says multi-node traffic is insecure by default and that its --api-key option should not be the only control, so a reverse proxy that allowlists user-facing endpoints is needed. Other work includes GPU capacity planning, monitoring through Prometheus-format metrics, backup, upgrades and on-call coverage. Assess whether your team has these skills, then decide between building in-house, a fixed-scope engagement with DeployOpen or ongoing support under a retainer.

Does Ollama avoid this problem for smaller workloads?

Ollama runs locally and states that it does not see your prompts when run on your own hardware, and cloud features can be disabled with OLLAMA_NO_CLOUD=1. Its local API at port 11434 has no authentication by default and binds to 127.0.0.1, so you must add a proxy with authentication before exposing it to other users. Its concurrency defaults, such as one parallel request per model, suit development and light internal use more than multi-user production traffic. Use it to test, then move to vLLM when concurrency and endpoint security requirements grow.

What should we ask for in a quote for this replacement?

Ask for a written scope that names the applications in scope, the model and hardware assumptions, the gateway and logging design, the network and authentication controls, the testing approach for output quality, the backup and upgrade process, and the handover documents. Ask which items are one-time and which recur under a retainer, and what you must provide. Ask how compliance evidence is prepared, and confirm with your auditor that the evidence meets your requirements, since DeployOpen prepares controls and evidence but does not issue audit reports or certifications.

Which sources to read before you decide

Before you commit to self-hosting for zero data retention, read the vLLM online serving documentation and the vLLM security documentation together. The serving page confirms that the vllm serve process exposes /v1/chat/completions and /v1/responses, so your applications can keep their OpenAI-style client code. The security page is the one that matters for data control: it states that communication between nodes in a multi-node deployment is insecure by default, and it explains that the --api-key option does not cover every endpoint, including /metrics and /health. Plan a reverse proxy that allowlists only the paths your users need, and have your security team review that allowlist before go-live. vLLM online serving docs and the vLLM security docs are the natural starting points.

If you test on a workstation or a small internal host, read the Ollama FAQ and the Ollama API authentication documentation. The FAQ explains that Ollama binds to 127.0.0.1 on port 11434 by default, that OLLAMA_HOST changes the bind address, and that OLLAMA_NO_CLOUD=1 turns off cloud features. The authentication page makes a point that is easy to miss: the local API at localhost:11434 has no authentication, so any exposure beyond the host needs a proxy in front of it. Confirm with your own engineers which configuration keys your deployment uses and check them against the current release notes. Ollama FAQ is the document to open first.

Do not treat this page as a statement of OpenAI's terms. Zero Data Retention and Modified Abuse Monitoring approval, eligible endpoints and the conditions attached to them change, and this page does not restate them. Ask your OpenAI account team in writing which endpoints your approval covers, whether the approval applies to tools and file features you plan to use, and what retention applies to requests on any endpoint outside that scope. Keep the written answer in your evidence file next to the self-hosting design so that auditors and your legal team see the same facts.

How DeployOpen deploys open-source LLM inference under a zero retention constraint

DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks. For zero retention work, we start from where prompts and completions would be processed and stored, then build the inference path so that no third party sits in that path.

How we scope

We map every endpoint your applications call and record which ones today send prompts to a third party.
We agree in writing which models, serving engine and gateway are in scope before any build work starts.
Engagements are fixed-scope, with deliverables and timelines confirmed in writing; a quote follows from that scope.

How we build

We deploy vLLM or Ollama in your environment, with the OpenAI-compatible endpoints your applications already use.
We restrict endpoint exposure with a reverse proxy allowlist, covering the /v1 routes users need and nothing else.
We configure request logging so prompt and completion bodies are excluded or redacted, then test that the setting holds.

How we hand over

You receive architecture and data-flow diagrams showing every hop a prompt takes inside your network.
Runbooks cover model loading, storage paths, log settings, access keys and how to verify that no outbound calls occur.
We document the evidence you would need for an assessment, while independent auditors and certification bodies issue any formal reports.

How we support

Ongoing support, upgrades and continuous testing are available as retainers after handover.
Retainers can cover serving engine upgrades, model version pinning and periodic re-checks of the zero retention settings.
Ask any provider to confirm, in writing, the exact retention behavior of each component in your own environment.