Ollama migration from the OpenAI API

What this migration involves

Moving an application from the hosted OpenAI API to Ollama is mostly a change of endpoint, model and operating responsibility. Ollama runs on infrastructure you control and exposes a REST API on port 11434 with OpenAI compatibility, so many client libraries can point at a new base URL instead of being rewritten. The work also includes choosing replacement open models, proving that output quality is acceptable for each use case, and moving traffic in stages. Teams usually start this work for one of four reasons: keeping prompts and data inside their own environment, making inference cost predictable, meeting residency or air-gap requirements, or reducing dependence on a single vendor's pricing and terms.

What changes in the code is usually small: base URL, model name, authentication headers, and any assumptions about rate limits, context length or streaming behavior. What does not change is the application's business logic, prompt design, and error handling for failed calls, which still need review because a different model may respond differently to the same prompt. Confirm with your own tests which OpenAI features your application calls, such as tool calling, embeddings or structured output, and check each one against the Ollama API documentation before you commit to a cutover plan.

Security comes first. The local Ollama API at localhost:11434 does not require authentication. Ollama binds to 127.0.0.1 by default, and the OLLAMA_HOST setting changes that. Before any application traffic reaches a server that listens beyond localhost, add access control in front of it, such as a reverse proxy with authentication and TLS. Otherwise anyone who can reach the port can use the models and consume your hardware.

In short, small teams with strong ML and platform skills often make the code changes in-house. Most regulated buyers outsource the model evaluation, cutover design and hardening to a partner, because those steps need documented decisions, independent test evidence and a rollback path that the organization's risk owners can review.

What the work includes

A migration to self-hosted Ollama moves the application's model calls off a hosted API and onto models you run. The work below is what a fixed-scope engagement typically covers, and each package ends in a named deliverable.

Inventory of every OpenAI API call

Every call site is listed by endpoint and feature: chat completions, embeddings, tool or function calling, streaming, vision, batch jobs, file handling and any fine-tuned model IDs. Each entry records the request shape, response fields the code depends on, and which teams own it. Features that Ollama's OpenAI-compatible layer may not cover should be flagged early, and you should confirm each one against the Ollama version you plan to run.

Client configuration using the compatible endpoint

Where the application uses an OpenAI-compatible client, the change is usually the base URL, the API key handling and the model name. The work covers those edits, removing hard-coded hosted-service assumptions, and setting timeouts and retries for local inference. Confirm the endpoint paths and parameters your code uses against the current Ollama documentation before editing production configuration.

Replacement open model selection per use case

Each use case, such as summarization, classification, extraction, code assistance or embeddings, is matched to candidate open models with stated license terms, context length and memory needs. Embedding models are treated separately because switching them changes stored vectors and requires re-indexing. The output is a shortlist per use case, with the reasoning for each choice recorded.

Evaluation harness with fixed test set

A fixed set of representative prompts, drawn from real traffic with sensitive data removed, is run against the current hosted model and each candidate. Acceptance criteria are written before the run, covering task accuracy, format compliance, tool call validity, latency and refusal behavior. Scoring rules and reviewer roles are documented so results can be repeated after a model or Ollama update.

Gradual traffic switch with feature flag or gateway route

Traffic moves to Ollama in steps using a feature flag in the application or a route in your API gateway, starting with internal users or low-risk endpoints. Each step has a defined monitoring window and an exit condition tied to the acceptance criteria. Confirm that your gateway or flag system can split traffic per endpoint, since that determines how fine-grained the steps can be.

Rollback path and cutover runbook

A rollback path returns traffic to the hosted API by reverting the flag or route, and the runbook states who makes that call and within what window. Keep the hosted API credentials and configuration valid until the migration is closed, and test the reversal before the first production step. The cutover runbook records each step, its checks and its owner, and serves as the basis for the handover notes.

What drives effort and duration

This table lists the factors that change how much work a move from the OpenAI API to self-hosted Ollama involves, why each one matters and how to keep it under control.

Driver	Why it matters	How to keep it down
Applications and call sites	Every service, script and batch job that calls the OpenAI SDK or REST endpoints must be found, changed and retested. Hidden call sites in notebooks or vendor tools are common.	Inventory all call sites with code search and API key usage logs before scoping. Route them through one internal client wrapper so the base URL changes in one place.
OpenAI-specific features in use	Tool calling, JSON mode, embeddings, vision, Assistants and file APIs each have different support in Ollama's OpenAI-compatible endpoint. Features without a compatible equivalent need redesign.	List which endpoints and parameters each application uses. Confirm support for each one against current Ollama documentation and your own test calls before choosing a design.
Size and diversity of the prompt set	Quality comparison needs representative prompts across tasks, languages, lengths and edge cases. A small or unrepresentative set hides regressions that users find later.	Build the evaluation set from real anonymized production traffic with agreed scoring rules. Reuse one set across all candidate models and all applications.
Number of distinct model replacements	Each hosted model replaced by a different open model needs its own evaluation, prompt adjustments and output checks. Mixing many models multiplies test cycles.	Consolidate use cases onto as few replacement models as quality allows. Record the chosen model and version per application in a single register.
Expected concurrency against Ollama defaults	Ollama's defaults allow 1 parallel request per model (OLLAMA_NUM_PARALLEL), a set number of loaded models (OLLAMA_MAX_LOADED_MODELS) and 512 queued requests (OLLAMA_MAX_QUEUE). Peak traffic above these limits needs tuning, extra servers or a gateway.	Measure peak concurrent requests per model from current OpenAI usage. Load test against the planned hardware and set the three variables from those results, not from defaults alone.
Hardware availability	Model size, memory and GPU availability determine which models can run at acceptable speed. Lead times for accelerators or capacity in your data center can set the schedule.	Confirm hardware delivery or capacity reservation early. Test candidate models on the hardware you already have before buying additional capacity.
Number of environments and approval gates	Each environment (development, test, staging, production) and each sign-off from security, legal or data owners adds setup and waiting time. Changes that touch sensitive prompts often need extra review.	Agree the gate list and evidence each gate needs at the start. Use the same deployment method in every environment so promotion is a repeatable step.
Gradual traffic switching	Shifting traffic in stages requires a router or feature flag, per-application rollback and comparison logging. Skipping this raises the risk of a full cutover failure.	Plan percentage or per-application switching with a rollback path for each stage. Log hosted and self-hosted outputs side by side during the early stages.

Migration phases and outputs

The six phases move traffic from hosted OpenAI calls to self-hosted Ollama in controlled stages, and each phase leaves a record that the next one depends on.

Inventory every OpenAI API call
Your team and DeployOpen list each application, service, script and prompt template that calls the OpenAI API, along with the endpoints, models, parameters, data types and owners. The output is a call inventory that shows which workloads can move first and which need redesign before they can run on a different model.
Classify use cases and shortlist models
Each use case is grouped by quality needs, latency, context length, data sensitivity and tool or structured-output requirements, then matched to open models that Ollama can serve. The output is a model decision record stating the chosen model, the rejected options and the reasons, which must be confirmed against the model's own license and the hardware sizing.
Build the environment behind access control
The Ollama server is installed in your infrastructure with its bind address and network exposure set deliberately, and the OpenAI-compatible endpoint is placed behind a reverse proxy with authentication, TLS and request logging, because the local API has no authentication of its own. The step closes when test clients reach the endpoint only through the controlled path and the server is confirmed not to send prompts to cloud services.
Evaluate against a golden prompt set
DeployOpen and your domain experts build a golden prompt set from real, anonymized production requests and agree acceptance thresholds for accuracy, format compliance, refusal behavior and latency with you before testing starts. The evaluation report records results per use case against those thresholds, and any use case that misses them goes back to model selection or prompt changes.
Run shadow traffic and canary cutover
Copies of production requests are sent to Ollama in shadow mode, with responses compared but not returned to users, and then a small share of live traffic is routed to Ollama through a feature flag or gateway rule with monitoring and a documented rollback trigger. The runbook covers routing changes, rollback steps and on-call contacts, and the cutover record logs each traffic increase with the metrics that justified it.
Complete the switch and decommission old keys
Once the canary holds against agreed metrics, all traffic for the approved use cases moves to Ollama, the OpenAI keys are revoked, and the configuration and secrets are removed from applications and CI systems. The closing check confirms that no remaining code path calls the old API, and the cutover record is updated with the revocation date and the final inventory state.

What teams keep in-house and what they outsource

This table shows who usually owns each area of a move from the OpenAI API to self-hosted Ollama and which skills each area needs, so you can see where a partner helps and where your own people must stay involved.

Work area	Usual owner	Skills needed
Application code changes	Your application team, with a partner for shared client libraries	Backend engineers who know the current OpenAI client calls, error handling, streaming and retry logic, and the OpenAI-compatible endpoint Ollama exposes
Model selection and benchmarking	Shared: partner runs tests, your team picks the models	ML engineers to compare candidate open models on your prompts, measure latency and output quality, and record hardware needs for each
Evaluation design and domain review	Your domain experts, with a partner building the test harness	Subject-matter reviewers who judge clinical, legal, financial or other domain answers, plus engineers who turn their criteria into repeatable tests. Domain review stays with your people whichever party builds the system
Access layer and proxy	Partner or your platform team, depending on who runs your gateway	Platform and network engineers to put TLS, authentication and rate limits in front of Ollama, which has no built-in authentication on its local API
GPU and storage capacity	Your infrastructure team, with a partner for sizing	Infrastructure engineers to size GPUs or CPU hosts, memory, and model storage, and to set Ollama concurrency and loaded-model limits for your peak traffic
Cutover and monitoring	Shared: partner builds dashboards and runbooks, your on-call team runs them	SRE and operations skills for gradual traffic shifting, latency and error alerting, and a rollback path back to the hosted API
Sign-off by security or compliance	Your security and compliance teams	Security and compliance staff to review data flows, model provenance, logging and access controls, and to decide what evidence they need. DeployOpen prepares evidence, but independent auditors issue any assessment outcome

What usually goes wrong

Most failures in this migration come from assumptions made at cutover. Check each item below before application traffic moves to Ollama.

Assuming full API parity

Ollama's OpenAI-compatible endpoint covers common chat and embedding calls, but request parameters, response fields, error shapes and streaming details can differ from the hosted API. Run every endpoint and field your application uses against Ollama and compare the results. Check current compatibility in the Ollama documentation before relying on it.

Switching models without task evaluation

An open replacement model changes output quality, formatting and refusal behavior. Prompts tuned for the hosted model often break on another model, so tune them per model. Build a fixed test set from your production tasks, score each candidate model and prompt version against it, and store the model tag and prompt together in version control.

Queueing when concurrency limits are reached

Ollama defaults to one parallel request per model (OLLAMA_NUM_PARALLEL) and holds up to 512 queued requests (OLLAMA_MAX_QUEUE). Traffic that the hosted API absorbed can back up on one local server and raise latency for every caller. Load test at expected peak concurrency, set parallelism and loaded-model limits on purpose, and alert on queue depth.

Exposing the API after changing the bind address

The local API on port 11434 has no authentication. Ollama binds to 127.0.0.1 by default, but setting OLLAMA_HOST to a network address makes the API reachable by any host that can reach that interface. Keep the loopback bind, or place a reverse proxy with authentication and TLS in front of it, then confirm from another network segment that unauthenticated requests fail.

No rollback after keys are revoked

Teams often revoke the hosted API key on cutover day, which removes the fallback if the local service degrades. Keep the hosted key active and the provider selection controlled by configuration until the Ollama deployment has passed its agreed acceptance period. Confirm with your provider how long keys remain usable, and document the switch procedure before go-live.

Unpinned version drift

Ollama publishes frequent releases. v0.40.5 was released on 14 October 2026, and v0.40.6 was published as a pre-release on 16 October 2026. An evaluation run on one version does not prove behavior on the next, so pin the evaluated version in your deployment and test every upgrade against the same evaluation set before rollout.

How the work changes by starting point

A single application on a developer workstation or one server is the smallest version of this work. The main question is whether the application can change its base URL and model name without a wider rewrite.

Scope: one application, one Ollama host, one set of model choices.
Endpoint change: point the OpenAI-compatible client at the local Ollama address and confirm the request and response shapes the application actually uses.
Evaluation: a smaller prompt set, reviewed by the people who own the feature.
Access: the local API has no authentication by default, so confirm whether the single host is reachable beyond the people who use it.
Effort driver: how many OpenAI-specific features the application calls, such as tools, streaming or structured output.

What the client needs to provide

Before implementation starts, the client needs to supply a complete inventory of every application, service and script that calls the OpenAI API, with a named owner for each. Missing callers are the most common reason a cutover stalls, because a forgotten batch job or internal tool keeps sending traffic to a hosted endpoint after the rest has moved. Alongside the inventory, provide the API keys in use, the usage logs for the current traffic profile (request volume, peak periods, token sizes and model names), and the list of models each caller depends on. Those logs set the capacity baseline for the hardware decision.

Next, the client must provide sample prompts and expected outputs for each use case, and name a reviewer who accepts those outputs as the quality bar. Without an accepted reviewer, no replacement open model can be judged as passing. The client also decides on hardware or cloud capacity, supplies a network and identity plan for the access layer (which clients reach the Ollama API, through what proxy, and how they authenticate, since the local API on port 11434 has no built-in authentication), and classifies the prompts and responses that will flow through the system.

The client must also decide whether cloud models from ollama.com are in or out of scope. Direct cloud inference at ollama.com/api and ollama.com/v1 requires an API key sent as a Bearer token, so those requests leave the client's infrastructure and carry the same data-handling questions as the hosted API being replaced. Record the decision in writing and confirm whether cloud features should be disabled on the server with OLLAMA_NO_CLOUD=1 or the server configuration file. Finally, name an approver for model choices and a named owner for cutover and rollback, since neither role can be filled by the implementation team.

Questions buyers ask before hiring help

What does an Ollama migration cost, and what should a quote include?

Cost depends on the written scope, not on a standard package. The main drivers are how many applications call the OpenAI API, how much prompt and response behavior must be re-tested, the hardware or cloud capacity needed to serve the replacement models, and how much integration work sits in your code. Part of the cost is one-time (evaluation, endpoint changes, cutover, handover) and part recurs (hardware, power, storage, and maintenance). A fixed-scope quote should itemize the applications in scope, the evaluation method and test set, the model selection work, changes to client code, traffic switching steps, monitoring, documentation and training, and what is excluded. DeployOpen does not publish prices. A quote follows from that written scope.

How long does it take to move from OpenAI to Ollama?

The schedule is set by the number of applications, the evaluation work needed to accept replacement models, and how quickly your team can approve hardware and network changes. Hardware procurement or cloud capacity approval often sets the start date, while output evaluation usually sets the critical path, because each application may need its own acceptance criteria. Gradual traffic switching adds time but lowers risk. Ask any provider for a timeline tied to named milestones and client review points, and confirm which steps depend on your infrastructure teams.

Can our own engineers migrate to Ollama without outside help?

Often yes for a single application with a small set of prompts, since Ollama exposes an OpenAI-compatible endpoint on port 11434, so many client changes are configuration changes. Teams usually need outside help when several applications share one model gateway, when evaluation must be designed and signed off by business owners, when GPU sizing and concurrency settings (such as OLLAMA_NUM_PARALLEL) must be tuned for real load, or when the server must sit behind a proxy with access control and logging. Map your own skills against those areas before deciding.

Will the local model give the same answers as the OpenAI model?

No, you should not assume it will. Open models differ from hosted models in quality, formatting, tool-call behavior, refusal patterns and tone, and results vary by model, quantization and prompt. The only reliable answer comes from measurement: build a test set from real, anonymized prompts, define pass criteria for each application, run the candidate model and the current model against it, and review the differences with the people who own the outputs. Plan for prompt adjustments and possibly a different model for some use cases.

Do prompts leave our network when we run Ollama?

Ollama states that when it runs locally it does not see your prompts or data, and cloud features can be disabled with OLLAMA_NO_CLOUD=1 or the server configuration file. Note that direct cloud inference at ollama.com uses an API key and is a different path. Verify this for your own deployment: confirm the server version and configuration, check outbound traffic with network monitoring or firewall rules, and test that requests never reach external hosts. Your security team should record the result as evidence.

Do we need a GPU to run the replacement models?

Not always, but the answer depends on model size, expected concurrency and latency targets. Ollama runs on CPU as well as GPU, and smaller models can serve light internal workloads without accelerators. Larger models and multiple concurrent users usually need GPU memory sized for each model plus headroom for the loaded-model and queue settings. Ask for a sizing based on your models and peak request volume, and confirm the hardware and driver support with your infrastructure team before purchase.

How DeployOpen runs this migration

DeployOpen scopes the move from the OpenAI API to self-hosted Ollama in writing, then builds and tests it inside infrastructure you control. Engagements are fixed-scope, so the agreed deliverables, responsibilities and timelines are settled before work starts.

How we scope

Inventory every application call to the hosted API, including model names, streaming use, function calling and request volumes.
Map each call to a candidate open model and define the quality checks each replacement must pass before traffic moves.
Agree in writing which Ollama hosts, GPUs or memory, model storage paths and API access layer are in and out of scope.
Ongoing support, upgrades and continuous testing can be added later as a retainer, so the scope should say what is handed over.

How we build

Point selected applications at the Ollama OpenAI-compatible endpoint on port 11434 and confirm request and response behavior for each client.
Keep the local API on 127.0.0.1 by default, or place it behind a reverse proxy such as Nginx if other services must reach it.
Set model storage location and concurrency limits (parallel requests, loaded models and queue depth) against the expected load.
Build a comparison harness that runs the same prompts against the hosted model and the replacement, then records differences for review.

How we hand over

Deliver a written record of the model choices, configuration values, proxy setup and the evaluation results behind each decision.
Document the gradual traffic switch, including the percentage steps, rollback trigger and the owner who approves each step.
Walk your engineers through how to add, update and remove models, and where prompts and outputs are logged.
Confirm that cloud features are turned off where your policy requires, using the documented configuration setting, and record the setting.

How we support

Agree the support boundary in the scope: which changes DeployOpen handles, which stay with your team and how issues are escalated.
Test new Ollama releases and model updates in a staging copy before any rollout to production traffic.
Keep version pins and a documented rollback path so a failed model or runtime update can be reverted quickly.
Offer a retainer for ongoing upgrades, evaluation runs and security testing if your team wants continued coverage after go-live.

Migrating from the OpenAI API to self-hosted Ollama