Importing custom GGUF and Safetensors models into Ollama

Ollama runs models through a llama.cpp backend, so many teams prepare custom weights with llama.cpp tooling before they import them. Converting and quantizing ahead of time lets you choose the file size and memory footprint of a model instead of accepting whatever the import path produces. The decision you are making is how much precision to trade for a smaller model that fits the hardware you actually operate, and that choice should be made before anyone builds a catalog entry around the file.

Quantization reduces the numeric precision of the weights. Lower precision generally shrinks the model file and the memory it needs at load time, which can decide whether a model fits on a given GPU, on CPU-only nodes, or within the memory budget of a shared host. The cost is output quality, and the loss is not uniform: a model may stay fluent on general prompts while degrading on your domain terms, structured output or long-context tasks. Weigh model size, quality and hardware together. A smaller quantized model on hardware you already run can serve your workload better than a larger one that forces a hardware purchase.

Before you convert anything, confirm which conversion scripts, input formats and quantization types your pinned Ollama release accepts. Ollama bundles its own llama.cpp version, so a format that works with a current upstream release may not import cleanly into the version you have pinned. Check the Ollama release notes and its model import documentation for the exact release, and record the llama.cpp version used for conversion next to the resulting file so the import can be reproduced later.

Quantized output must be tested against your own evaluation set, not only against a generic benchmark. Build a set of representative prompts from your real use cases, including edge cases and the outputs that matter most to your business, then run the same set against the unquantized reference and each candidate quantization. Score the results with criteria your reviewers agree on, and keep the scores with the model record. Repeat this test whenever the conversion tooling, the quantization type or the pinned Ollama release changes.

Ask your internal team or any implementation partner to quote this work against a written scope. The quote should name the conversion and quantization steps, the evaluation set and who builds it, the hardware used for testing, and the number of candidate formats compared. Ask how the evaluation results will be recorded and reviewed before a model is promoted to the internal catalog.

The first sizing decision is where imported models live. Ollama's documented default store is ~/.ollama/models on macOS, /usr/share/ollama/.ollama/models on Linux, and the .ollama\models folder in the user's profile on Windows. Imported weights can be large, especially full Safetensors checkpoints before quantization, so most self-hosted deployments set OLLAMA_MODELS to move the store onto a dedicated volume. Confirm that the account running the Ollama service can write to that path on your operating system, and that the volume holds the source files, the created model and every other model in your internal catalog. Set the variable in the service environment, not only in an interactive shell, so restarts keep the same location.

OLLAMA_NUM_PARALLEL controls how many requests one loaded model serves at the same time. Its documented default is 1, so each model handles requests one at a time until you raise it. Raising it is not free. Each parallel slot needs working memory for its context, so higher parallelism increases memory use on the same hardware. For an imported model, measure memory at the context length your applications actually send and at the concurrency your users will generate. Record the figures before you change the setting, so you can tell whether a throughput gain is worth the memory it costs.

OLLAMA_MAX_LOADED_MODELS sets how many models can stay in memory at once. Its documented default is three times the number of GPUs, or three when running on CPU. If your catalog holds several large imported models that get requested together, the loaded-model limit and available memory decide whether requests trigger unloads and reloads. Those reloads add latency that can look like a throughput problem. OLLAMA_MAX_QUEUE, documented at 512 queued requests by default, bounds how much work waits. A deep queue hides overload behind long waits, while a shallow one turns short bursts into refused requests. Confirm how your clients handle a refused request, including whether they retry and with what backoff.

Treat the documented defaults as a starting point. Throughput depends on model size, quantization, context length, accelerator memory and the request mix from your applications, and the defaults know none of these. Run a load test with your own imported models at the context lengths and concurrency your users will produce. Record latency, memory use, unload events and queue depth for each combination of the three settings. Set the variables only after the results show where latency and memory start to diverge. Repeat the test after any model update, Ollama upgrade or hardware change, and ask whoever runs the test to report results per model.

The operating model for approved custom models is simple to state: each GGUF or Safetensors artifact is checksummed when it is accepted, stored on internal storage, imported into Ollama with a documented Modelfile, and recorded in an internal register that names the owner, the source, the quantization and the approval reference. Nothing enters a serving host unless it has passed that path. The register is the document your auditors, incident responders and platform team will read first, so it should be treated as a controlled record rather than a wiki page that drifts.

The checksum step deserves attention because it is the only proof that the file serving users today is the file that was reviewed. Record the hash algorithm you use, store the hash in the register next to the artifact path, and verify it again before every import and after every copy between storage locations. Keep the Modelfile in version control with the same review rules as application code, so that the system prompt, template and parameter values are as traceable as the weights. Where a model is sharded, the checksum set and the shard order must be recorded together.

Keeping models on internal storage has a clear security reason. Ollama can publish custom models to ollama.com, and a model that contains proprietary fine-tuning data, internal prompts or licensed weights should not leave your network through a publish step that someone runs by habit. The local Ollama API on your hosts does not require authentication by default and does not expose prompts or data when it runs locally, so the protection comes from your storage, network and access controls. Confirm with your platform team that the import hosts cannot reach the public registry for push operations, and that the cloud features are disabled where policy requires it, for example with OLLAMA_NO_CLOUD=1.

Change control and model review have to map onto this workflow, and that mapping is specific to your organization. Ask your change advisory board and model risk or AI governance owners which step counts as approval: the artifact acceptance, the Modelfile review, the import into a staging host, or promotion to production. Confirm who may write to the internal storage location, who may run the import, and who may update the register. Decide how a rollback works when a model is withdrawn, and whether the register entry, the file and the running model are removed together.

Plan for drift and for the register itself becoming stale. Models will be re-quantized, re-imported after Ollama upgrades and retired, and each change should create a new register entry rather than overwrite the old one. Schedule a periodic reconciliation between the register, the storage contents and the models listed on each host, and treat any mismatch as an incident to investigate. Ask your compliance lead which records must be retained and for how long, because that retention period will decide how much history the register and the stored artifacts must keep.