RAG knowledge assistant in air-gapped networks

Why a RAG knowledge assistant in an air-gapped network is a different build

A RAG knowledge assistant can run fully offline on self-hosted components, so staff get cited answers from internal documents without any route to hosted model APIs or cloud vector stores. The catch is that every model file, package, container image and document update must be brought in through a controlled transfer. Nothing is fetched on demand, so the build starts with an inventory of what the assistant needs and a repeatable process for moving each item across the boundary. That changes the work in three ways compared with a connected build. There is no hosted fallback, so if a model is too slow or a parser fails, the fix is local. Models are not pulled when a user asks for them, so each model version must be selected, downloaded elsewhere, checked and loaded before anyone depends on it. Every dependency must be vetted and mirrored: container images, Python and system packages, embedding and chat model weights, and the vector database itself. Ask the implementation partner to produce a bill of materials for each release and to show how an internal registry or package mirror serves the enclave. For the platform, a practical starting point is Open WebUI or AnythingLLM for the front end, RAGFlow where document parsing depth matters, such as scanned files and complex tables, Qdrant for vectors, and Ollama or vLLM for models. Confirm each component's license terms before transfer, since Open WebUI's license restricts branding changes above a user threshold. Also confirm that the self-hosted Qdrant instance is secured before go-live, because its documentation states that default deployments listen on all interfaces without authentication.

Which requirements shape an offline RAG build, and where they come from

This table maps each requirement that affects an air-gapped RAG build to its source and to the design decision it forces. Read it as a checklist of questions to settle with your security, compliance and contracting teams before choosing components, since most rows apply only if your organization, data or contracts trigger them.

Requirement	Source of the requirement	What it means for the build
No runtime internet access	Your network security policy or accreditation boundary	Every container image, model weight, embedding model, parser package and vector index update must be prepared outside the boundary and carried in under review. Confirm the boundary definition with your security team before design starts.
Qdrant secure by default	Qdrant security documentation	Self-deployed instances listen on all interfaces with no authentication by default. Configure admin and read-only API keys, TLS for REST and gRPC, and per-collection JWT access so retrieval respects source permissions.
Log recording and six-month retention	EU AI Act Article 12(1) and Article 19(1)	Where the system is high-risk, it must technically allow automatic event logging over its lifetime, and providers must keep those logs for at least six months unless other law says otherwise. Confirm whether your use case is high-risk and plan log storage inside the enclave.
Software integrity and authenticity checks	NERC CIP-013-2 supply chain risk management	For bulk electric system operators, procurement must address verifying integrity and authenticity of vendor software and patches. Each package, model file and container image carried across the gap needs a recorded hash check and approval step.
Interactive remote access controls	NERC CIP-005-7 Electronic Security Perimeter requirements	If vendors support the system remotely, remote sessions to high or medium impact systems must pass through an Intermediate System with encryption and multi-factor authentication, and active vendor sessions must be detectable and disableable. Confirm whether the system falls in scope before granting vendor access.
Federal contract scope	32 CFR Part 170 (CMMC Program) and FedRAMP scope guidance (M-24-15)	Whether CMMC or FedRAMP applies depends on the contract and the data handled, not on the architecture alone. Ask the contracting authority which requirements apply and which systems count as in scope before choosing hosting.

Which open-source platforms fit an offline RAG assistant and why

An offline RAG assistant is a chain of components, and each one has a license, an operating model and an air-gap story of its own. The cards below take one decision at a time, from the front end to the gateway, and note what to confirm before a platform goes into a network with no internet route.

Front end under Open WebUI or AnythingLLM

Open WebUI can run entirely offline with Ollama or OpenAI-compatible APIs, which makes it a practical chat front end for staff. Its license is BSD-style, but clause 4 prohibits altering or removing Open WebUI branding except for deployments with no more than 50 end users in any rolling 30-day period, with written permission, or under other listed exceptions. Confirm the branding terms against your user count before rollout. AnythingLLM is MIT licensed, and its latest release is v1.18.0, so confirm which features your air-gapped build needs.

Parsing and retrieval with RAGFlow

RAGFlow is an open source retrieval-augmented generation engine under Apache-2.0, so its parsing and retrieval logic can be audited and kept inside the network. It is the component that turns scanned files and tables into chunks with citable positions. Confirm which document parsers and models it downloads on first run, because a default install may reach out to external model hubs. Pre-stage those artifacts on the offline side and verify them before import.

Vector storage on self-hosted Qdrant

Qdrant is Apache-2.0 licensed, and its latest release on GitHub as of October 2026 is v1.19.2. Qdrant needs block-level storage with a POSIX-compatible file system and cannot run on NFS or S3-style object storage, so the storage design is a hardware decision, not a detail. By default, self-deployed instances listen on all interfaces with no authentication, so enable API keys and TLS before the cluster joins the network. Confirm the replication factor, which doubles storage at a factor of 2.

Local runtime with Ollama and vLLM

Ollama is MIT licensed, latest v0.40.5, and runs models on a single workstation or server. It binds to 127.0.0.1 port 11434 by default and needs an explicit OLLAMA_HOST setting plus a proxy for shared access. vLLM is Apache-2.0 licensed, latest v0.31.0, and suits shared GPU serving for many users through an OpenAI-compatible endpoint. Choose Ollama for small teams and evaluation, and vLLM when concurrency and throughput matter. Confirm model weights are staged offline and that cloud features are disabled with OLLAMA_NO_CLOUD=1.

Gateway and tracing with LiteLLM and Langfuse

LiteLLM is MIT licensed except its enterprise directory, and it gives applications one OpenAI-format endpoint with logging, budgets and guardrails. Langfuse is MIT licensed except its ee directories, and it records prompts, retrieved chunks and answers in your own storage. Together they supply the logs that record-keeping duties in your framework may require. Confirm which features sit in the enterprise directory, and that the trace store stays on your network.

How updates cross the gap without breaking the assistant

The hardest offline problem is not running the assistant once, it is moving change across the boundary without breaking it. Model weights, container images, Python and system packages, Qdrant snapshots and document changes each need their own transfer route, and each route needs a verification step before anything is imported. Hash every artifact at the source, sign where the vendor or your build pipeline supports it, and recompute the hashes inside the enclave before import. Keep a manifest for each release that lists every artifact with its version, hash, source and approval record. That manifest is also the evidence trail for the software integrity expectations in NERC CIP-013-2 if the environment falls under that standard, and the same discipline supports supply chain review elsewhere. Confirm with your security team which signing tooling is approved for the boundary before you design the transfer process.

Two settings need attention on purpose. Ollama stores models in the directory set by OLLAMA_MODELS, so point that variable at a controlled volume that receives only approved model files, and do not rely on the default user path. Ollama's local API at port 11434 requires no authentication and binds to 127.0.0.1 by default, so decide the bind address and put an authenticated reverse proxy in front of it if other hosts must call it. The same logic applies to Qdrant, which listens on all interfaces with no authentication unless you configure API keys or TLS. For the vector store, Qdrant snapshots and the Qdrant migration tool move collections between environments, which lets you stage an index built on a connected test system and carry it across as a single artifact. For the cluster itself, the kubeadm image list and pull commands fetch control plane images ahead of time, and the --image-repository flag points installation at a private registry inside the enclave.

Re-index only what changed. Compare the new document export against the previous manifest by file identifier and content hash, then delete and re-embed the chunks for modified or removed files rather than rebuilding the full collection. Full rebuilds are useful after an embedding model change, because vectors from different models are not comparable, so treat a model swap as a planned release with its own evaluation run. Illustratively, a transfer that carries only a few hundred changed policy files can be verified and indexed in a single maintenance window, while a model swap may require a full rebuild and retest. Confirm with your vendors which upgrade paths between Open WebUI, RAGFlow or Qdrant releases are supported offline, and rehearse one full transfer on a staging enclave before the first production update.

Which air-gap pattern fits: fully disconnected, one-way transfer or controlled mirror

In a fully disconnected enclave, nothing crosses the boundary except by approved physical media. Every container image, Python package, operating system patch, embedding model, chat model and document set is prepared on a staging side, scanned, signed off and carried in by hand. Approval usually sits with a security owner who signs the transfer record, with the platform team preparing the bundle. Updates tend to be the least frequent of the three patterns, so the team should plan for longer gaps between model and corpus refreshes.

Import approval: A named security owner and a platform owner sign each media batch, with a manifest of hashes and versions.
Components needing mirrors: Usually none in the strict sense, because the enclave holds offline copies of the images, packages and model files, but a local container registry and package repository are still common for maintainability.
Update rhythm: Qualitative only. Expect infrequent, batched refreshes, so corpus and model currency lag behind the source systems.
Main risk: Stale or unverified content, and the operational habit of importing ad hoc media outside the approved process.
Boundary simplicity: The fewest network paths to defend, which suits small, tightly scoped assistants.

Confirm with your security office which media handling and cross-domain rules apply, and whether the enclave's audit trail can prove what was imported and when.

How big the vector index gets and what drives its storage

Vector index storage is not a single number you can look up. It is the product of several drivers: the number of chunks your ingestion pipeline produces, the embedding dimension of the model you choose for offline use, the replication factor, how many snapshots you retain, and the volume of logs your enclave keeps. Chunk count depends on your corpus and your chunking settings, so the first decision is how many documents you will index and how they are split. Embedding dimension is fixed by the model, so changing the embedding model later usually means re-embedding the whole corpus. Replication factor is the driver most often underestimated: a factor of 2 doubles the storage for every collection, and snapshots add a further copy for each retained point in time. Ask your partner to show the arithmetic for each driver on your own corpus before hardware is ordered.

Qdrant places hard constraints on the storage layer. It needs block-level storage with a POSIX-compatible file system and cannot run on network file systems such as NFS or on object storage such as S3. Inside an air-gapped enclave, that makes local SSD or NVMe volumes the practical baseline, and the storage team should confirm that any virtualized or SAN-backed volume presents block semantics before go-live. Qdrant's distributed mode uses Raft consensus for cluster topology and splits collections into shards across nodes. The Qdrant distributed deployment documentation recommends at least three nodes for production so quorum survives a node failure, and a shard count that is a multiple of the node count so data spreads evenly. Plan the cluster from the measured corpus and embedding model, not from a generic estimate.

Measure before you size. Run a pilot ingestion of a representative slice of your documents through the exact embedding model you will use offline, record the vector count, dimension and index size, then extrapolate to the full corpus with replication, snapshot retention and log growth added. The Qdrant Helm chart is a community-supported path that lacks enterprise features such as zero-downtime upgrades and automatic shard rebalancing. In an enclave with limited maintenance windows and no direct pull from upstream registries, that gap matters: confirm with Qdrant whether the enterprise Kubernetes operator is available for your deployment model, and if not, agree in writing how upgrades will be staged across the gap, how rebalancing is handled and how much capacity headroom you need to carry through an upgrade.

How an offline RAG assistant moves from scoping to supported operation

The sequence turns an open-source assistant into one that runs inside a closed network, with each phase producing a record that proves what was built, what was tested and how the system will be updated.

Scope the boundary and classify the data
The client's security, compliance and business owners agree which enclave will host the assistant, which document collections are in scope and which data classes must stay out of the index. DeployOpen maps the components the design needs, including the front end, the vector database, the embedding model, the chat model and any connectors. The output is a signed scope statement and a data flow diagram that shows every path in and out of the boundary. The step is done when the approvers have accepted both documents in writing.
Build the full stack on the connected side
Engineers assemble the complete stack on a staging network that has internet access, pinning every container image, package and model file to a fixed version. Each artifact receives a recorded hash, and the team lists every dependency in a manifest that covers the front end, Qdrant, Ollama or vLLM, parsers and operating system packages. Where the platform supports it, the team also prepares offline copies of documentation and models. The output is a dependency manifest with checksums and a bill of materials that the client's security team can verify before anything crosses the gap.
Run the offline acceptance test in an enclave replica
The stack is installed in a replica of the target enclave with outbound routes removed, and the team confirms that the vector database, the model runtime and the front end start and answer queries without internet access. Network captures and firewall logs are checked for any outbound connection attempt, and DNS and telemetry features are disabled and verified. The team also confirms that Ollama or vLLM serves models from local storage and that the interface loads its assets locally. The output is a network test record listing each check, its result and the evidence file that supports it.
Ingest the corpus and evaluate retrieval
Documents are parsed, chunked and embedded inside the enclave, and source permissions from the client's identity system are mapped onto each collection so retrieval filters results for every user. The team builds an evaluation set of representative questions with expected source passages and reviews whether answers cite the right documents and decline when the corpus has no answer. Storage for the vector index is sized from the embedding dimension, chunk count and replication setting, and the team records the result. The output is an evaluation report covering retrieval hits, citation accuracy, refusals and permission tests.
Capture logs, evidence and access reviews
The team configures audit logging for queries, retrievals, administrator actions and model changes, and sets retention and access controls for those logs in line with the client's policy. Where the assistant may later be assessed against AI logging obligations, the logs are scoped to capture the events the client's compliance lead specifies. Access to the admin console, API keys and the vector store is reviewed and documented, with a named owner for each role. The output is a logging specification, a sample of captured events and an access review record signed by the client.
Hand over with an update runbook for each transfer
DeployOpen walks the client's operations team through the runbook for moving updates across the gap, which covers how packages and models are downloaded on the connected side, how hashes are verified on arrival, how the offline repository is refreshed and how a rollback is performed. The team also documents backup and restore for the vector database, since Qdrant snapshots and the migration path depend on the storage layout chosen. Support responsibilities, escalation paths and the scheduled review of the evaluation set are agreed in writing. The output is an updated runbook, a handover record and a rehearsed transfer that the client's team has completed.

Which evidence auditors ask for and which component produces it

This table maps the evidence an assessor is likely to request from an air-gapped RAG assistant to the component that produces it. Read each row as a question to answer before go-live, and note which records come from the network layer, the staging pipeline, Qdrant, the gateway or front end, and the transfer runbook. DeployOpen prepares this evidence, but independent auditors and certification bodies issue any audit report or certification.

Evidence item	What it shows	Component that produces it
Network test record	No outbound connections from the enclave, checked after each import	Firewall or network monitoring logs, with a signed check result stored per transfer
Approved software bill of materials	Every container image, model file and package carried into the enclave and its version	Staging pipeline that builds the bundle outside the enclave
Hash records for images, models and packages	Integrity of each artifact as it crossed the gap, matching the approved list	Staging pipeline and the import step inside the enclave
Qdrant audit logs	Authenticated API operations on the vector store, available since v1.17.0	Qdrant JSON audit logging, forwarded to the organization's log store
Qdrant access configuration	Which admin, read-only and JWT roles can read or write each collection	Qdrant API key and JWT configuration, exported and reviewed on a schedule
Gateway request logs	Which application or user called which model, and when	LiteLLM or the chosen gateway, with retention set by the organization
Chat and retrieval logs	Questions asked, sources cited and answers returned for the retention period	Front end such as Open WebUI or AnythingLLM, with logs sent to the organization's store
Retention and high-risk mapping	How log retention meets EU AI Act Article 12 and Article 19 where the system is high-risk	Retention policy document, confirmed against the organization's legal review
Update and change records	Each transfer, who approved it, what changed and rollback steps	Transfer runbook and change ticket linked to each import

What to ask an implementation partner before an offline build

Start with proof of isolation, because every other control depends on it. Ask how the partner demonstrates that the enclave makes no outbound calls, which tool or method they use to show it, and whether the same test runs again after every import of models, images, packages and vector data across the boundary. Ask who signs off the result and where the evidence is stored. Ask them to confirm, in writing, which components try to reach the internet on first start, including Ollama's cloud features, which can be turned off with OLLAMA_NO_CLOUD=1, and how they confirm that setting is in force. Also ask how they handle the offline installation path for each component, since some vendors document one and others do not.

Next, ask how every model, container image and package is verified before it crosses the boundary. A strong answer names the checksum or signature method, the record kept for each transfer, and who approves the import. Ask how this maps to the vendor software integrity control in NERC CIP-013-2, if the enclave supports a regulated utility, and how they will treat patches and vulnerability data, including whether Wazuh's offline vulnerability repository or an equivalent feed will be used. Ask for the written scope, deliverables and acceptance criteria before work starts. For the vector store, ask how they set up Qdrant authentication, TLS and per-collection access. The Qdrant security documentation says self-deployed instances listen on all interfaces with no authentication by default, so ask the partner to show the configuration keys they set, the API key types they issue, and how JWT-based read or write permissions per collection are mapped to the source document permissions. Ask how permissions are carried into retrieval filters, how they test that a user cannot retrieve a chunk they could not open in the source system, and how they size storage. Qdrant needs block storage and does not run on NFS or S3, so ask which drives they plan to use. Ask which replication factor and node count they recommend, because the Qdrant distributed docs recommend at least three nodes for production and a replication factor doubles storage at 2. Finally, ask for a license review of every component, including Open WebUI branding terms, the Dify multi-tenant restriction, and whether air-gapped features are paid-edition features for the products in the stack.

What buyers ask about an air-gapped RAG assistant before hiring help

How much work does an air-gapped RAG assistant build take?

The effort depends mainly on how hard the boundary is and how much work sits in the document pipeline. A build usually covers five areas: the offline transfer path for software and models, the number and type of document sources, parsing needs for scanned files and tables, permission mapping from source systems, and the scope of answer evaluation. Each area adds work on its own, and they multiply when combined. Ask a partner to break the scope into these five areas in writing before agreeing on a timeline. A fixed-scope agreement records scope, deliverables and timelines before work starts, so the boundary and source list should be settled first.

What does an air-gapped RAG build cost, and what recurs?

Costs fall into a one-time build and recurring operations. The one-time build covers architecture, installation, connectors, parsing pipelines, permission design, evaluation and handover documentation. Recurring costs include transfer cycles across the gap, patching and upgrades of each component, re-indexing when documents change, ongoing evaluation as the corpus grows, and support. Hardware or GPU capacity is a separate line that the organization usually owns. Costs rise with source count, scanned content, the size of the evaluation set and the number of review cycles. Ask any quote to list each component, who is paid for it, and which items are one-time or recurring. DeployOpen does not publish a price list, and a quote follows from the written scope.

Can our own team build and run an air-gapped RAG assistant?

Yes, when the team already has the core skills. A team that runs Kubernetes, can operate GPU or CPU inference servers, understands identity and access control, and can run a security review is well placed to build this. The team must also supply several things that no vendor can provide: written approvals for the boundary and data flows, access to the source documents and their owners, a representative test corpus with questions and expected sources, and time from security and records staff. Missing any of these usually delays the project more than a technical problem. A partner can fill gaps in skills or capacity, but the approvals and document knowledge stay with the organization.

Can a SaaS or hosted RAG service run inside an air gap?

Generally no. Most SaaS knowledge assistants and hosted RAG platforms need outbound connections for model inference, updates, licensing or telemetry, which a true air gap does not allow. Some vendors offer self-managed or sovereign editions that run in a customer's network, but that is a separate product and deployment model. Before shortlisting any vendor, ask in writing whether every component, including embedding models, parsers, updates and license checks, runs without outbound connectivity. Ask for the list of network endpoints the product contacts at install and at runtime, and confirm how it receives updates across the boundary.

Which licenses and paid editions must we confirm first?

Confirm the license and edition for every component before choosing a stack, because air-gapped support is often a paid-edition feature. Vendors that list air-gapped deployment do so on specific plans, such as Mattermost on Enterprise Advanced, Rocket.Chat outside its free Starter plan, and Metabase on its Enterprise plan. Check each component separately:

Vector database: Qdrant's open-source engine is Apache-2.0 and free to self-host, while Qdrant Hybrid Cloud and Private Cloud are priced through sales.
RAG and chat layer: Open WebUI's license restricts removing branding beyond its listed limits, and Dify's license restricts multi-tenant use.
Model runtime: Ollama is MIT licensed, and running models on your own hardware is unlimited, but cloud models are a separate paid plan.

Ask a partner to produce a license register covering each component, its terms and any paid features your deployment needs.

How long before the first answer comes out of an air-gapped assistant?

The schedule depends on the factors that set the critical path rather than on a fixed duration. The longest items are usually approvals for the boundary and data flows, the transfer process for software and models, and access to the first document sources. Next come parsing quality for scanned and tabular files and the design of permission filtering, since answers must respect source access. A working prototype on a small, approved corpus can come much earlier than production readiness, which needs evaluation results, handover and operational runbooks. Ask a partner for a milestone plan that separates a first answer on test data from a production release, and confirm which approvals each milestone needs.

How DeployOpen scopes, builds and supports offline RAG assistants

DeployOpen plans, implements, integrates, hardens, documents and supports self-hosted open-source platforms inside the client's own air-gapped network. For an offline RAG assistant, that means fixing the document scope, model and vector store choices, and the controlled transfer process before anything is installed.

How we scope

Inventory the document sources, file types and scanned content that will be indexed inside the enclosed network
Agree which embedding and chat models will run locally, and confirm hardware available for inference and indexing
Record the written scope, deliverables and timeline before work starts, since most engagements are fixed-scope

How we build

Deploy the assistant, vector database and model runtime into the client's own air-gapped environment, with no outbound calls
Configure permission-aware retrieval, citations and access controls, then test retrieval and answer quality against agreed question sets
Set up a documented offline update path so models, packages and document batches are verified before they cross the gap

How we hand over

Deliver architecture, configuration, sizing notes and runbooks written for the client's operations and security teams
Document the transfer procedure, including checksums, approval steps and the records kept for each imported package
Walk through restore and re-indexing steps so the client's own staff can operate the assistant after handover

How we support

Offer ongoing support, upgrades and continuous testing as a retainer, scoped in writing with the client
Review each offline update batch for new model versions, dependency changes and vector index growth before release
Prepare evidence for compliance assessments, such as logs and configuration records, while independent auditors issue any assessment outcome