LLM observability in air-gapped networks

Every transfer starts with a bill of materials that lists each container image, Helm chart, database package and model file the Langfuse deployment needs, with its exact version and source. For Langfuse that list covers the Langfuse Web and Langfuse Worker images, PostgreSQL, ClickHouse, Redis or Valkey, the S3-compatible blob storage component, and any utility or exporter images referenced by the chart. Pull these from the internet side in a controlled staging zone, mirror them into a private registry inside the enclave, and keep the manifest with the transfer record. The Kubernetes offline pattern applies to the cluster itself: kubeadm config images list and kubeadm config images pull fetch control plane images in advance, and --image-repository points kubeadm init at the private registry. Confirm with the vendor which chart version pins which ClickHouse, Redis or Valkey image tags, and whether the chart can be rendered with every image reference overridden to your registry without manual edits.

Integrity checks belong before anything enters the enclave. Compute checksums for every artifact in the bill of materials, verify the publisher's signatures where they exist, and record the results beside the manifest. Re-check digests after the registry import, because a mismatch at that point usually means a copy error or a substituted tag. Treat tags such as latest as unacceptable for a reproducible import, and pin by digest. The same discipline covers local evaluation models: an LLM-as-judge or embedding model used for Langfuse evaluations is a file the enclave must hold, so its weights, tokenizer files and license terms go through the same manifest, checksum and approval steps. Log who requested, reviewed and approved each transfer, with the date and the artifact versions, so a later auditor can trace which code ran in production.

Before the first transfer, ask the vendor or your partner to confirm three things in writing: the offline registry layout the chart expects, every chart dependency that pulls from an external repository at install or upgrade time, and whether the Langfuse Enterprise features you need are covered by the license key that must also be imported. Run a dry-run install in a disconnected test cluster and watch for any outbound DNS or HTTPS call. Network-isolated test runs catch hidden dependencies, such as a telemetry hook or a model download on first startup, before they reach the production enclave.

Connected teams can take a new Langfuse release, pull the image and roll it out the same week. Air-gapped teams cannot. Every Langfuse release, every ClickHouse release and every base image has to be downloaded on a connected network, imported across the gap, scanned, and tested in a pre-production enclave before it reaches production. Plan that cycle as a standing operational duty with an owner, a calendar and a rollback path, not as an emergency activity that happens when a bug forces it. Updates within a major Langfuse version are documented as non-disruptive, and database migrations run automatically at startup. That convenience cuts both ways: an imported patch can change schema on the first start, so the restore point must be taken before the upgrade begins. Major upgrades follow their own migration guides. The v3 to v4 guide requires ClickHouse 25.12 or later (26.4 recommended), PostgreSQL 15 or later and Redis 7.0 or later, and it moves to an observations-first ClickHouse data model. Confirm the target versions of all four components against that guide before you import anything, because a ClickHouse upgrade may be needed before the Langfuse upgrade can proceed. The guide also advises about 3x ClickHouse disk headroom when backfilling historic data, so size the storage before you start the migration, not after it stalls.

ClickHouse has its own support window, and it limits how long you can stay on a line. The ClickHouse security policy lists versions 26.9, 26.8, 26.7 and 26.3 as receiving security updates, and all other versions as unsupported. LTS lines such as v26.8.29.10-lts and v26.3.53.4-lts were published on October 16, 2026, and LTS releases are supported for a year after release. Choose the line your team can sustain, track the end of support in your change calendar, and budget import cycles for it. Read the changelog for every release before importing it, because minor backward-incompatible changes can occur even within a stable line. A pre-production enclave copy with representative data and queries is the only practical test. Restore a recent backup into it, run the new Langfuse and ClickHouse versions, then check ingestion, the trace UI, evaluation jobs and a sample of queries before promoting anything. Ask your implementation partner how the enclave is kept in step with production, who signs off the promotion, and how long a rollback takes if the restored data does not behave as expected.

Start by asking how the partner builds and verifies the bill of materials and transfer manifest for every image, Helm chart and package that crosses the gap. Ask who signs the manifest, how checksums are recorded, and how the partner proves that what entered the enclave is what was approved outside it. The control behind this is supply chain verification under NERC CIP-013-2 for regulated energy operators, and similar software integrity expectations apply elsewhere. Ask for a sample manifest from a previous build, not a description of one. Next, ask how the ClickHouse Keeper quorum and backups will work when the enclave has no external object storage. Keeper needs at least three nodes on dedicated hosts in production, so confirm where those hosts sit and who patches them. Ask where the BACKUP destination lives (local disk, an internal S3-compatible store or another enclave system), how often backups run, and how the partner will test restores on a spare cluster. The ClickHouse documentation is explicit that a backup never test-restored will probably not restore when needed, so ask for the restore test schedule and the evidence each test produces. Also ask how upgrades will be planned when every Langfuse and ClickHouse release must be imported by hand. Ask which ClickHouse line they will pin (stable or LTS), how they check the current security support list before choosing, and how long they expect an import cycle to take. Ask which Langfuse features they would need under a license key, such as RBAC roles, data retention policies, audit logs and server-side masking, and who decides whether to buy them. Ask how local evaluation models are chosen and served, what GPU or CPU hardware they assume, and what storage that adds. Finally, ask for a written scope with deliverables before work starts, the evidence pack they will produce for an assessor, and their approach to remote access, support windows and patch imports.