Self-hosted vector database at billion-vector scale

Why a self-hosted vector database at billion-vector scale is a different project

If your document-derived embedding corpus has outgrown a single node and must stay inside your own infrastructure, the decision is no longer which vector product to pick. It is whether your organization can run a distributed system that stores, replicates and serves hundreds of millions to billions of vectors without leaving your control. At this size the work shifts from product selection to cluster design: how shards are placed, how copies are kept, where the index and raw vectors live, and who is paged when a node fails. pgvector, which many teams start with on PostgreSQL they already run, is generally not the target at this size, so plan for a dedicated engine from the start. On the platforms, Milvus Distributed is the engine built for this shape: it is a Kubernetes-native cluster that separates stateless proxies, coordinators and worker nodes, uses etcd for metadata, object storage such as MinIO, S3 or Azure Blob, and a write-ahead log such as Woodpecker, Kafka or Pulsar. Qdrant and Weaviate scale by splitting collections into shards and keeping extra copies, but both depend on quorum. Qdrant's distributed docs recommend at least three nodes for production so quorum survives a node failure, and its replication factor doubles storage at a factor of 2. Weaviate's replication is disabled by default and set per collection, with consistency levels chosen at query time. Confirm with each vendor how they handle rebalancing, index build time on your hardware, and upgrade paths for a cluster of your size.

Qdrant's installation docs say it needs block storage with a POSIX-compatible file system and cannot run on NFS or S3, so your storage design must rule out shared file systems and object storage for the vector data itself. Also confirm that Qdrant's Helm chart, which is community-supported, lacks zero-downtime upgrades and automatic shard rebalancing, both of which an operations team at this scale will need, so you may depend on the enterprise operator. Ask Milvus how its object storage and write-ahead log choices affect recovery time, and ask every vendor for a tested procedure to restore a full cluster from snapshots or backups. Treat the choice of engine as the smallest decision in this project. Before you commit, ask your implementation partner to produce a written sizing model for your vector count, dimensions and query rate, a node count that respects each vendor's quorum rules, and a named owner for on-call, upgrades and restores. Secure defaults also differ: the Qdrant security docs state that self-deployed instances listen on all interfaces with no authentication until you configure it, so access controls must be part of the first deployment, not a later hardening step.

Which requirements shape a billion-vector build

This table lists the requirements that most affect a very large vector deployment, the source that defines each one, and what it means for the build. Read it as a checklist to agree with a partner before sizing hardware, since several rows (storage type, quorum, and logging) change node counts and tooling rather than just settings.

Requirement	Source	What it means for the build
Shard and replica planning	Qdrant distributed deployment docs	Shards split collections across nodes and a replication factor of 2 doubles storage, so raw vector size is only the starting point for capacity.
Quorum for node failure	Qdrant distributed deployment docs	Production clusters need at least three nodes so quorum survives a node failure, and shard counts should be a multiple of the node count.
Block storage, not NFS or S3 (Qdrant)	Qdrant installation docs	Qdrant needs block-level POSIX storage and cannot run on NFS or S3, so SSD or NVMe volumes must be budgeted per node and in the air-gapped or private cloud plan.
Object storage and write-ahead log (Milvus)	Milvus architecture overview	Distributed Milvus depends on etcd for metadata, MinIO, S3 or Azure Blob for objects, and Woodpecker, Kafka or Pulsar as the log, so each dependency needs its own HA, backup and on-call ownership.
Index and memory choice	Milvus README	HNSW, IVF, FLAT, SCANN and DiskANN, with quantization and memory-mapped files, trade memory for latency; confirm the index and disk profile with a test on your own vectors before sizing nodes.
Authentication and TLS defaults	Qdrant security docs, Weaviate authentication docs	Self-deployed Qdrant listens on all interfaces with no authentication by default, so API keys or JWT roles and TLS must be configured before any network exposure.
Audit logging of queries and writes	Qdrant security docs, EU AI Act Articles 12(1) and 19(1)	Qdrant logs authenticated API operations from v1.17.0; where the system is high-risk, Article 12(1) requires automatic event logs and Article 19(1) requires keeping them at least six months unless other law sets a longer period.
Backup and restore	Qdrant snapshots docs, Qdrant migration tool docs	Snapshots must be restored into a matching version, and restore drills at billion-vector size should be timed and recorded as evidence before go-live.

Which open-source engines fit at this scale and why

At billion-vector scale, the engine choice drives the node count, the storage layout and the operations team you need. The three dedicated engines below differ in how they distribute data, how they handle consensus and how their enterprise controls are licensed, so each needs its own check before you commit.

Milvus for disaggregated scale

Milvus is the engine built for the largest collections among the options here. It is Apache-2.0 licensed and runs in three modes: Milvus Lite as a Python library, Standalone on one machine and Distributed as a Kubernetes-native cluster. Its architecture separates stateless proxies, coordinator and worker nodes, etcd for metadata, object storage such as MinIO, S3 or Azure Blob, and a write-ahead log such as Woodpecker, Kafka or Pulsar. Each of those components is another system your team must run. Confirm which index types (HNSW, IVF, DiskANN, quantization, memory-mapped files) your target recall and memory budget require, and confirm that RBAC and TLS are configured in your deployment.

Qdrant with Raft-coordinated shards

Qdrant is Apache-2.0 licensed and uses the Raft consensus protocol for cluster topology and collection structure. Collections are split into shards across nodes, and the replication factor sets how many extra copies exist, so a factor of 2 doubles storage. Qdrant's distributed guidance recommends at least three nodes for production so quorum survives a node failure, with a shard count that is a multiple of the node count. Qdrant also documents that self-deployed instances ship without authentication, so configure API keys, TLS and per-collection JWT permissions before exposure. Its JSON audit logging of authenticated API operations arrived in v1.17.0, and you should confirm that the log volume and retention meet your evidence needs at this size.

Weaviate and leaderless replication

Weaviate's core code outside the wl directory is BSD-3-Clause, but code inside wl requires a separate Weaviate enterprise license, and enterprise features are enabled by a license key. Confirm exactly which features your scale requires, such as multi-tenancy controls, RBAC depth or support terms, and whether they sit behind that key. Its replication is leaderless and disabled by default, so you must enable it per collection with a replication factor and then choose ONE, QUORUM or ALL consistency at query time. That choice trades latency against how many replicas must answer, and it should be tested under node loss before you size the cluster.

Confirm index and memory fit per engine

Index support varies by engine and version, so a vendor's feature list is not proof that a given index runs at your dimensions and recall target. Ask each candidate which index types it supports for your collection size, whether disk-based or quantized indexes are production-ready, and how much RAM the working set needs under your replication factor. Qdrant needs block-level storage with a POSIX-compatible file system and cannot use NFS or S3 for its data, so storage class is a sizing input, not a detail. Run a benchmark on your own embeddings before fixing node counts.

Operations team and cluster overhead

A distributed cluster adds work that a single node does not: rolling upgrades, shard rebalancing, backup coordination across nodes, consensus health monitoring and capacity planning. Qdrant's community Helm chart lacks enterprise features such as zero-downtime upgrades and automatic shard rebalancing, so confirm which path your team would run in production. Milvus adds etcd, object storage and a log broker to operate alongside it. Estimate the on-call coverage and platform skills these components need, and ask any implementation partner to name which of those responsibilities it takes on and which remain with your staff.

PostgreSQL with pgvector at this scale

PostgreSQL with pgvector is a sound choice for smaller collections when your team already runs PostgreSQL with mature backup and high availability. At billion-vector scale it is usually the wrong default, because it inherits the single-cluster limits of a relational database and lacks the sharded, consensus-based design of the dedicated engines. Only choose it if you already operate that stack at the required size, and confirm its index types, dimension limits and filtering behavior against your workload before you rely on it.

How sharding, replication and quantized indexes change memory and disk

The first decision that changes your hardware bill is how many stored copies of your vectors the cluster keeps. In Qdrant's distributed mode, a collection is split into shards spread across nodes, and a replication factor keeps extra copies of each shard. A replication factor of 2 doubles storage, so the raw footprint you calculate for one copy of the vectors is only the starting point. Before you size anything, confirm with the vendor how each engine you are evaluating stores replicas, whether replicas also duplicate index structures, and whether the same rule applies to backups and snapshots, which are separate copies on disk.

Node count follows from two rules rather than from data volume alone. The Qdrant distributed deployment documentation recommends at least three nodes for production so quorum survives a node failure, and recommends a shard count that is a multiple of the node count so shards spread evenly. Weaviate takes a different route: replication is leaderless, disabled by default, enabled per collection with a replication factor, and consistency is chosen per query at ONE, QUORUM or ALL. Your sizing inputs therefore include the availability target, the acceptable write and read consistency for each collection, and the expected growth in shard count, because changing the shard layout later is an operational project of its own.

Memory and disk trade against recall and latency. Quantization compresses vectors so more of the working set fits in RAM, at some cost in search accuracy that you must measure on your own queries. Disk-based indexes such as DiskANN, which Milvus lists among its index types, keep most of the index on storage and accept higher latency for a smaller memory footprint. Qdrant needs block-level storage with a POSIX-compatible file system and does not run on NFS or S3, so SSD or NVMe capacity, not object storage, is the disk you budget. Gather the vector count, dimensions per vector, target recall, p95 query latency, replication factor and backup retention, then benchmark quantized and disk-based settings against a representative sample before committing to a node shape.

Deployment patterns and the operations team a cluster needs

A billion-vector deployment is less a database decision than a set of infrastructure and operating decisions. The cards below cover the deployment method, storage, backup, monitoring and the people who keep the cluster running, and each one names a check to make before committing to a design.

Choose the deployment method deliberately

A community Helm chart gets a cluster running, but the Qdrant docs list zero-downtime upgrades and automatic shard rebalancing among the enterprise features it lacks. Those are the features that matter most when you resize or patch a billion-vector collection. Ask whether your design needs an operator, a vendor-supported private deployment, or a managed Hybrid Cloud option, and confirm in writing which upgrade and rebalancing behavior each one provides in your environment.

Specify block storage on SSD or NVMe

Qdrant needs block-level storage with a POSIX-compatible file system and cannot run on network file systems such as NFS or object storage such as S3. SSD or NVMe drives are the recommended choice. Storage sizing is where large collections often go wrong, so have your platform team confirm the drive class, the file system and the per-node capacity for both vectors and payload indexes before any node is ordered or provisioned.

Keep backups and snapshots off the cluster

Snapshots taken on the same volumes they protect do not protect you from a failed disk, a bad delete or a corrupted index. Plan a separate backup target, with its own access controls and retention, and test restores into a cluster that runs the same database version. Ask your partner how snapshots are scheduled per collection, where they are stored, and how long a full restore of your largest collection takes in your environment.

Build monitoring and on-call before go-live

A distributed cluster fails in ways a single node does not, such as a lost replica, a shard that falls behind, or a node that runs out of memory during indexing. Define the metrics that page someone, the runbooks for each failure mode and the escalation path before launch. Confirm that alerts cover replica health, memory pressure and disk latency, and that the on-call rota can respond within the hours your business requires.

Staff platform and database roles

Running a distributed cluster needs named owners for the Kubernetes platform, the database itself, and the storage and network underneath it. The platform team owns nodes, upgrades and capacity. The database team owns collection design, shard counts and recovery. Agree these ownership lines in writing, and ask your partner which of these roles they will supply and which must come from your own staff.

Assign security and data engineering duties

Security owns authentication, TLS, per-collection permissions and audit log retention, which Qdrant configures through API keys, JWT access control and JSON audit logging. Data engineering owns the embedding pipeline that writes vectors and the deletion process that removes them when source records change. Without a named owner for each, a billion-vector store tends to drift into an unmanaged state. Confirm both owners before the first production load.

How a billion-scale rollout moves from sizing to production

The sequence turns an unproven scale target into a sized, tested and handed-over cluster, with each phase closing on a written output the client can check before the next one starts. Durations are agreed in the written scope for each engagement.

Size the workload and filtering needs
DeployOpen works with the client's data and platform owners to document the vector count, embedding dimensions, expected growth, query rate, latency targets, metadata filters and hybrid keyword needs. The team also records which embedding model produces the vectors and how often it will change. The step closes with a sizing study that states the candidate engine, deployment mode, node count and storage class, and lists the assumptions the client must confirm, such as memory per node and replica count.
Prove scale on representative data
Before any production build, the team loads a representative sample of the client's own vectors and metadata into a test cluster and runs queries that match real filters and concurrency. The aim is to measure recall, latency and memory use under the chosen index type, such as HNSW, IVF or DiskANN in Milvus, or quantized indexes in Qdrant or Weaviate. The output is a proof-of-scale report with measured results, a list of index and quantization settings that passed, and a clear record of what did not meet the targets and why.
Build the cluster with a security baseline
The team deploys the chosen engine in the client's own cloud accounts, data centers or air-gapped network, using block storage with a POSIX-compatible file system rather than network file systems. Security is set before any data loads: authentication, TLS for API traffic, role-based access or per-collection permissions, network restrictions, and audit logging enabled. Object storage and write-ahead log components are provisioned where the chosen architecture needs them. The step closes when a configuration review confirms that no instance is exposed without authentication and every component is documented.
Test load, node loss and restore
DeployOpen runs sustained load at the sized query rate and ingestion rate, then deliberately removes nodes to confirm that quorum, shard placement and replicas behave as the design states. The team also restores a snapshot or backup into a separate environment and checks that the restored collection returns the expected results. Failures are logged with timestamps and root causes. The output is a test record covering load results, failover behavior and a timed restore, which becomes the baseline for the operations runbook.
Plan the cutover and re-embedding
The team agrees how production traffic moves to the new cluster, including dual-writing or batch backfill, the cutover window, and the rollback path. It also documents how the embedding pipeline will re-embed the collection if the model, dimensions or chunking strategy change, and how deletions from source systems reach the vector store. The client confirms ownership for each step. The output is a signed cutover runbook with a rollback procedure and a re-embedding plan that the client's change board can review.
Cut over and monitor production
Production traffic moves to the cluster according to the cutover runbook, with DeployOpen staff watching latency, error rates, memory pressure, replication status and audit log volume. The team validates a sample of query results against the proof-of-scale baseline. Any rollback is executed through the documented path. The step closes when the client confirms that production queries and ingestion meet the agreed targets for the monitoring period set in the written scope.
Hand over to run and upgrade operations
DeployOpen delivers the operations runbook, monitoring dashboards and alert thresholds, backup and restore schedules, the node replacement procedure, and the upgrade plan for the engine's release line. The client's operations team is trained on these procedures and on the on-call escalation path. Ongoing support, upgrades and continuous testing can run under a retainer if the client chooses. The output is a documented handover with named owners, which the client accepts in writing.

Evidence auditors and regulators ask for from a large vector store

This table maps each piece of evidence an assessor or regulator may request to the component that produces it, such as access configuration, audit logs, TLS settings, snapshot inventories and restore tests. Read it as a checklist to confirm with your engine and your auditor, since the exact artifacts depend on the platform and the assessment scope. DeployOpen prepares this evidence for the client but does not issue audit reports or certifications.

Evidence	Produced by	What the assessor checks
Access configuration	API key, admin and read-only key settings, or RBAC role definitions in the engine	That no instance runs unauthenticated and that each role has only the permissions its job needs
Per-collection permissions	JWT-based access control in Qdrant, RBAC roles in Weaviate, or Milvus role grants	That read and write rights are scoped to the collections or tenants each team owns
Audit logs of API operations	Qdrant JSON audit logging of authenticated operations, or the equivalent Milvus and Weaviate logs you confirm for your version	That authenticated writes, deletions and schema changes are recorded with identity and time, and are retained
Log retention	Log shipping to your central store with retention settings you define	For high-risk AI systems under the EU AI Act, that logs are kept at least six months under Article 19(1), unless other law sets a longer period
TLS configuration	Certificates and TLS settings for REST, gRPC and inter-node traffic	That data in transit between clients, nodes and object storage is encrypted, with certificate ownership and renewal records
Backup and restore test records	Snapshot or backup job logs plus dated restore test reports on a separate cluster	That a restore was performed, the collection counts and checksums matched, and the time taken was measured
Snapshot inventory	A list of snapshots with collection, version, location and retention date	That backups are findable, match the engine version that will restore them, and are deleted when retention ends
Change records for upgrades	Version pins, upgrade plans, rollback notes and test results per release	That each engine or chart upgrade was approved, tested on a non-production copy and traceable to a change ticket

What buyers ask before hiring help for a billion-vector cluster

What drives the cost of running a billion-vector cluster?

The biggest drivers are the memory and disk needed to hold the index, the number of replicas, the storage behind the write-ahead log and object storage, and the people who run the cluster. Vector count, embedding dimensions and query rate set how much of each you need. Memory-heavy nodes recur every month, while design, migration and load testing are largely one-time. When you ask for a quote, request a line-item breakdown covering node sizing for your dimensions and quantization choice, replica count, backup storage, re-indexing time, and on-call coverage. Ask who pays for each line: your cloud account, your hardware, or the partner's labor.

How long does a billion-vector rollout take?

The duration depends on the scale you target, the number of collections, how much of the embedding pipeline must be built, and how long load and recovery testing takes. Fixed-scope engagements agree the timeline in writing before work starts, so the scope document is where you should look for it. Expect the schedule to be driven by the largest unknowns: how quickly your source documents can be embedded, how long a full re-index runs on your hardware, and how many restore drills your compliance team requires. Ask any partner to show the phases and the test evidence each phase produces.

What does the client team need to provide for this work?

Your team needs to supply the infrastructure, the identity integration, the embedding pipeline owners and the people who approve data classification and retention. In practice that means:

Cloud accounts or data center capacity with block-level SSD or NVMe storage, since Qdrant cannot run on NFS or S3
An identity provider and key management process for API keys, TLS certificates and role assignments
Named owners for source systems, embedding models and deletion requests
A change window for upgrades and restore tests

Confirm these responsibilities in the written scope so that gaps show up before go-live, not during it.

Can our own team operate the cluster after handover?

Yes, if the handover includes runbooks, monitoring, tested restore procedures and training for the people who will be on call. A distributed cluster adds components such as coordinators, metadata stores, object storage and a write-ahead log, so operators need to understand how a failed node, a rebalance or a stalled re-index looks in practice. DeployOpen plans, implements, documents and supports self-hosted platforms in your own infrastructure, and ongoing support, upgrades and testing can be covered by a retainer. Ask for the runbook list and the training plan as part of the scope, and decide early who holds the on-call rota.

Does Qdrant Hybrid Cloud or Milvus managed hosting change the build?

Yes, because a managed option moves some operational work to the vendor and changes what you must design yourself. Qdrant Hybrid Cloud runs managed clusters on your own infrastructure, and Qdrant Private Cloud is a dedicated enterprise deployment, both priced through Qdrant sales. Zilliz Cloud offers fully managed Milvus with Serverless, Dedicated and bring-your-own-cloud options. Whichever path you choose, confirm where data, backups and logs physically reside, who holds the keys, which upgrade and support terms apply, and whether the vendor's control plane touches your vectors. These answers determine how much of the architecture in this page you still build.

How does the license affect enterprise features in a self-hosted vector database?

The license decides which features you can run without a commercial agreement, so check it before you commit to a design. Qdrant is Apache-2.0 licensed and free to self-host, and its security features such as JWT-based access control and audit logging are part of the open-source release according to its security documentation. Weaviate's LICENSE states that code in its "wl" directory is under a separate enterprise license and that enterprise features are enabled by a license key. Milvus is Apache-2.0 licensed. Ask any partner to list every feature your design depends on and to confirm, in writing, which license covers each one.

What should we ask a partner about sizing and quorum?

Ask how they calculated memory and disk from your vector count and dimensions, and how many nodes the design needs to keep quorum through a failure. The Qdrant distributed deployment documentation recommends at least three nodes for production so quorum survives a node failure, and a shard count that is a multiple of the node count. It also notes that a replication factor of 2 doubles storage. Ask for the sizing worksheet, the test results for re-indexing and query latency at your target scale, and the plan for what happens when a node is replaced during peak load.

What does an engagement cover beyond installation?

A complete engagement covers architecture, deployment, hardening, integration with your embedding pipeline and identity provider, backup and restore testing, monitoring, documentation and handover. DeployOpen prepares systems, controls and evidence for compliance assessments, but it does not issue audit reports or certifications, so independent auditors remain responsible for those decisions. Security testing of the APIs and the surrounding application is a separate scope item that you can add. Ask for the deliverables list and acceptance criteria in the written scope before you sign.

What to ask an implementation partner before signing

Start with sizing, because a billion-vector plan that is wrong on memory or disk fails in production. Ask the partner to show, for your target vector count and dimensions, how they estimate RAM for the index, disk for the vectors and payloads, and headroom for snapshots and compaction. Ask which index type they would use (HNSW, a disk-based or quantized option) and what recall and latency they measured to justify it on a representative dataset. Ask how they set shard counts and replication: for Qdrant, the distributed deployment documentation recommends at least three nodes for production and a shard count that is a multiple of the node count, and a replication factor of 2 doubles storage. A good answer ties each number to your query rate and failure tolerance, and it names the calculation so you can re-run it when volume grows.

Then test the failure and recovery claims, not the architecture diagram. Ask the partner to name the exact drill they run: stopping a node under load, confirming quorum holds, and restoring a collection from object storage or a snapshot into a clean environment. Ask for the measured restore time and the version-matched procedure they follow, and whether the restore includes the write-ahead log or metadata store, such as etcd for Milvus. Ask how they will handle re-embedding during upgrades, including whether the embedding model is pinned, how old and new vectors are kept apart, and who approves cutover. Finally, ask how they secure a cluster that ships without authentication. Qdrant's security documentation states that self-deployed instances listen on all interfaces with no authentication by default, so require network policy, API keys or JWT per-collection permissions, TLS and audit logging to be in the first build, with evidence. Close by requesting the runbooks, the on-call handover plan and the named owners who will operate the cluster after go-live.

How DeployOpen scopes, builds and supports a self-hosted billion-scale cluster

DeployOpen works on a fixed-scope basis, so the first deliverable is a written scope that the client agrees before build work starts. For a very large collection, that scope has to pin down the vector count, dimensions, index choice, replication targets, the object storage and write-ahead log dependencies, and the infrastructure where the cluster will run. DeployOpen plans the work inside the client's own cloud accounts, data centers or air-gapped networks, so the scope also records who owns each account, network path and key.

Workload profile: Agree the vector count at launch and at planned growth, the embedding dimensions, the query rate, and whether writes arrive in bulk or continuously.
Platform choice: Record which engine is in scope (for example Milvus Distributed, Qdrant in cluster mode or Weaviate) and why, with its license terms reviewed against the client's use.
Infrastructure boundary: Name the Kubernetes clusters, storage classes, object storage buckets and identity provider that the deployment will use.
Named deliverables: List the architecture document, runbooks, hardening baseline, backup and restore tests, security test report and compliance evidence pack, each with an acceptance step.
Support model: Decide whether handover is complete at go-live or whether a retainer for upgrades, monitoring and testing will follow.

Ask for the scope document in writing and check that it states what is excluded, who approves changes, and how a change to scope is priced.