Self-hosted LLM inference for SOC 2 controls

Self-hosted LLM inference for SOC 2: what the availability criteria ask of a model service

If your SOC 2 report must now cover an in-house model endpoint, the availability criteria apply to the GPU service itself, and the common criteria apply to who can reach it. Availability is one of five trust services categories in the AICPA's 2017 Trust Services Criteria, and it is included in your report only when you put it in scope. Security is the exception: the common criteria CC1 to CC9 apply to every SOC 2, so the model endpoint falls under logical access (CC6), system operations (CC7) and change management (CC8) whether or not you select availability. The 2017 criteria contain three availability criteria, A1.1 to A1.3. For a model service, A1.1 asks you to measure current capacity and use, forecast demand and act on the forecast. A1.2 asks you to run and monitor backup and recovery processes and the infrastructure behind them. Confirm the exact wording and points of focus in the AICPA 2017 Trust Services Criteria with revised points of focus before you map controls to them.

Decide the scope first, because it determines which evidence you must produce. Write the system description for the model service as management's description of the system, prepared against the 2018 description criteria (DC 200), and list the GPU nodes, the inference engine, the gateway in front of it, the model weight store and the configuration repository. A SOC 2 Type 1 report describes control design at a point in time, while a Type 2 report adds an opinion on operating effectiveness across a period, so most SaaS teams need Type 2 evidence that capacity reviews, restores and access reviews happened on schedule. DeployOpen plans, builds, documents and tests these controls in your environment and prepares the evidence package, but the service auditor issues the report and your management signs the representations. Ask your auditor early whether they expect the model service to be a subservice organization or part of your own system boundary, since that changes what you collect from third parties.

Mapping A1.1, A1.2 and common criteria to inference controls

This table maps the SOC 2 availability and common criteria that a model service will touch to the inference controls that address them. Read each row as the control, what the team builds, and the evidence an assessor will ask to see over the audit period.

Requirement	What the function must do	How to configure it
A1.1 Capacity monitoring and forecasting	Measure current GPU and endpoint use, forecast demand and act on the forecast before queues or latency degrade.	Scrape vLLM /metrics for request counts, token throughput, latency histograms and KV cache use. Set Prometheus retention to cover the full audit period. Define written thresholds that trigger a GPU addition review.
A1.2 Backup and recovery of model artifacts and configuration	Back up model weights, container images, engine and gateway configuration and the LiteLLM database, and prove restores work.	Store weights and image digests in a versioned, access-controlled repository. Back up the PostgreSQL database behind LiteLLM virtual keys on a schedule. Run and record restore tests on a separate host.
CC6 Logical access	Restrict who and what can call the engine, and limit each caller to the models and budgets it needs.	Set vLLM --api-key, but do not rely on it alone. Put a reverse proxy in front that allowlists only /v1 paths. Issue LiteLLM virtual keys with budgets, rate limits and model restrictions, and use SSO for admin access.
CC7 System operations and monitoring	Detect failed nodes, error spikes and unusual usage, and route alerts to a person who responds.	Alert on health endpoint failures, error rates and GPU faults. Forward gateway and proxy logs to central storage. Keep the metrics endpoint off public networks.
CC8 Change management for engine upgrades	Test, approve and record every engine and gateway change before it reaches production.	Track vLLM minor releases, which arrive roughly every two weeks. Test each candidate in staging against a fixed evaluation set. Record approvals, rollback steps and version pins in the change log.
CC9 Risk mitigation for vendor and model provenance	Assess open-source engines and open-weight models as third parties and confirm where each model came from.	Keep a model inventory with source, license and checksum. Verify artifacts against published hashes. Review license terms for each model and engine, and document the review and its owner.
Evidence to collect	Give the assessor a traceable record for each control across the period, not a snapshot.	Export capacity dashboards, backup and restore logs, access reviews of virtual keys, change tickets and the model inventory. Confirm with your auditor which samples they will select.

Which open-source components fit an availability-scoped inference service

An in-house model service enters SOC 2 scope as a system component, so each part that serves, routes, records or restores requests needs an owner and evidence. The cards below cover the serving engine, a smaller single-node option, the gateway that controls access, trace evidence, and the cluster, metrics and backup layer that the availability criteria depend on.

vLLM as the serving engine

vLLM is Apache-2.0 licensed with no paid edition, so licensing cost is zero and the real cost is GPU capacity and the people who operate it. It exposes OpenAI-compatible endpoints such as /v1/chat/completions, so applications need no code change to call it. Check the release cadence: minor releases arrive about every two weeks, so the team must define how it tests and approves upgrades. Confirm the GPU vendor and parallelism setup your hardware needs before you commit to a sizing.

Ollama for single-node footprints

Ollama is MIT licensed and suits smaller, single-node services such as a department pilot or a team server with a few models loaded. It is simpler to operate than a multi-GPU serving stack, but it has fewer capacity controls, so you should confirm which of your availability and capacity measurements it exposes before it carries a SOC 2 system. Record the version you run, because the engine changes often and your change management evidence must show each upgrade.

LiteLLM for keys, budgets and logs

LiteLLM sits in front of the engine and enforces who can call which model, using virtual keys with per-team budgets, TPM and RPM limits and model restrictions. Its core is MIT licensed outside the enterprise/ directory, but SSO beyond five users, SCIM, audit logs and team RBAC require an Enterprise license, so confirm which controls your SOC 2 evidence depends on. Multiple proxy instances need PostgreSQL and Redis 7.0 or newer for shared rate limits and router state, which adds components to your availability design.

Langfuse or Phoenix for trace evidence

Choose Langfuse or Phoenix for request traces and capacity-relevant records that you keep in your own storage. Langfuse is MIT licensed outside its ee directories, while Phoenix is under the Elastic License 2.0, which prohibits providing it to third parties as a hosted or managed service. If you run it only for your own staff, that restriction may not apply, but confirm the terms with counsel before offering traces to customers. Decide retention for trace data early, since it becomes audit evidence.

Kubernetes, Prometheus and backup tooling

Kubernetes and Prometheus carry the availability evidence: scheduling, replica counts, GPU utilization and the metrics vLLM publishes at /metrics. vLLM documents several Kubernetes deployment options, including native manifests and Helm, so pick one and record why. Backup tooling must cover model weights, serving configuration, gateway database and key records. Restore tests are the evidence most often missing, so schedule them and keep the results. Confirm each tool's license and support terms in your own environment before you adopt it.

How to show capacity is monitored, forecast and acted on

Criterion A1.1 asks the organization to maintain, monitor and evaluate current processing capacity and use of system components, so it can manage demand and add capacity. The auditor will look for three linked things: measured usage, a forecast, and changes made because of that forecast. For an in-house model service, the components to measure are the GPUs themselves (memory, utilization and temperature), the KV cache, request queue depth, token throughput and latency. vLLM exposes request counts, token throughput, latency histograms and KV cache use in Prometheus format at /metrics, so the metrics pipeline can collect them from day one. Confirm with your own team which GPU-level metrics your exporter provides, since the vLLM metrics page covers the engine, not the hardware.

Multi-GPU scaling changes what capacity means. vLLM splits a model across GPUs with tensor parallelism inside a node and pipeline parallelism across nodes, and it uses Ray or its own multiprocessing backend for multi-node runs. The vLLM security documentation states that node-to-node traffic is insecure by default and must sit on an isolated network. Capacity planning therefore has to name the parallelism layout, the node count and the interconnect for each model, because adding a GPU does not always add usable throughput. Write down a headroom threshold per model and per endpoint, such as the sustained utilization or queue depth at which the team must act. The threshold itself is a management decision, so record who approved it and when it was last reviewed.

The evidence trail is the metric history plus a written forecast review. Keep the exported metrics for the full audit period, record each forecast with its assumptions (user growth, new models, longer context lengths), and log the review meeting with the decision taken: buy, rent, reschedule or do nothing. A Type 2 examination tests operating effectiveness over a period, so a single forecast at the start of the year will not show the control working. Ask your partner to set up dashboards and a dated review record in the first sprint, and confirm with your auditor how often they expect reviews to occur.

Backing up model artifacts and configuration so A1.2 recovery can be shown

This sequence produces an inventory of everything the inference service needs to run, hashed copies of those items in a separate location, and dated restore runs that show the team can rebuild the service and state the recovery times it actually achieved.

Inventory weights, engine images and gateway configuration
The platform team lists every component the service depends on: model weight files and their source and checksum, the vLLM container image or package version, any Ollama or llama.cpp model files, the LiteLLM proxy configuration, routing settings, model-to-endpoint mappings, reverse proxy allowlist rules, and the secret references that hold the master key. Record the exact versions in a controlled register. Confirm in your own environment which items are stored outside the cluster, such as in a model registry, Helm values repository or object storage, because those locations need their own backup. The step is done when a reviewer can rebuild the list from the register without asking the engineer who set it up.
Store hashed copies in a separate location
Copy the inventoried artifacts to a backup location that is isolated from the production network and from the account or cluster that runs inference. Compute and record a cryptographic hash for each file and image digest so that a later restore can prove the copy matches the original. Encrypt the backup store and restrict write access to a small group separate from the operators who run the service. Confirm with your security team which storage, encryption and retention settings your backup policy requires, since those choices are yours to make. The step is done when the hash manifest is stored alongside the copies and a spot check of one artifact matches its recorded hash.
Back up the PostgreSQL virtual key and budget store
LiteLLM stores virtual keys, per-key and per-team budgets, rate limits and model restrictions in PostgreSQL, so that database is the most state-heavy part of the service. Take a logical or physical backup on a schedule your operations team sets, and record the backup time, size and checksum. Keep the LITELLM_MASTER_KEY in a secret manager and back up its reference, not the plain value in a document. Confirm the backup method and retention with your database team and the schedule with the owner of the availability commitment. The step is done when a backup file exists for a recorded time and its checksum is in the register.
Restore the stack into an isolated environment
Rebuild the service in a separate environment using only the backed-up artifacts, hashes, container digests and configuration, without reaching into the production system for anything. Restore the PostgreSQL backup, deploy the pinned engine version, load the weights, apply the gateway configuration and confirm the endpoints answer a test request. Note every point where the restore needed information that was not in the inventory, because those gaps are findings to fix. The step is done when the restored service returns a valid response from the test model and the team has a list of any missing items.
Record recovery times and file a dated restore report
During each restore run, time each stage from the start of the procedure to a working endpoint, and record who ran it, which backup versions were used, and which steps needed manual intervention. Write a dated restore report and store it with the evidence set. Do not quote a recovery duration in the report that you have not measured in your own environment, and compare the measured time against the recovery objective your organization has committed to. The step is done when the report exists, is signed by the reviewer, and identifies the actions that remain open.
Schedule and document the recurring restore test
Set a recurring restore test on a cadence that your change and risk owners approve, and assign an owner for it. Each test repeats the restore from the separate location, updates the inventory for any changes to models, engine versions or gateway settings since the last run, and produces a new dated report. Keep prior reports so an assessor can see the history of tests and the corrections made after each one. Confirm the cadence and scope with your auditor before the observation period begins, since the evidence an assessor expects depends on your commitments. The step is done when the test calendar, owner and report history are in the compliance repository.

Protecting endpoints the engine leaves open

The vLLM server exposes OpenAI-compatible routes such as /v1/chat/completions and /v1/embeddings, and it also exposes operational routes. The --api-key option protects the endpoints under /v1, /v2, /inference and /cohere, but the vLLM security documentation states that it does not cover /invocations, /pause, /resume, /abort_requests, /metrics or /health, and warns against relying on it alone. Those open routes matter because a caller could pause or abort work, or read metrics that reveal load and usage patterns.

Allowlist at a reverse proxy: Place a reverse proxy in front of the server and forward only the routes meant for users, as the vLLM security docs recommend.
Block operational routes from users: Keep /pause, /resume and /abort_requests reachable only from an administrative network segment, if at all.
Scrape metrics from a restricted path: Prometheus scrapes /metrics, so allow that route only from the monitoring host.
Keep the API key in a secret store: Load the key from a secret manager rather than a shell history or a plain configuration file.
Test the proxy rules: Send requests to each blocked route from a user network and record the refused responses as evidence.

Confirm with your engineering team that the proxy rules match the vLLM version in production, since route lists change with each release.

Evidence auditors ask for and the component that produces it

This table maps each evidence item an auditor is likely to request for an in-house model service to the component that generates it and a retention note. Read it as a collection checklist: confirm each retention period with your auditor before you set schedules, and note that the EU AI Act log expectation applies only where that regulation applies to your system.

Evidence item	Producing component	Retention note
Capacity reviews and forecasts	Written review records covering GPU utilization, queue depth and growth projections, drawn from monitoring data	Confirm the period with your auditor; keep dated, signed reviews that show the forecast was acted on
Prometheus metric exports	vLLM /metrics endpoint scraped by Prometheus (request counts, token throughput, latency, KV cache use)	Retain exports covering the full audit period; confirm scrape and storage retention with your auditor
Backup logs and restore test records	Backup tool job logs for model artifacts and gateway configuration, plus signed restore test results	Confirm with your auditor how long backup and restore evidence must cover the examination period
Change tickets for engine and gateway upgrades	Ticketing system linked to vLLM and LiteLLM release notes, test results and deployment records	Keep tickets with approvals and test evidence for the full examination period; confirm the period with your auditor
Access reviews of virtual keys and admin users	LiteLLM virtual key and user listings, exported on a schedule and reviewed by an accountable owner	Retain each review with the reviewer's sign-off; confirm the period with your auditor
SSO configuration	LiteLLM admin UI SSO settings for Google, Microsoft Entra ID, Okta or generic OIDC, with configuration snapshots	Keep dated configuration exports and change records; confirm retention with your auditor
Network allowlist configuration	Reverse proxy rules limiting exposed endpoints, including blocking vLLM paths such as /metrics and /health from users	Keep versioned rule files and approval records; confirm retention with your auditor
Gateway request logs	LiteLLM proxy logging, with logs held in the organization's own storage	Where the EU AI Act applies, Article 19(1) expects at least six months of logs for high-risk systems unless other law sets a longer period; confirm applicability with counsel

What open-source tooling does not satisfy on its own

Running vLLM or LiteLLM well does not produce a SOC 2 report. The engine and gateway generate technical signals, but the description of the system, the periodic reviews and the vendor oversight are management activities that an examiner will test separately.

Policies and the system description

Management writes the system description that the examiner evaluates against the AICPA's 2018 description criteria, and it must state the model service's boundaries, components, data flows and the controls around them. A configured vLLM server does not describe itself. Check that the description names the GPU nodes, the gateway, the key store, the model artifact repository and the monitoring stack, and that written policies cover each one.

Operating effectiveness across the period

A Type 2 report tests whether controls worked over a period, not only on a date. For capacity, backup and access controls, that means dated evidence across the whole window: forecast reviews, restore test records, key rotation logs and alert responses. Ask your partner how evidence is collected continuously so the audit window does not depend on screenshots taken in the final weeks.

Reviews and sign-off by people

Dashboards show metrics, but a person must review them, decide on capacity changes and approve exceptions. Assign named owners for GPU headroom reviews, access recertification of gateway admins and model release approvals, and record their sign-off with dates. Tools cannot attest that a review happened or that its conclusion was reasonable.

Model source and subprocessor management

Open-weight models come from public hubs, and container images and engine releases come from upstream projects. Each source is a third party in your system boundary, so track which weights were used, from where, under which license and who approved them. Confirm how you review upstream vendor reports and changes, and document how a new engine release is tested before it reaches production.

Proposed attestation standards

The AICPA Auditing Standards Board exposed proposed revisions to its baseline attestation standards in February 2026. If adopted, they would apply to engagements beginning on or after June 15, 2029. Confirm the current status with your auditor and plan for the evidence and representations they may require, rather than treating the proposal as settled.

Questions SaaS teams ask before adding a model service to SOC 2 scope

What does it cost to bring a self-hosted model service into SOC 2 scope?

The cost comes from four places: GPU or accelerator capacity, the surrounding infrastructure, staff time to operate and monitor it, and the work to collect evidence for the auditor. vLLM itself has no paid edition, so the largest recurring line is usually the hardware you run or rent, and the largest one-time line is usually the readiness and control work. Ask any partner for a quote that separates one-time items (design, configuration, control documentation, first evidence cycle) from recurring items (monitoring, patching, support retainer, hardware leases or power). A SOC 2 report also has a fee charged by the CPA firm, which is outside any implementation quote and should be requested separately.

How long does a SOC 2 readiness pass for a model service take?

The duration depends on how many systems and controls the model service touches, how mature your existing SOC 2 program is, and how much of the capacity, backup and access evidence already exists. A team that already runs monitoring and access reviews for other systems usually has less to build than one starting from scratch. We do not publish a fixed duration, and you should confirm the readiness timeline with your CPA firm, since the auditor decides the observation period for a Type 2 report. Ask your implementation partner for a written scope with milestones before work starts.

Can our own engineering team run this without an outside partner?

Many teams can, provided they already operate Kubernetes or GPU hosts, have someone who owns the SOC 2 control environment, and can staff on-call for the inference tier. The harder parts are rarely the install. They are writing the capacity forecasting procedure, testing restores of model artifacts, and producing dated evidence that a CPA can sample across the review period. A partner is most useful for a fixed-scope readiness pass and handover, while your team keeps ownership of the control decisions. Be honest with yourselves about on-call capacity before you decide.

Can DeployOpen issue our SOC 2 report?

No. DeployOpen prepares systems, controls and evidence for compliance assessments, but it does not issue audit reports, attestations or certifications. A SOC 2 report is issued by an independent CPA firm after its examination of your system description and controls. DeployOpen's role is to help you configure the model service, document how availability, backup and access controls operate, and organize the evidence your auditor will ask for. Your auditor decides what it will test and how, so involve them early in scope decisions.

Should we aim for a SOC 2 Type 1 report before Type 2?

A Type 1 report describes your system and evaluates whether controls are suitably designed as of a specific date, while a Type 2 report adds an opinion on whether those controls operated effectively over a period. Many organizations first obtain a Type 1 to test the design, then build the operating history needed for a Type 2. For a new model service, that sequence lets you fix capacity monitoring and backup gaps before the observation window starts. Whether it suits you depends on your customers and contracts, so confirm the expected report type with your CPA firm and the people who receive the report.

Do LiteLLM's SSO limits matter for our user count?

They can. According to LiteLLM's documentation, SSO for the admin UI is free for up to 5 users from v1.76.0, and an Enterprise license is needed beyond that. Enterprise also adds SCIM, audit logs and team RBAC, which auditors often look for when they test access control. If your SOC 2 access review depends on centrally managed identities and audit trails for administrators, confirm your licensing against your headcount and the current LiteLLM terms before design work begins. Ask the vendor directly how the license applies to your deployment.

What evidence will an auditor expect from GPU capacity monitoring?

Under the availability criteria, the auditor will look for a process that measures current use of capacity, forecasts demand and results in changes when the forecast shows a shortfall. In practice this means retained metrics from the inference server, such as the Prometheus metrics vLLM exposes at /metrics covering request counts, token throughput, latency and KV cache use, plus dated capacity reviews, alert records and records of any capacity changes. Confirm with your CPA firm which periods and samples they will test, since the expected evidence format is theirs to define.

Which endpoints must stay off the public network for the audit?

The vLLM security documentation states that its --api-key option protects only some endpoints, and does not cover /metrics, /health, /pause, /resume, /abort_requests or /invocations. The documentation recommends a reverse proxy that allowlists only the endpoints meant for users. Your SOC 2 access control narrative should show which paths are reachable, who can reach them, and how that is tested. Confirm the current list of unprotected endpoints against the release you deploy, since engine versions change about every two weeks.

What a SOC 2-ready inference build covers in practice

A SOC 2-ready build starts with a written scope agreed before any work begins, naming the systems, environments and teams the model service covers. That scope defines which components the auditor will read: the LiteLLM gateway or other AI gateway, the serving engine (vLLM, Ollama or llama.cpp), the reverse proxy in front of it, the GPU nodes, the metrics stack and the backup target. DeployOpen configures each of these in the client's own infrastructure, then documents the configuration so the organization can show what is in scope and how it is operated. Decide early whether the model service sits inside the existing SOC 2 system boundary or is carved out as a subservice, because that choice changes which controls the report must describe.

Each build area maps to a deliverable. Dashboards cover request counts, token throughput, latency histograms and KV cache use from the Prometheus-format /metrics endpoint on the vLLM API server, which supports the capacity measurement and forecasting that the availability criteria expect. Backup and restore runbooks cover model artifacts, container images, gateway configuration and the PostgreSQL database that holds LiteLLM virtual keys, with restore tests recorded as evidence. The access control design defines who can create keys, set budgets and reach each endpoint, and the proxy allowlists only the routes meant for users, since the vLLM --api-key option does not protect every path.

The deliverable that ties these together is an evidence index that lists each control, the criterion it supports, the system owner, the artifact produced and where that artifact is stored. Confirm with your auditor which points of focus they will test and how often they expect evidence, because the Trust Services Criteria treat points of focus as guidance rather than separate requirements. DeployOpen prepares the systems, controls and evidence for that review. It does not issue audit reports, attestations or certifications; those come from the independent CPA firm that performs your SOC 2 examination.

How DeployOpen scopes, builds and supports SOC 2 evidence for model services

DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside the client's own infrastructure, so the model service and its evidence sit where the SOC 2 scope expects them. We prepare systems, controls and evidence for the auditor, while the independent CPA firm issues the report.

How we scope

Agree in writing which inference servers, gateways, GPU nodes and admin tools fall inside the SOC 2 system boundary
Map each capacity, backup and access control to the A1.1, A1.2 and CC6 criteria it supports
Confirm with your auditor the period, report type and evidence samples before build work starts

How we build

Deploy the engine behind a reverse proxy that allowlists only user-facing endpoints, with monitoring of /metrics
Configure capacity dashboards from Prometheus metrics and document the thresholds that trigger scaling
Set up backups of model artifacts, container images and gateway configuration, then test restores and record the results

How we hand over

Deliver a system description, runbooks for capacity review and restore testing, and an evidence calendar
Walk your operations and security teams through access reviews, change records and engine upgrade procedures
Provide a written list of open items, owners and the evidence each control should generate going forward

How we support

Offer ongoing support, upgrades and continuous testing as a retainer, sized to the written scope
Review each vLLM or gateway release for control impact before it reaches production
Prepare evidence packages for each audit period so your auditor can sample from a consistent set

Self-hosted LLM inference for SOC 2