Why self-hosted LLM inference changes the SaaS subprocessor list
Self-hosting keeps prompts and outputs inside infrastructure you control, so a model provider does not have to appear on your subprocessor list, but you take on the GPU capacity, uptime and operations that the provider used to carry. The roadmap now includes LLM features, customers are asking which subprocessors touch their data, and engineering has to commit to an availability figure that depends on hardware you may not yet own. The decision therefore turns on the controls your SOC 2 report will test. The AICPA 2017 Trust Services Criteria cover security, availability, processing integrity, confidentiality and privacy, and the security common criteria CC1 to CC9 apply to every SOC 2 examination. An in-house model service pulls several of these into scope at once: access control on inference endpoints (CC6), capacity monitoring and recovery of model artifacts (availability), and change management when the engine is upgraded (CC8). Confirm with your auditor which categories the report will include before you design the service.
Two engines are the main candidates. vLLM is an Apache-2.0 serving engine built for high-throughput, multi-GPU production use, and its server exposes OpenAI-compatible endpoints such as /v1/chat/completions. Its own security documentation says the --api-key option does not cover every endpoint, including /metrics and /health, so a reverse proxy that allowlists only user-facing routes belongs in your design from the start. Ollama is an MIT-licensed runtime with a llama.cpp backend that suits smaller teams and single-node deployments; it binds to 127.0.0.1 on port 11434 by default and its local API does not require authentication, so exposing it needs the same proxy layer. Confirm each engine's current behavior against the release you plan to run, because both ship updates every few weeks.
Neither engine carries a license fee for self-hosting. vLLM has no paid edition, so your cost sits in accelerator capacity, the infrastructure around it and the people who run it. Ask any implementation partner to price these components separately: GPUs (purchased or leased), power and facilities, the engineering time to build and monitor the service, and on-call coverage that your availability commitment requires. These costs are largely recurring, and utilization is the variable that moves them most. Ask for a written scope that names who patches the engine, who rotates keys and who owns the SOC 2 evidence for the inference tier before you sign.
Which requirements a SaaS inference service must meet
This table maps the requirements a SaaS vendor faces when it serves LLM features from its own GPUs, with what each one asks for and what it means for the build. Read each row as a question to settle before architecture work starts. The main takeaway is that the inference engine alone does not meet these requirements, so the gateway, network and evidence design carry most of the work.
| Requirement | What it asks for | What it means for the build |
|---|---|---|
| SOC 2 scope | Security common criteria CC1 to CC9 apply to every SOC 2 report. The inference service falls inside the system description if it processes customer data. | Decide early whether the model service is in scope. A Type 1 report covers control design at one date, while a Type 2 report tests operating effectiveness over a period, so start evidence collection before the observation window opens. |
| Processor terms (GDPR Article 28(3)) | If the vendor still has EU customers, its contracts must bind it to process personal data only on documented instructions and to apply Article 32 security measures. | Document how prompts and outputs are stored, retained and deleted, and list any sub-processing. Self-hosting removes a model provider from the subprocessor list but not your own obligations to customers. |
| EU Data Act switching duties | As a data processing service provider, the vendor must let customers switch to another provider or to on-premises infrastructure, with a maximum two-month notice period and a 30-day transitional period (Articles 23, 25 and 50). | Design exports of tenant configuration, prompt templates, logs and any fine-tuned artifacts in open formats. Confirm with counsel which services fall under the Act for your offering. |
| Endpoint authentication | vLLM's --api-key option protects only /v1, /v2, /inference and /cohere paths. It does not cover /invocations, /pause, /resume, /abort_requests, /metrics or /health, and the docs advise against relying on it alone. | Place a reverse proxy in front of the engine that allowlists only the endpoints customers call, and authenticate every request there. Test each open path from outside the cluster. |
| Multi-node traffic | Communication between nodes in a multi-node deployment is insecure by default, without authorization or encryption, and must be protected by an isolated network. | Put tensor and pipeline parallel traffic on a dedicated, segmented network. Confirm how Ray or the multiprocessing backend is secured in your chosen topology before go-live. |
| Availability evidence (metrics) | vLLM exposes Prometheus-format metrics at /metrics covering request counts, token throughput, latency histograms and KV cache use. | Scrape these metrics into your monitoring stack and keep alert history. The /metrics path must stay internal, so expose the dashboard to operators, not to customers. |
| Per-tenant rate limits | Each customer needs its own request, token and concurrency limits so one tenant cannot degrade others, and usage must be attributable to that tenant. | Handle limits at an AI gateway or API gateway with per-tenant keys, not inside the engine. Ask the implementation partner how usage records are kept for billing disputes and audits. |
Which inference engines and gateways fit a multi-tenant SaaS
A multi-tenant SaaS product needs an engine that serves many customers at once, a gateway that enforces per-tenant limits and keys, and tracing that shows what each tenant consumed. The cards below cover the main open-source candidates for each layer and what to confirm before any of them sits in front of paying customers.
vLLM as the production engine
vLLM is the serving engine most likely to carry multi-tenant production traffic on your own GPUs. It exposes OpenAI-compatible chat, completion, embedding and responses endpoints, so product code written against the OpenAI API can point at it. Scaling across GPUs uses tensor parallelism within a node and pipeline parallelism across nodes. Confirm which GPU vendor and driver stack your hardware needs, and whether your Kubernetes platform will use the vLLM production-stack, KServe or another option from the vLLM deployment list.
Ollama for smaller tenants
Ollama is an MIT-licensed runtime that suits smaller deployments, dedicated tenant nodes and edge servers. It offers an OpenAI-compatible REST API, but its default concurrency is one parallel request per model (OLLAMA_NUM_PARALLEL), which is too low for shared production traffic unless you raise it and size memory to match. Its local API has no authentication, so it must sit behind a proxy that enforces keys. Confirm the throughput your tenants need before choosing it over vLLM for the main service.
LiteLLM for per-key budgets
LiteLLM is an AI gateway that presents many model backends in OpenAI format and records cost, logs and guardrail events. For a SaaS vendor it is the layer that issues one virtual key per tenant, enforces budgets and routes requests across engine replicas. The enterprise directory is under a separate license, so confirm which LiteLLM features your product needs fall inside the MIT-licensed core and which require a commercial agreement with the project.
APISIX or Kong for tenant rate limits
Rate limiting per tenant, per key and per endpoint belongs at an API gateway in front of the AI gateway. Apache APISIX (Apache-2.0) and Kong Gateway (Apache-2.0) both support AI-oriented routing and limits, and both are Apache-licensed. Choose one, not both, and confirm that its limiting plugins count tokens or only requests, because GPU cost follows token volume. Also confirm that the gateway allowlists only the user-facing endpoints of the engine, since vLLM's own --api-key does not cover every route.
Langfuse for per-tenant tracing
Langfuse is an open source LLM observability and evaluation platform that records traces, prompts, outputs and cost per request. Tagging each trace with a tenant identifier lets you answer which customer drove a cost spike and reconstruct an incident for a customer audit request. The core is MIT-licensed, but its ee directories are separately licensed. Confirm where trace storage lives, how prompt content is retained and redacted, and who in your organization can query tenant data.
Deployment patterns for SaaS: shared pool, dedicated tenant or hybrid
A shared pool runs one set of vLLM servers behind a gateway that authenticates each tenant, applies per-tenant rate limits and token budgets, and routes requests to whichever GPUs have capacity. It gives the best utilization because bursty traffic from many customers fills the same batches, and it keeps the fixed cost of GPUs spread across the whole customer base. The trade-off is that a single noisy tenant can affect latency for others unless quotas are enforced before requests reach the engine.
- Gateway-enforced quotas: Place a gateway such as LiteLLM or Apache APISIX in front of the engine to hold per-tenant keys, requests per minute and token budgets.
- Endpoint allowlisting: The vLLM
--api-keyoption does not cover/metrics,/healthor/invocations, so put a reverse proxy in front that exposes only the endpoints customers need. - Shared failure domain: An outage or a bad model update affects every tenant at once, so the uptime you can promise depends on redundancy across nodes.
- Per-tenant metering: Token counts per tenant feed billing and usage evidence, and they need to come from the gateway or the engine logs.
- Data separation in logs: Prompts from many customers share storage, which makes access control and retention rules part of the design.
Before committing, confirm with your architect how quotas are enforced at peak load and what happens to in-flight requests when a node fails.
A dedicated pool gives a large or regulated customer its own GPUs and its own model deployment, so its prompts never share a batch or a queue with other tenants. This is often the answer when a contract requires isolated compute or when a customer's security review rejects shared inference. The cost is lower utilization, because reserved GPUs sit idle whenever that customer is quiet, and that idle capacity is paid for whether or not it is used.
- Utilization as the main cost driver: A pool sized for one customer's peak runs well below capacity at other times, so the per-request cost rises as the pool shrinks.
- Isolation by design: Separate endpoints, separate keys and separate storage make it easier to show that one customer's data stays apart.
- Scope creep in SOC 2: A dedicated pool that you operate for a single customer may still fall within your own system description, so confirm the boundary with your auditor.
- Capacity planning per customer: Each pool needs its own forecast for growth, model updates and failover.
- Pricing alignment: Ask whether the customer pays for reserved capacity, a dedicated tier or a per-token rate with a minimum commitment.
Ask your team to model the idle-capacity cost for each dedicated pool against the price the customer will pay before agreeing to one.
In a hybrid model, a customer that needs its inference inside its own network receives a single-tenant deployment on its premises or in its own cloud account, built from the same vLLM image and configuration you run in your environment. Your team keeps one build pipeline, one set of tests and one upgrade path, while the customer controls where the hardware sits. The work shifts toward packaging, remote support access and version alignment across environments.
- One image, many targets: Build a single vLLM image and deployment manifest, then vary only model, GPU count and endpoint settings per customer.
- Tensor parallelism within a node: vLLM splits a model across the GPUs in one server, which is the usual first step when a model does not fit on one card.
- Pipeline parallelism across nodes: When a model spans several servers, pipeline parallelism divides layers between them, and vLLM uses Ray or its own multiprocessing backend to coordinate them.
- Isolated networking for multi-node runs: The vLLM security documentation states that communication between nodes is insecure by default, without authorization or encryption, so nodes must sit on an isolated network.
- Upgrade coordination: vLLM ships minor releases about every two weeks, so define which versions each customer runs and how fixes reach them.
Confirm with the customer who operates the hardware, who holds the keys to the nodes and how your team gets access for patching and incident response.
How a SaaS team moves from pilot to production inference
The sequence turns a working pilot into a model service that serves paying tenants within agreed latency and tenant limits, with the security controls and SOC 2 evidence in place before customers depend on it.
Agree scope and the tenant model in writing
The product, security and platform leads agree which features call the model, which models and context lengths are in scope, how tenants are separated (shared pool, dedicated capacity or priority tiers), and which data may be sent to the endpoint. DeployOpen documents these decisions in a written scope before any hardware is ordered or configured. Done means every decision has a named owner and the SOC 2 system boundary draft lists the inference service, its logs and its model artifacts.
Size GPUs and load test against target concurrency
The team turns the agreed concurrency, time-to-first-token and response-length targets into a GPU plan, covering accelerator type, number of GPUs per model replica and whether tensor parallelism is needed within a node. A load test with realistic prompt and output lengths runs against the candidate configuration, and the results are compared with the targets. Ask your vendor or hardware supplier to confirm which accelerators your chosen vLLM release supports. Done means a signed-off test report shows the configuration meets the targets with headroom for peak tenant traffic.
Place a gateway and allowlist in front of vLLM
vLLM's own API key option does not protect every endpoint, and the vLLM security documentation recommends a reverse proxy that exposes only the endpoints users need. The team places an authenticating gateway in front of the engine, issues per-tenant keys, sets per-tenant rate limits and blocks administrative and metrics routes from tenant traffic. Multi-node communication is kept on an isolated network. Done means tests confirm that unlisted routes return denial from outside the cluster and that each tenant key is limited as designed.
Build metrics, alerting and capacity dashboards
vLLM exposes Prometheus-format metrics covering request counts, token throughput, latency histograms and KV cache use. The team scrapes these metrics internally, builds dashboards for per-tenant consumption and GPU saturation, and sets alerts for latency drift, queue growth and cache pressure. Capacity thresholds are written down so that someone acts before tenants feel degradation. Done means alerts have been fired in a test, the on-call runbook names an owner for each alert, and the capacity dashboard is reviewed on a set cadence.
Map SOC 2 controls and start collecting evidence
Before the observation window opens, the team maps the inference service to the relevant security common criteria and availability criteria in the SOC 2 scope, including access reviews for endpoints and admin consoles, change management for model and engine updates, backup of model artifacts and configuration, and log retention. Evidence collection starts now, so the Type 2 period is not the first time controls are exercised. Your SOC 2 auditor confirms the control set and the evidence format, because DeployOpen prepares controls and evidence but does not issue the report. Done means each control has an owner, a procedure and an evidence sample that has been generated at least once.
Run an upgrade process for a fast release cadence
vLLM ships new minor releases about every two weeks, so the team cannot treat the engine as a fixed component. The process covers a staging environment that mirrors production, regression tests for the OpenAI-compatible endpoints the product uses, a load test on each candidate release, a documented rollback to the previous image, and a decision record for each upgrade. Model weight updates follow the same change path. Done means one full upgrade cycle has been run end to end in staging, with the change record and rollback test stored as evidence.
Cut over with a fallback and an exit plan
Traffic moves to the self-hosted endpoint in stages, starting with internal users and a small set of tenants, while the existing model path stays available as a documented fallback. The team confirms the fallback can be switched on by a named role, and the exit plan describes how tenant data, logs and model artifacts would move if the service is retired. Customer contracts and subprocessor documentation are updated to match the final architecture. Done means the staged rollout passed its success criteria, the fallback switch has been tested, and the exit plan has been reviewed by legal and security.
Evidence a SOC 2 auditor will ask for and which component produces it
This table maps each SOC 2 evidence area for an in-house model service to the component that produces it, such as the vLLM server, the gateway, GPU nodes or the ticketing system. Read it as a checklist of what to collect and where it lives, then confirm the scope and control design with your auditor. DeployOpen prepares this evidence, but the independent auditor decides what it proves.
| Evidence area | Component that produces it | What the auditor will expect to see |
|---|---|---|
| System description of the inference service | Architecture diagram covering vLLM servers, the API gateway, GPU nodes, model storage and network zones | A written boundary showing which components are in SOC 2 scope, which tenants and data flow through them, and which are carved out or provided by a subservice organization |
| Access control on user endpoints | Gateway access logs and allowlist config that expose only the /v1 endpoints meant for customers | Evidence that vLLM's --api-key is not the only control, and that /metrics, /health, /pause and /abort_requests are unreachable from tenant networks |
| Admin path and key management | Gateway key inventory, key rotation records and admin access logs for the GPU hosts | Proof of issuance, rotation and revocation of API keys per tenant, plus named, reviewed access to the nodes that run the engine |
| Availability and capacity | Prometheus metrics scraped from /metrics (request counts, token throughput, latency, KV cache use) and alert history | Retained metric history over the audit period, capacity thresholds that trigger action, and sample alerts with follow-up |
| Incident records | Ticketing system entries and post-incident reviews for GPU failures, queue saturation or endpoint outages | Tickets showing detection, response, root cause and corrective action for the audit period, including any tenant-visible downtime |
| Change management for engine and model upgrades | Pull requests or change tickets for vLLM version bumps and model weight swaps, with test results from a staging node | Approved changes before deployment, checksum or signature verification of new model files, and a rollback record for each change |
| Backup and recovery of model artifacts and configuration | Backup job logs for model weights, gateway config, vLLM launch arguments and infrastructure-as-code, plus restore test results | Evidence that backups run on schedule, are stored separately, and have been restored in a test with recorded results |
| Request and tenant logging for processing integrity | Gateway logs and vLLM request logs carrying tenant ID, key ID, model name, timestamps and token counts, stored in a central log system | Logs that link each request to a tenant, show completeness and retention, and restrict who can read prompt content |
What do SaaS teams ask before self-hosting LLM inference?
What does self-hosting LLM inference cost a SaaS company?
The engine itself carries no license fee, because vLLM is open source under Apache-2.0 and has no paid edition. The cost comes from four places: GPU or accelerator capacity (bought or rented), the surrounding infrastructure such as networking, storage and monitoring, implementation work, and the engineers who operate the service. GPU capacity usually recurs every month if rented and is a capital purchase if owned, so ask each bidder to separate one-time setup from recurring run costs. Utilization is the biggest driver: idle GPUs cost the same as busy ones. A written quote should itemize hardware, hosting, implementation scope and support hours, and state which costs you would carry if you later move to a hosted API.
How long does a self-hosted inference rollout take?
The schedule depends most on GPU procurement, so confirm delivery dates with your hardware vendor or cloud provider before anything else. Software setup, model loading, endpoint hardening and load testing follow once the hardware is racked or provisioned. Ask a partner to show the phases, the dependencies they cannot control and the acceptance tests that close each phase, all in the written scope. Duration figures should come from that scope, not from general claims about the market.
Can our own engineers run a self-hosted model service alone?
Many teams can, but they need people who understand GPU scheduling, CUDA or ROCm driver updates, model artifact handling and Kubernetes or systemd operations. Plan for an on-call rotation, because a single failed GPU node can take the service down. Assess your team against four questions: who patches the engine, who watches the /metrics endpoint, who handles capacity requests, and who owns the runbook. If the answers are unclear, a retainer for upgrades and support can cover the gap while your staff learn the stack. DeployOpen plans, implements, hardens, backs up, upgrades and documents self-hosted platforms in your own infrastructure, and support is available as a retainer.
Can a SOC 2 Type 2 period start before the model service is live?
A Type 2 report covers operating effectiveness over a period, so controls need to be in place and running for that period, not only designed. The practical question is which date your auditor will treat as the start of the in-scope system. Discuss this with your auditor early, because the service description must cover the model endpoints, GPU nodes, model artifacts and access paths. A Type 1 report can document design as of a date while the Type 2 period builds. Collect evidence from day one: access reviews for the endpoints, change records for engine and model upgrades, capacity monitoring and backup test results. Auditors, not DeployOpen, decide whether your controls meet the criteria.
What happens to Data Act switching duties if we run our own inference?
The EU Data Act has applied since 12 September 2025 and requires data processing service providers to let customers switch to another provider or to on-premises ICT infrastructure, with a maximum two-month notice period and a 30-day transitional period. From 12 January 2027, providers may not impose switching charges. If you run inference yourself, you are the customer and the provider, so these duties mainly shape what you may promise your own customers and what you must check in any hosted fallback. Keep an exit plan that covers model weights and their licenses, container images, prompt templates, logs and evaluation data. Confirm with counsel how the Data Act applies to your contracts.
Does a single GPU node put our uptime promise at risk?
Yes, if the service runs on one node with no redundancy. A GPU failure, driver fault or maintenance window stops inference for every tenant on that node. Uptime targets should match the architecture: a single node cannot support a high availability commitment on its own. Ask for a design with a second node or a failover path, health checks, and a capacity plan that keeps headroom for peaks. Multi-node vLLM deployments need an isolated network, because the vLLM security docs state that inter-node communication is insecure by default. Put the agreed availability figure in your customer contract only after load tests and failover tests confirm it.
How do per-tenant rate limits fit with the engine?
The inference engine does not give you tenant accounts, so per-tenant limits sit in front of it. An AI gateway such as LiteLLM or Apache APISIX can issue keys per tenant, enforce request and token budgets and log usage. Ollama can restrict concurrency per model with OLLAMA_NUM_PARALLEL and OLLAMA_MAX_QUEUE, but tenant-level fairness still needs a gateway. Check that the gateway, not the application, enforces limits, and that rejected requests are logged for your SOC 2 evidence.
What to ask an implementation partner before signing
Start with GPU sizing, because it sets your margins and your uptime. Ask how the partner derives capacity from your tenant mix: peak concurrent requests per tenant, prompt and output lengths, the model and quantization chosen, and the context lengths you promise. Ask for the throughput and latency numbers they measured on your hardware, not vendor claims, and for the headroom they keep for a node failure. Ask how KV cache pressure was measured for long-context tenants, since vLLM exposes that through its metrics endpoint and a sizing that ignores it will fail under real load. Confirm what hardware is in scope and what happens to capacity when a GPU node is lost.
Next, ask how they close the endpoints the engine leaves open. The vLLM --api-key option does not protect /metrics, /health, /pause or /abort_requests, and its security documentation recommends a reverse proxy that allowlists only the routes meant for customers. Ask who writes that allowlist, how per-tenant keys are issued and rotated, where tenant identity is enforced for rate limits, and whether the partner tests those paths from outside the cluster. Also ask how node-to-node traffic in a multi-node deployment is isolated, because the same documentation states that traffic is insecure by default.
Then ask how engine releases and model updates reach production. vLLM ships minor releases about every two weeks, so ask for the patch policy, the staging path, the regression tests that compare outputs and latency for each tenant tier, and the rollback procedure for a bad release. Ask who approves a model change that could alter customer-facing behavior, and how model weights and container images are verified before promotion. On SOC 2, ask which evidence they will produce for change management and which they will not: a partner that offers a SOC 2 attestation for your service is overstepping, since only an independent CPA firm issues the report and the partner can only supply controls and records. Ask who operates the service after handover, which roles and on-call hours the retainer covers, what the handover runbook contains and how your own staff are trained to run it.
How DeployOpen works on this
DeployOpen is a Chainscore Labs company that plans, implements, hardens, documents and supports self-hosted open-source platforms inside the client's own cloud accounts, data centers or air-gapped networks. For a SaaS inference platform, that means your GPUs, your endpoints and your evidence, with work agreed in a written scope before it starts.
How we scope
- Written scope covers model list, context lengths, GPU capacity and expected concurrency per tenant.
- Scope defines which endpoints customers reach and which stay internal to the platform.
- Scope names the SOC 2 systems, controls and evidence the platform must produce, agreed with your auditor.
- Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts.
How we build
- Deploy the serving engine on your GPU nodes with a reverse proxy that exposes only user-facing endpoints.
- Configure per-tenant rate limits, authentication and request logging at the gateway layer.
- Test endpoint exposure, access controls and tenant isolation, including the engine's unprotected routes.
- Set up metrics for request volume, token throughput, latency and KV cache use for capacity review.
How we hand over
- Document architecture, network isolation, configuration and the upgrade procedure for engine releases.
- Back up model artifacts, container images and configuration, and test restoration before handover.
- Prepare runbooks for capacity alerts, failover and access reviews that your operations team can run.
- Deliver evidence packs mapped to the agreed SOC 2 controls, noting that auditors make the attestation.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers after handover.
- Retainers can cover engine and model updates, tested against your tenant rate limits before release.
- Periodic security testing can cover the inference endpoints, gateway and agent-facing integrations.
- Confirm response expectations and service levels in the retainer agreement, since none are set here.
