Why LLM observability for SaaS products differs from internal tools
SaaS teams shipping LLM features to customers face a different problem from teams observing an internal assistant. Prompts and outputs belong to many tenants at once, enterprise buyers ask where those traces are stored and who can read them, and security questionnaires expect concrete answers. For most such teams, self-hosted Langfuse is the usual fit for keeping traces in the company's own infrastructure, because its core features are available under the MIT license and the same codebase runs in self-hosted and cloud deployments. Arize Phoenix is a capable tracing and evaluation tool, but its Elastic License 2.0 prohibits providing the software to third parties as a hosted or managed service, so a SaaS vendor that exposes trace views to customers needs to read that license with counsel before choosing it.
The multi-tenant part shapes the design. Each trace should carry a tenant identifier so cost, latency and error rates can be attributed per customer, and the team needs to decide whether customers see their own traces at all. Project-level roles, data retention policies, audit logs and server-side data masking are among the features the self-hosted documentation lists as requiring an Enterprise license key, so confirm which of these your tenant isolation and buyer commitments depend on before the architecture is fixed. Ask your implementation partner to show how tenant identifiers flow from the application through OpenTelemetry or the SDK into the stored records.
Enterprise buyers will also ask about sub-processors, data location and deletion. Storing traces in your own cloud account, with ClickHouse and S3-compatible storage under your control, gives you a clearer answer than a third-party tracing service, but the answer only holds if backups, restore tests and access paths are documented too. Confirm with your compliance lead which SOC 2 or ISO/IEC 27001 controls the observability stack will be cited under, and treat the questionnaire as the first draft of your trace retention and access policy.
What SaaS buyers and regulators require from LLM traces
This table maps each requirement that SaaS buyers and regulators place on LLM traces to its source, what it asks for, and where self-hosted Langfuse setups usually fall short. Read it as a checklist for deciding which obligations your product triggers before you choose storage, masking and license tiers. The main takeaway is that the duties depend on your role (provider, processor or ICT service provider) and on whether the feature is high-risk, so confirm those facts first.
| Requirement | What it asks for | Evidence to prepare |
|---|---|---|
| EU AI Act Article 12 and 19 logs | If the feature is high-risk, the system must technically allow automatic event logging over its lifetime, and providers keep logs for at least six months unless other law says otherwise | Log schema, retention settings in ClickHouse and object storage, and a documented retention period agreed with legal counsel |
| GDPR Article 28(3) processor terms | Where you process customer personal data in traces, the contract must limit processing to documented instructions and require Article 32 security measures | Data processing agreement covering trace storage, sub-processors, deletion on instruction and transfer rules |
| SOC 2 common criteria CC1 to CC9 | Security common criteria apply to every SOC 2 examination, covering control environment, risk, monitoring and logical access for the trace platform | Access reviews for Langfuse projects, change records for upgrades, and monitoring alerts on the ingestion pipeline |
| NIS2 and ICT service duties | Cloud, data center and managed service providers fall under NIS2 Annex I high-criticality sectors, so the tracing service may be in scope as supplier infrastructure | Confirm your classification with counsel, then document security measures and supplier risk for the observability stack |
| Data Act switching rights | Data processing service providers must let customers switch providers or move to on-premises infrastructure, with no switching charges from 12 January 2027 | Export procedure for traces and evaluations in an open format, and a tested migration path to another self-hosted instance |
| Per-tenant cost attribution | Usage and cost per customer must be traceable to support pricing, billing disputes and capacity planning | Tenant ID set on every trace, cost fields validated against model provider invoices, and a documented tagging scheme |
| Retention by tenant | Retention periods may differ by customer contract and by regulation, so one global setting may not fit all tenants | Retention matrix by data class, plus a purge job tested against backups |
| Masking of personal data | Prompts and outputs can carry personal data, so masking should happen before data reaches storage, using client-side controls | Masking rules in the SDK or OpenTelemetry Collector, sample traces showing masked fields, and a review of which fields remain |
| Tenant isolation | Customer traces must be readable only by authorized users of that customer and by no other tenant | Project-level separation, role mapping, and the license tier that provides project-level RBAC for self-hosted Langfuse |
| Langfuse enterprise license key features | Project-level RBAC, data retention policies, audit logs, server-side masking, and SCIM are enterprise features in self-hosted Langfuse | Written decision on whether each feature is needed, with the license cost and the owner of the license in your quote |
Which platforms fit a SaaS product and why
A SaaS product that ships LLM features to customers needs tracing that serves many tenants, storage that scales with ingestion, and licenses that allow the service to be offered to others. The platforms below play different roles in that stack, and their license terms decide which of them can sit in front of customers.
Langfuse as the tracing core
Langfuse is the platform most SaaS teams evaluate first for this role. Its core features and APIs are in the MIT-licensed version, and self-hosted deployments run the same codebase as Langfuse Cloud. Check which directories fall under the enterprise license, since project-level RBAC, data retention policies, audit logs and server-side masking need a license key. Confirm every feature your tenant isolation depends on before you design around it.
Arize Phoenix for internal use only
Phoenix is licensed under the Elastic License 2.0, which prohibits providing the software to third parties as a hosted or managed service. In a SaaS context, that means your engineers can run Phoenix for internal debugging and evaluation, but it cannot be the tracing backend your customers or tenant administrators reach through your product. Have counsel review the license against your exact architecture before you build a customer-facing trace view on it.
LiteLLM for cost and request logging
LiteLLM sits between your application and model providers as an AI gateway. It calls more than 100 LLM APIs in OpenAI format and adds cost tracking, guardrails, load balancing and logging upstream of the tracing platform. Tag each request with tenant and feature identifiers at this layer so cost per tenant can be attributed before traces reach storage. The gateway is MIT licensed except its enterprise directory, so confirm which features your design needs before you commit.
Dify and multi-tenant licensing
Dify is a capable application platform, but its modified Apache License 2.0 requires a commercial license to operate a multi-tenant service from the source code. If your product exposes Dify-built features to many customers from one deployment, that clause applies directly. Treat Dify as a licensing decision first and a technical choice second, and ask the vendor in writing how its terms apply to your hosting model.
ClickHouse as the storage engine
Self-hosted Langfuse stores high-volume trace data in ClickHouse, which is open source under Apache-2.0. Ingestion volume, not seat count, drives the sizing of this layer, so model trace volume per tenant and per feature before you choose node counts. Plan replication with ClickHouse Keeper and a tested backup path, because replication alone does not protect against human error or corruption. Confirm the supported version lines before you pick a release.
How ClickHouse sizing and cost per tenant work in practice
Ingestion volume, not user count alone, drives ClickHouse and worker load. A SaaS product with many quiet tenants can be lighter than one with a few agentic features that produce hundreds of spans per request. Langfuse's scaling guide lists minimum resources of 2 CPU and 8 GiB for ClickHouse, and recommends at least 16 GiB of ClickHouse memory for larger deployments. Before committing to a size, ask your partner to estimate traces, observations and spans per day from load tests of your real prompts, and to scale the worker containers on CPU load. Under heavy ingestion, Langfuse recommends splitting langfuse-web into separate ingestion and user interface deployments, so that a burst of SDK traffic does not slow the UI your customer success team uses.
Cost per tenant starts with tagging. Every trace should carry a tenant identifier from your application, set through the SDK or OpenTelemetry attributes, so that token usage, model cost and latency can be grouped by customer. Decide early which identifier is used, how it is hashed or pseudonymized if customer identifiers are sensitive, and who may query per-tenant views. Expect to reconcile these figures with your billing system, and confirm with your partner how they handle traces that arrive without a tenant tag. For major upgrades, the v3 to v4 guide advises about 3x ClickHouse disk headroom when backfilling historic data, so plan that space before any migration of an existing trace store.
Cost components fall into one-time and recurring groups. One-time work includes sizing, instrumentation of each application, evaluation setup, the initial backfill and restore testing. Recurring costs include compute and storage for ClickHouse, PostgreSQL, Redis or Valkey and S3-compatible storage, backups, staff time for upgrades, and any Langfuse Enterprise license if you need features such as project-level RBAC or data retention policies. Ask for a quote that separates these lines, states the assumed trace volume and retention period, and names who operates each component. Confirm the ClickHouse version your partner will support, since the support window for each release line is documented in the ClickHouse security policy.
How a SaaS team rolls out self-hosted tracing
The sequence moves from agreeing which features and tenants are in scope to a production deployment in the company's own cloud account, then to masking, evaluation gates and restore testing. Each phase closes with a named output that the next phase depends on.
Scope features, tenants and data before any install
The product, security and platform leads agree which LLM features, model providers, tenant tiers and environments send traces, and which data fields may enter a trace at all. Your DeployOpen contact or internal team records expected trace volume per feature, retention needs and which tenants are in scope for per-tenant cost reporting. The output is a written scope document that sets the boundary for every later phase, including whether any customer data is excluded from tracing entirely.
Prove the stack on a single VM
The team runs Langfuse with Docker Compose on one virtual machine in a non-production account. Langfuse documents this setup for testing only, with no high availability, so it proves connectivity, SDK integration and basic dashboards rather than resilience. The proof is complete when a test application sends traces that appear in the UI with the expected prompts, token counts and cost fields, and when the team has noted every configuration value it will need to reproduce in production.
Deploy production on Kubernetes in your cloud account
Platform engineers deploy Langfuse on Kubernetes using Helm or Terraform in the company's own cloud account, as Langfuse documents for production on AWS, Azure and GCP. The deployment runs Langfuse Web and Langfuse Worker with PostgreSQL, ClickHouse, Redis or Valkey and S3-compatible blob storage, sized against the scaling guide's minimums and the expected ingestion volume. The output is a production environment with network controls, SSO configured, and a sizing record that shows how ClickHouse storage grows with trace volume.
Instrument applications and mask sensitive fields
Developers add the Langfuse SDK or OpenTelemetry instrumentation to each in-scope feature and tag traces with tenant identifiers so cost can be attributed per tenant. Masking of personal and contractual data runs in the SDK or the OpenTelemetry Collector before data reaches storage, because some server-side masking controls sit in Langfuse's enterprise tier. The output is a test report showing that traces carry tenant and cost metadata and that sample personal data fields arrive masked.
Build evaluation datasets and set release gates
Product and ML owners assemble datasets from representative production-like requests and from past failures, then define the scores each LLM feature must meet. Scoring can be model-based or human, and the team names who reviews results before a prompt, model or retrieval change ships. The output is a set of evaluation runs linked to each release candidate, with a documented threshold and an approver for each feature, so a change that fails its gate does not reach customers.
Test backup and restore on a spare cluster
The platform team configures ClickHouse backups to S3-compatible or Azure Blob Storage, and confirms replication settings, including ClickHouse Keeper nodes for replicated tables. ClickHouse documentation states that replication does not protect against human error or corruption, and that a backup never test-restored will probably not restore when needed. The output is a successful restore of a recent backup into a spare cluster, recorded with the time it took and the checks that passed.
Run upgrades on a fixed discipline
The team tracks Langfuse releases and ClickHouse support windows, and rehearses each upgrade on the non-production environment first. Langfuse applies database migrations at startup for updates within a major version, while major upgrades such as v3 to v4 follow a dedicated guide that requires ClickHouse 25.12 or later and advises about 3x ClickHouse disk headroom for backfills. The output is a written upgrade runbook with a pre-upgrade backup, a rehearsal record and a named owner for each release cycle.
Evaluation before releases: options compared
Model-based scoring uses an LLM to grade outputs against criteria such as relevance, groundedness, tone or policy compliance. In a self-hosted setup, the evaluation model is a separate call your team configures, and the scores are written back to the trace or dataset item they grade. The evaluator model can run on your own inference endpoint so that prompts and outputs in the evaluation step stay inside your infrastructure. Scores are only as reliable as the rubric, so they need to be checked against human judgment before anyone treats them as a release gate.
- Evaluator placement: Decide whether the judge model runs on a self-hosted endpoint, which keeps evaluation traffic in your network.
- Rubric design: Write one criterion per score so a failing result points to a specific defect.
- Score storage: Scores are attached to traces in your ClickHouse and PostgreSQL deployment, so retention follows the same policy as the traces.
- Calibration: Compare judge scores with a labeled sample before setting a pass threshold.
- Cost per run: Every evaluation call is a model call, so ask who pays for evaluation tokens and where that spend is recorded.
Confirm with the implementation partner which evaluator models you can run locally and how calibration will be documented.
Human review queues route a sample of traces to named reviewers who score them against a guide. Reviewers are usually product owners, subject matter experts or support leads, and their scores are stored alongside automated scores. Queues are the usual way to validate that a model-based judge agrees with people, and they create the evidence trail for who approved a change. Reviewer access to traces is access to the content of those traces, so queue membership should be limited to people who are already allowed to see the underlying customer data.
- Reviewer roles: Assign reviewers through project-level roles, which need an Enterprise license key in self-hosted Langfuse.
- Sampling rules: Define which traces enter the queue, such as a fixed share of new prompt versions or traces flagged by low automated scores.
- Review guides: Keep the scoring guide versioned so that scores from different months can be compared.
- Disagreement handling: Record cases where two reviewers disagree and who resolves them.
- Reviewer exposure: Mask personal data before it reaches the queue where your policy requires it.
Ask the partner how queue membership, review guides and reviewer exposure to customer data will be controlled and recorded.
A dataset built from production failures turns real incidents into fixed test cases. Traces where users rejected an answer, a tool call failed or a guardrail fired are copied into a dataset with the expected behavior. Each new prompt or model version is then run against that dataset before release, so a regression on a known failure blocks the change. This approach tests the cases your customers actually hit, but the dataset itself contains customer content, so it inherits the storage, masking and retention rules of the source traces.
- Failure selection: Tag traces from support tickets, negative feedback or low scores to build the set.
- Expected outputs: Store the expected behavior for each item, which may be a reference answer or a set of required facts.
- Version control: Version each dataset so a release can be traced to the exact test set it passed.
- Data minimization: Remove or pseudonymize customer identifiers before copying traces into a dataset.
- Dataset storage: Confirm which database holds dataset items and whether they are included in backups.
Check with the partner how datasets are versioned, who may add items, and how deletion requests reach copies in datasets.
Pre-release evaluation only counts as control evidence if the approval step is defined. A release gate should name who may approve a prompt or model change, which scores are required, and where the decision is recorded. In a self-hosted deployment the scores, dataset runs and prompt versions are stored in your PostgreSQL and ClickHouse instances and can be backed up with them. Approval itself usually happens in your change management tool, so the link between a score and a ticket or pull request needs to be designed, not assumed.
- Approver roles: Separate the people who write prompts from those who approve them where your change policy requires it.
- Protected labels: Protected prompt labels, which need an Enterprise license key in self-hosted Langfuse, restrict who can move a label to production.
- Decision record: Store the release decision with the score run, dataset version and prompt version it relied on.
- Score retention: Align score retention with trace retention and document both in your records.
- Restore testing: ClickHouse backups only protect scores if restores are tested, so include score tables in restore tests.
Ask the partner which parts of this approval flow are included in the Enterprise license scope and which you will implement with existing tools.
Evidence auditors and customers ask for, and what produces it
This table maps the evidence that SOC 2 assessors, EU AI Act reviewers, GDPR processors and enterprise customers commonly request to the platform records that produce it. Read each row as a question to answer before launch: what proof exists, where it lives and who keeps it. DeployOpen prepares systems, controls and evidence for these reviews but does not issue audit reports, attestations or certifications; independent auditors and certification bodies do.
| Evidence requested | What produces it in a self-hosted Langfuse and ClickHouse stack | What to confirm or ask for |
|---|---|---|
| SOC 2 access reviews | Periodic exports of Langfuse project members and roles, plus ClickHouse users and grants from access control settings, reviewed and signed by the owner. | Whether project-level RBAC roles need an Enterprise license key, and how reviews are scheduled and recorded. |
| SOC 2 change management | Change tickets linked to Langfuse upgrades, ClickHouse version changes and pre-production test results kept with the release record. | Which changelog review and pre-production sign-off steps your change policy requires before each upgrade. |
| Backup and restore test records | Dated ClickHouse BACKUP and RESTORE logs from a spare cluster, with row counts and checks that restored traces match the source. | Restore test frequency, who signs the result and whether PostgreSQL and object storage backups are tested in the same cycle. |
| ClickHouse replication and availability | Keeper cluster configuration, replica health records and recovery logs showing automatic resync from a healthy replica. | Keeper node count and hosting, plus the alerting that flags a lagging or read-only replica. |
| EU AI Act log retention | Configured retention for trace storage in ClickHouse and object storage, with a written rule that keeps automatically generated logs for at least six months under Article 19(1). | Whether your legal team sets a longer period and whether any other law applies to the same records. |
| GDPR processor terms | A signed data processing agreement covering documented instructions and Article 32 security measures, plus the list of sub-processors used for hosting and support. | Whether the implementation partner will sign the processor terms and where any support access to trace data is permitted. |
| GDPR erasure handling | A documented procedure for deleting traces by user or session identifier across ClickHouse, object storage and any backups, with a log of each request. | Whether server-side data masking and retention policies require the Enterprise license key, and how backups are purged. |
| Masking of personal data in traces | SDK or OpenTelemetry Collector masking rules, plus test records showing sample prompts were masked before storage. | Where masking runs, since client-side masking does not depend on the enterprise tier, and who approves changes to the rules. |
| Audit logs for trace access | Logs of who viewed, exported or deleted traces, available through Langfuse audit logs under the enterprise license or through ClickHouse query logs. | Whether audit logs are included in your license tier and how long they are kept. |
What SaaS teams ask before hiring help with LLM observability
What drives the cost of self-hosted LLM observability for a SaaS product?
The main cost drivers are the number of applications you instrument, the volume of traces and spans you ingest, how long you keep them, and how much infrastructure you run to store them. Self-hosted Langfuse runs two application containers, Langfuse Web and Langfuse Worker, plus PostgreSQL, ClickHouse, Redis or Valkey, and S3-compatible blob storage, so you pay for those compute, storage and backup resources in your own cloud account or data center. Instrumentation work, evaluation setup and any Enterprise license are separate line items. Ask any partner for a quote that separates one-time setup from recurring infrastructure, support and upgrade effort, and that states the trace volume and retention period it assumes.
How long does a production deployment of self-hosted Langfuse take?
The timeline depends on how many services you instrument, how sensitive the trace data is, and how much storage and backup design your environment needs. Langfuse documents Docker Compose on a single VM for testing with no high availability, and Kubernetes with Helm or Terraform for production on AWS, Azure and GCP, so a production target usually means cluster capacity, networking and identity work before any tracing starts. DeployOpen agrees scope, deliverables and timelines in writing before work begins, so ask for a written plan that lists each phase and its dependencies.
What does our team need to provide for an observability engagement?
You need a cloud account or data center allocation where the platform will run, engineers who own the applications being instrumented, and data owners who decide what prompts, outputs and customer identifiers may be stored and for how long. Your team also needs to approve the retention policy, the access model for trace data and the storage and backup targets. Expect your platform or infrastructure team to own the Kubernetes or VM layer, and your security team to review masking rules before production traffic is sent. Confirm these owners and their time commitments in the scope document.
Can our engineers run self-hosted Langfuse without an outside partner?
Yes, a capable platform team can run Langfuse, because the open source version provides all core features and APIs without limits and uses the same codebase as Langfuse Cloud. The harder parts are ClickHouse operations, including replication with ClickHouse Keeper, which the replication guide recommends at least three nodes for in production, plus tested backups with BACKUP and RESTORE, and a repeatable upgrade process. Ask yourself whether you can staff those skills and keep up with the ClickHouse stable and LTS release cycle, then decide whether to hire help for build, operate or both.
How do Langfuse Enterprise license keys change what we can build?
The MIT-licensed core covers tracing, evaluation and APIs, but several governance features need an Enterprise license key on self-hosted deployments. These include project-level RBAC roles, protected prompt labels, data retention policies, audit logs, server-side data masking, UI customization, SCIM through the org management API and the instance management API. Files under the ee directories fall under a separate enterprise license. If customer-facing tenants need isolated access or retention controls, confirm which features your design depends on and price the Enterprise plan, which adds a named support engineer and a support SLA, before you commit to the architecture.
How do upgrades work when moving from Langfuse v3 to v4?
Updates within a major version are non-disruptive and apply database migrations automatically at startup, but v3 to v4 is a major upgrade with its own guide. That guide requires ClickHouse 25.12 or later (26.4 recommended), PostgreSQL 15 or later and Redis 7.0 or later, and it moves to an observations-first ClickHouse data model. It also advises about 3x ClickHouse disk headroom when backfilling historic data, so plan that storage before you start. Test the upgrade in a pre-production environment with representative data, and read the ClickHouse changelog, since minor backward-incompatible changes can occur.
How is customer data masked before it reaches the trace store?
Masking should happen on the client side, in the SDK or the OpenTelemetry Collector, before a prompt or output leaves the application, so personal data never reaches storage in raw form. Langfuse's server-side data masking is an Enterprise license feature on self-hosted deployments, so if you rely on it, confirm the license scope first. Your data owners should define which fields, such as names, account numbers or email addresses, are redacted, and your security team should test those rules with realistic sample traffic. Ask any partner to show how masking rules are tested and how erasure requests are handled across ClickHouse and blob storage.
Where the Phoenix license limit and Langfuse enterprise features bite
The first thing to settle is whether the observability tool can sit behind a customer-facing feature at all. Arize Phoenix is licensed under the Elastic License 2.0, which prohibits providing the software to third parties as a hosted or managed service. For a SaaS company, that bar is easy to cross without noticing. If customers log in to a trace dashboard, view their own prompts and scores, or receive a reporting view built on Phoenix, the product is arguably offering Phoenix as a managed service to third parties. Even internal use can become a question if a partner or subsidiary operates the instance for your customers. Treat Phoenix as an internal engineering tool unless counsel confirms that your customer-facing use falls outside the restriction, and check the current text of the Phoenix license before the architecture review.
Langfuse takes a different position. Its core features and APIs are available under the MIT-licensed open source version, and its self-hosted deployments use the same codebase as Langfuse Cloud. However, several controls that a SaaS team may need for customer-facing governance sit behind an Enterprise license key. These include project-level RBAC roles, data retention policies, audit logs, server-side data masking, and the organization management API with SCIM. Protected prompt labels, UI customization and the instance management API also need the key. Confirm with Langfuse which of these you need before you design tenant isolation, because retention and masking decisions made early are expensive to change later. Ask counsel to review the separate enterprise license terms in the ee/ directories of the Langfuse license file alongside your own customer contracts.
Put these questions into the cost model and the vendor request, not just the legal review. Ask whether a Self-Hosted Enterprise plan, which Langfuse lists with a named support engineer and a support SLA, is needed for your rollout, and get the price in writing because Langfuse does not publish it. Ask how per-tenant cost attribution and customer-visible dashboards will be built without relying on Phoenix in a hosted role. Record the decision and its legal basis in your AI system documentation. That record helps when customers, auditors or the SOC 2 assessor ask how tenant data is separated and who can view it.
How DeployOpen works on this
DeployOpen plans, implements, hardens, documents and supports self-hosted Langfuse inside the client's own cloud accounts or data centers, so SaaS traces, prompts and cost records stay under the client's control. The work is fixed-scope, with the scope agreed in writing before any build starts.
How we scope
- Inventory the LLM features shipped to customers and the frameworks used to call models
- Size ingestion volume, trace retention and ClickHouse storage against agreed tenant and traffic assumptions
- Confirm which Langfuse features need an enterprise license key before the architecture is fixed
- Agree written deliverables, timelines and acceptance tests for the observability stack
How we build
- Deploy Langfuse Web, Langfuse Worker, PostgreSQL, ClickHouse, Redis or Valkey and S3-compatible storage in the client's environment
- Instrument application code with the SDK or OpenTelemetry and set up per-tenant cost attribution
- Configure evaluations and a pre-release test set so prompt and model changes are scored before they ship
- Test restores of ClickHouse backups and verify replication and failover behavior before go-live
How we hand over
- Deliver architecture, runbooks, sizing records, backup and restore procedures and access model documentation
- Document which controls and logs are in place to support evidence for SOC 2 or ISO/IEC 27001 assessments, without claiming any certification
- Walk the client's platform and engineering teams through upgrades, including ClickHouse version checks and the changelog review
- Agree an upgrade procedure for Langfuse major versions and ClickHouse releases in a pre-production environment first
How we support
- Offer ongoing support, upgrades and continuous security testing as a retainer after handover
- Run periodic restore tests and monitor ClickHouse disk headroom as tenant traffic grows
- Review evaluation results and tracing gaps with the client's engineering team on an agreed cadence
- Confirm with the client's team which support level and enterprise features fit their contract terms
