Monitoring LLM output quality in production starts with who approves a change
An LLM feature is live, quality is checked by someone reading a handful of answers now and then, and prompt or model changes go out without a recorded score. This page covers how to run continuous evaluation of live output on self-hosted infrastructure and, just as important, who reviews the scores before a prompt or model change ships. The short answer is that a named reviewer and a documented threshold must exist before the first scored release, because a score nobody is accountable for does not stop a regression. Tracing alone does not answer the quality question. A trace records what the application sent and received, which is necessary for debugging and for audit records, but it does not say whether an answer was correct, grounded or safe. Evaluation adds scores to traces: sampled production traces scored by model-based judges, by rules, or by human reviewers, with the results stored next to the traces they describe. For the self-hosted path, Langfuse is the main option to assess first, since its MIT-licensed core includes evaluation features and its data stays in your ClickHouse and blob storage. Opik, licensed under Apache-2.0, is the alternative to evaluate if you want an Apache-2.0 platform with LLM-as-a-judge metrics and online evaluation rules. promptfoo fits pre-release and CI evaluation, where a prompt change is tested against a fixed dataset before it merges, and it runs locally on your machine. Before choosing a tool, confirm three things with your team. First, who signs off on a change when scores drop, and whether that person can read the trace data under your access rules. Second, which failures become regression test cases, so that every production failure you fix stays in the dataset. Third, how judge models are hosted, because a model-based scorer that calls an external API sends production content outside your boundary. Ask any implementation partner to show how these decisions are recorded in the platform itself.
Requirements that shape a production evaluation setup
This table maps the requirements that shape a production evaluation setup to their source and to what each one means for the build. Read it as a checklist to settle before choosing tools, since sampling, retention, reviewer sign-off and log-keeping duties decide where traces are stored and who can change a prompt or model.
| Requirement | Source | What it means for the build |
|---|---|---|
| Trace sampling and volume | Operational constraint: ingestion load and storage sizing | Decide which traces are captured in full, which are sampled and which are scored online. Langfuse's scaling guide ties worker and ClickHouse sizing to ingestion, so confirm volume before sizing storage. |
| Retention of traces and scores | Operational and legal constraint: retention rules set by the organization | Set retention per data class before go-live. Some retention policy controls sit in Langfuse's enterprise license key features, so confirm which ones your deployment needs. |
| Evaluation coverage per feature | Operational constraint: each LLM feature has its own failure modes | Map every production feature to at least one automated check and a named owner. A feature without a scored evaluation has no measured quality after a prompt change. |
| Model-based and human scoring | Operational constraint: judge models and reviewers each have known blind spots | Pair LLM-as-a-judge metrics with periodic human review on a sample. Confirm that judge models run inside your boundary, since the judge receives production text. |
| Datasets built from production failures | Operational constraint: regressions come from real user inputs | Promote flagged traces into versioned datasets and rerun them before each prompt or model change. Keep the mapping from each dataset item to its source trace. |
| Documented reviewer sign-off | Operational and governance constraint: a change needs an accountable approver | Define who reviews scores before a prompt or model ships, record the decision and the evaluation run it relied on, and keep that record with the change ticket. |
| Log-keeping for high-risk systems | Regulation: EU AI Act Article 12(1) and Article 19(1) | If the system is high-risk, logs must be recorded automatically across its lifetime and kept at least six months unless other law requires longer. Confirm high-risk classification with counsel before designing retention. |
| Access control on trace data | Operational constraint: traces often contain personal or client data | Restrict who can view raw prompts and outputs and separate reviewers from the people who change prompts. Role separation may require the Langfuse enterprise tier, so confirm it early. |
Which self-hosted platforms fit production evaluation and why
Production evaluation needs a place where traces are stored, scores are attached to them, and test sets are kept under the organization's control. The platforms below cover different parts of that loop, so most teams will combine an observability backend with a separate test runner.
Langfuse for traces and scores
Langfuse holds traces, scores and datasets in one self-hosted store, and its core features are under the MIT license outside the ee directories. Before you rely on it for production review, check which controls need an enterprise license key: project-level RBAC, audit logs, data retention policies and server-side masking are listed as enterprise features. Confirm with the vendor whether your review workflow depends on any of them.
Opik for online evaluation
Opik is Apache-2.0 licensed, and its README states the full platform, including evaluations, can be self-hosted without a commercial license. It offers LLM-as-a-judge metrics for hallucination, moderation and RAG, plus online evaluation rules that score live traces as they arrive. Its infrastructure uses ClickHouse, MySQL and Redis, so plan storage and backups for that stack before you go live.
promptfoo for CI evaluation
promptfoo is MIT licensed and runs evaluations from a CLI, a library or a CI/CD pipeline, so a prompt or model change can be blocked until its test set passes. Evals run on your machine and talk directly to the model. It also covers automated red teaming against the OWASP LLM Top 10 and related frameworks. Note that OpenAI acquired promptfoo, and confirm the license and roadmap terms in writing before you depend on it.
promptfoo self-hosted app limits
The self-hosted promptfoo web app stores data in a local SQLite database that cannot be shared across replicas. It has no RBAC, horizontal scaling, built-in authentication or SSO, and the documentation does not recommend it for production. Teams that need shared review should look at the Enterprise or On-Premise tiers, or keep the app for single-user work and store results in another system. Ask the vendor which applies to your setup.
Phoenix under Elastic License 2.0
Arize Phoenix offers LLM tracing and evaluation, but it is licensed under the Elastic License 2.0, which prohibits providing the software to third parties as a hosted or managed service. Internal use inside your own organization is a different case from offering a product to customers. Have counsel confirm that your use, especially in a SaaS product, falls within the license terms before you choose it for production evaluation.
Pairing a tracing backend with a test runner
Most production programs use one platform to store live traces and scores and another to run fixed test sets before release. Decide which system is the source of truth for each score and dataset, how a failing production trace becomes a regression test, and who may approve a change. Ask any implementation partner to show this data flow in writing before the rollout starts.
How sampled traces become scored evaluations
The flow starts with an instrumented application that sends every trace to the self-hosted observability platform, whether that is Langfuse or Opik. Traces hold the prompt, retrieved context, model output, tool calls, latency and cost for each request. A sampling rule then selects a share of live traces for evaluation, often weighted toward specific routes, user groups, model versions or traces that already carry a negative signal such as a thumbs-down or a failed tool call. Sampling is a design decision for the team, not a fixed setting, so the rule should be written down and reviewed as traffic changes.
Selected traces are then scored by two kinds of reviewer. Model-based judges apply an LLM-as-a-judge metric, for example for hallucination, groundedness or moderation, and return a score per trace. Human reviewers check a smaller set against the same criteria and calibrate the judges where they disagree. Judge models can run inside the organization's own infrastructure, served through a self-hosted inference endpoint, so the prompts and outputs being scored never leave that environment. Confirm with the platform vendor which evaluation features need a license key in a self-hosted deployment, since some controls sit in enterprise tiers.
Scores need a written rubric and a named owner before they mean anything. The rubric defines what counts as a failure, how severity is graded and what a judge must return. The owner decides who signs off on score thresholds, reviews disputed traces and approves changes to the rubric. Failing traces are attached to a dataset so the same cases are rerun before any prompt or model change ships. Ask who signs off on a release when scores fall, and whether that person can see the traces that contain client data.
How the rollout moves from spot checks to a release gate
The sequence turns ad hoc review of LLM answers into a repeatable process, where traces, scores and reviewed datasets decide whether a prompt or model change ships. Each phase produces an artifact that the next phase depends on, so the release gate rests on evidence the organization already controls.
Scope features and assign risk tiers
Start by listing every LLM feature in production, the users it serves, and the data its prompts and outputs contain. Classify each feature into a risk tier based on the consequence of a bad answer and whether personal or client data appears in traces. Owners from the product, security and compliance teams sign off on the list. The output is a feature inventory with a tier and an accountable owner for each entry, which sets how much evaluation coverage each feature needs.
Instrument applications and capture traces
Engineers add tracing to each application, either through the Langfuse or Opik SDK, OpenTelemetry instrumentation, or the framework integrations those platforms provide. Masking of personal data is configured in the client or collector before data reaches storage, and trace storage is confirmed to sit in the organization's own ClickHouse and object storage. The output is a staging and production trace stream where prompts, outputs, model versions, latency and cost are visible per request, verified by sampling a set of traces end to end.
Define rubrics and name the reviewers
Write a scoring rubric for each risk tier that states what a good answer looks like, such as factual grounding, policy compliance, format and tone. Decide which criteria a model-based judge scores and which only a person can judge, and calibrate the judge against a small set of human-labeled examples before trusting its scores. Name the reviewers, their backup and the escalation path for disputed scores. The output is a versioned rubric and a reviewer roster that the evaluation pipeline references by name.
Run online evaluation on sampled traces
Enable online evaluation rules that score a defined sample of live traces against the rubric, with higher sampling for high-risk features. Scores are written back onto the traces so a reviewer can open the original prompt, retrieved context and output together. Confirm the sampling rate, the judge model and its location, and who can see the scored traces, since judged traces still contain client data. The output is a running score history per feature, with a regular review of low-scoring traces that feeds the next phase.
Build datasets from production failures
Reviewers move low-scoring and user-flagged traces into a dataset, after removing or masking data that should not be reused. Each entry records the input, the expected behavior, and the reason it failed, so the same failure can be tested again later. Owners tag entries by feature and risk tier to keep the regression set representative. The output is a versioned failure dataset that becomes the regression suite for prompt and model changes, and its growth is tracked as the feature changes.
Wire offline evaluation into change approval
Connect an offline evaluation run, using the failure dataset and a held-out set, to the change process for prompts, models, retrieval settings and gateway routes. A tool such as promptfoo can run these evaluations in CI/CD, while the results are stored in the same observability platform as production scores. Define the pass thresholds per risk tier and the named approver who signs off when a score falls short. The output is a merge or release gate that blocks a change until the evaluation result is recorded and approved.
Review gate results and adjust coverage
After each release, the owners compare production scores with offline results to find gaps, such as failures that the gate did not catch or rubric criteria that reviewers disagree on. Judge calibration and sampling rates are adjusted and the rubric is versioned. Access reviews confirm that only authorized staff see scored traces. The output is a dated review record showing what changed in the rubric, dataset and thresholds, which supports the evidence an audit or validation team will ask for.
Who reviews scores at each stage
The product owner signs off on what the feature is supposed to do and the rubric that defines a passing answer. Before any prompt or model change ships, this person confirms that the scores still reflect the product's purpose, not only the metrics that are easy to compute. Rubric changes should be treated as versioned decisions, because a score that moves after a rubric edit is not a measure of model quality.
- Rubric approval: Approve each criterion, its scale and the threshold that separates acceptable from unacceptable output.
- Release decision: Sign the go or no-go for a prompt or model change based on the evaluation results compared with the previous baseline.
- Dataset scope: Confirm which production failures enter the regression dataset and which are out of scope.
- Accepted risk: Record any known regression the team chooses to ship, with a reason and a review date.
Langfuse and Opik can store evaluation scores and experiment results, but neither decides what a good answer is. Confirm with your partner which rubric version was attached to each score, and whether your change process requires the product owner's approval to be stored next to the release record.
The domain reviewer is the subject-matter expert who judges whether an answer is correct for the business context, such as a credit policy, a clinical guideline or an HR rule. Automated judges can flag likely problems, but a reviewer confirms samples and resolves disagreements between human and model scores. Their judgment is most useful where the rubric is ambiguous, and their labels become the ground truth used to calibrate model-based scoring.
- Sample review: Label a regular sample of production traces on the agreed rubric, including traces the automated judge scored highly.
- Judge calibration: Compare human labels with LLM-as-a-judge scores and report where they disagree before trusting the judge.
- Failure triage: Tag each confirmed failure with a category so it can be turned into a test case.
- Reviewer coverage: Name at least one backup reviewer so scoring does not stop during absence.
Traces that contain client or patient data should be reviewed inside the access model your security team has approved. Ask your partner how annotation queues and reviewer roles are set up in the self-hosted tool, because some role controls are part of the enterprise license tier in Langfuse.
The security or compliance reviewer approves who may see scored traces, how long they are kept and what counts as an exception to the normal review path. Scores are derived from prompts and outputs, so they often carry the same personal or confidential data as the traces themselves. This reviewer also decides whether the evidence trail is sufficient for model risk, ISO/IEC 42001 or EU AI Act record-keeping expectations, without assuming the tool produces compliance on its own.
- Access approval: Approve reviewer accounts, role assignments and the projects that contain sensitive traces.
- Exception sign-off: Approve any threshold exception, such as shipping with a known safety regression, and require a written rationale.
- Retention and masking: Confirm retention periods for traces and scores, and that masking happens before storage.
- Evidence review: Check that the change record links to the evaluation run, the rubric version and the approvals.
Article 19(1) of the EU AI Act sets a minimum six-month log retention for providers of high-risk systems, unless other law provides otherwise, so confirm with counsel how that interacts with your retention policy. Verify with the vendor which controls, such as data retention policies or audit logs, require an enterprise license key in your chosen version.
The platform engineer signs off on the infrastructure and the evaluation pipeline itself: that scores are computed on the intended model version, that sampling is reproducible and that the scoring service is available when a release needs it. Their records prove how a number was produced. Without that, a dashboard value cannot be defended in a review, even if the product and domain sign-offs are complete.
- Pipeline configuration: Approve the sampling rate, the judge model, the evaluation code version and the environment in which scoring runs.
- Run integrity: Confirm that each evaluation run records the model, prompt version and dataset version it used.
- Availability: Confirm the evaluation service and its storage meet the release calendar, including backups.
- Upgrade control: Approve upgrades of the observability stack and test them against existing evaluation results.
The tool records traces, scores and run metadata. The team must document who approved each rubric, sample and exception in its change process, because most tools do not capture the decision itself. Ask your partner which fields of an evaluation run are stored automatically and which must be added to the change ticket.
Evidence a regulator or auditor will ask for
This table maps the evidence items an assessor usually asks for in production LLM monitoring to the component that produces each one. Read it as a checklist for designing the platform so that every item is captured, stored and retained in the organization's own infrastructure. DeployOpen prepares this evidence and the controls behind it, but independent auditors and certification bodies issue any audit report or certification.
| Evidence item | Produced by | What to confirm before go-live |
|---|---|---|
| Trace records of live requests | Langfuse or Phoenix tracing via SDK or OpenTelemetry, stored in ClickHouse and object storage | Which fields are captured, how prompts are masked before storage, and where the data lives |
| Evaluation scores with rubric versions | Model-based judges or promptfoo evaluations, with the rubric version stored on each score | Each score links to the exact rubric and judge model version used at the time |
| Dataset lineage from production failures | Trace-to-dataset export in Langfuse or Opik, with IDs linking back to source traces | Who approved each failure case and how it was selected for the dataset |
| Reviewer decisions and annotations | Human annotation queues in the evaluation platform, with reviewer identity and timestamp | Reviewer roles, separation between reviewers and prompt authors, and how disagreements are resolved |
| Change approvals for prompts and models | Ticketing or change workflow outside the LLM platform, referencing evaluation run IDs | That no prompt or model change ships without a linked evaluation result and a named approver |
| Retention settings | Data retention policies on the platform (an Enterprise license feature in self-hosted Langfuse), plus object storage lifecycle rules | Retention periods match the regulatory or internal schedule and deletion is verified in storage |
| Access logs for trace data | Audit logs (an Enterprise license feature in self-hosted Langfuse) and reverse proxy or identity provider logs | Who viewed traces containing client or personal data, and whether those logs are retained |
| Monitoring and drift alerts | Online evaluation rules and dashboards, with alert history retained | Thresholds, owners, and the response record when a score drops below the agreed level |
How much effort does production LLM evaluation take to set up?
What drives the cost of setting up production LLM evaluation?
Cost depends mostly on four components: instrumenting the applications, designing the scoring rubrics, the time domain experts spend reviewing outputs, and the infrastructure that stores the traces. Software licenses for Langfuse and promptfoo's community edition are free to use, so most spend goes to people and to hosting. Ask any quote to split these lines separately, and to state which items are one-time (instrumentation, rubric design, initial datasets) and which recur (reviewer hours, infrastructure, upgrades, support).
How much infrastructure does a self-hosted Langfuse deployment need?
Langfuse's scaling guide lists minimum resources of 2 CPU and 4 GiB each for Langfuse Web, Langfuse Worker and PostgreSQL, 1 CPU and 1.5 GiB for Redis or Valkey, and 2 CPU and 8 GiB for ClickHouse, with at least 16 GiB of ClickHouse memory for larger deployments. These are starting minimums, not sizing for your traffic. Ingestion volume, number of evaluations run per trace and retention period determine the real storage and compute, so ask the partner to size these from your expected request volume. Docker Compose on one VM suits testing only, since it has no high availability.
How long does it take to reach a first evaluation gate?
No fixed duration can be given without the scope, the number of applications and the state of your existing logging. The first gate usually follows a sequence: a running observability stack, instrumented traces from one production application, a seed dataset drawn from real failures, and one agreed scoring method. Each step depends on access and on reviewer availability, which often sets the pace more than the software. Ask for a written milestone plan with a named gate and its acceptance criteria before work starts.
What must our team provide before evaluation work starts?
Plan for three inputs. First, domain experts who can say whether an output is correct, safe and on-policy, and who can commit time to scoring. Second, a representative sample of production requests and responses, with personal data handling agreed in advance. Third, access to the applications, hosting accounts, identity provider and the ClickHouse, PostgreSQL and object storage the platform will use. Confirm these with your security team early, because access approvals often take longer than the technical build.
Can an internal team run production LLM evaluation on its own?
Yes, if it has engineers who can operate Kubernetes or Docker, manage databases and object storage, and write evaluation code, plus reviewers who own the rubrics. Langfuse's open source features are available without a license key, and Opik is Apache-2.0 licensed and self-hostable without a commercial license, so the software need not be the constraint. The harder part is recurring work: upgrades (Langfuse's v3 to v4 guide requires ClickHouse 25.12 or later), rubric maintenance and on-call for the stack. A partner can take on some of this under a fixed scope or a retainer.
When does the Langfuse enterprise license key matter for RBAC or audit logs?
The license key matters once you need controls the open source version does not include. Langfuse's documentation lists project-level RBAC roles, protected prompt labels, data retention policies, audit logs, server-side data masking, the org management API with SCIM and the instance management API as enterprise features. If several teams share one instance, or an auditor expects a record of who viewed or changed traces and prompts, budget for the license and confirm the current price with Langfuse. Self-hosted promptfoo, by contrast, has no RBAC or SSO, so teams with those needs should plan for its Enterprise or On-Premise tier.
How do reviewer hours change the total effort?
Reviewer time is often the largest recurring cost, and it scales with sample size and rubric complexity. Model-based scoring, such as Opik's LLM-as-a-judge metrics or promptfoo's evaluation runs, reduces manual review for routine checks, but it still needs human spot checks to confirm the judge agrees with experts. Start with a small sample per release and a narrow rubric, then widen it as disagreement rates become known. Ask any partner how many reviewer hours per release their plan assumes and how they measure judge agreement.
Questions to ask a partner before scoping production evaluation
Start with how the partner will decide what gets scored. Ask how they instrument each application, whether through the Langfuse SDK, OpenTelemetry or the Opik @track decorator, and how they select which traces are sampled for evaluation. A good answer names the sampling rule, the share of traffic it covers, and how failed or low-confidence responses are pulled into the review queue. Ask whether online evaluation rules will run on live traffic or only on exported batches, and who approves a change to those rules before it reaches production.
Then press on the judges. Ask which model acts as the LLM-as-a-judge, where that model runs, and how its scores are checked against human labels before anyone trusts them. The partner should describe a calibration step with a labeled sample, the agreement rate they will accept, and what happens when a judge drifts after a model upgrade. Ask how datasets are built from production failures: who labels them, how personal data is removed before a failure becomes a test case, and where the dataset is stored. Confirm which Opik features, such as online evaluation rules and Opik Guardrails, and which Langfuse features, such as RBAC roles, audit logs and data retention policies, need an enterprise license key, and get that cost into the written scope. Ask how the partner plans to use promptfoo for pre-release evals in CI/CD, since its self-hosted application stores data in SQLite, has no RBAC or SSO, and is not recommended for production use. A credible answer keeps promptfoo as a local test runner and keeps the score history in the observability platform.
How DeployOpen scopes and supports production evaluation
This sequence turns a spot-check habit into a scoped, self-hosted evaluation program with written deliverables, named reviewers and a support path after go-live. Each step produces a document, a configured component or a recorded check that the client can inspect. The end state is continuous scoring of live output inside the client's own infrastructure, with a documented release gate for prompt and model changes.
Agree the scope and evaluation goals in writing
DeployOpen starts with a discovery session covering the LLM applications in scope, the user populations, the data classes that appear in prompts and outputs, and the decisions the scores will inform. The client's security, compliance and product owners confirm which failure modes matter most, such as hallucination, policy violations, data leakage or cost drift. The output is a written statement of work listing deliverables, responsibilities, assumptions and what is out of scope. Work does not begin until the client has signed it. Ask the partner to name the people who approve the scope on both sides.
Map the hosting and data boundary
DeployOpen documents where the evaluation stack will run inside the client's cloud accounts, data centers or air-gapped network. The review covers the Langfuse or Opik components, the storage for traces and datasets, the model endpoints used as judges, and which systems can read scores. Access paths, network egress rules and the location of backups are recorded. The output is an architecture and data-flow document that the client's security team can approve. Confirm that no trace or score leaves the boundary unless the client has agreed to it.
Deploy and configure the evaluation stack
Engineers install the self-hosted tracing and evaluation platform in the client's environment, configure authentication, project roles and the storage back ends, and connect the first applications through their SDK or OpenTelemetry instrumentation. Where the license tier matters, the team checks which features need an enterprise license key before they are enabled. The output is a running environment with a documented configuration and a smoke test showing traces arriving from each instrumented application. Ask for the list of enabled features and their license status.
Define sampling, judges and human scoring rules
DeployOpen configures how much live traffic is sampled, which model-based judges score it, and which rubrics they use. Human review queues are set up for cases the judges cannot settle or that touch sensitive content. Thresholds and alert conditions are written down with the client's reviewers. Judge prompts and rubrics are versioned so that a score can be traced back to the rule that produced it. The output is a scoring specification and a test run showing judge results compared with reviewer labels on a sample set, so the client can judge whether the automated scores are trustworthy enough to act on.
Build datasets from production failures
Engineers help the client's team move flagged traces into versioned datasets, removing or masking personal data before the examples are reused. These datasets become the regression set for future prompt and model changes. The team agrees on how each failure is labeled, who approves additions and how old examples are retired. The output is a dataset register with owners, labeling rules and a record of masking applied. Confirm with the client's privacy function that the dataset handling matches their retention and access rules.
Set the release gate and reviewer roles
DeployOpen writes the procedure that decides whether a prompt or model change can ship. It names the reviewers who see the scores, the minimum evidence required, the escalation path for a failed gate and the approval record that is kept. Separation between the people who change prompts and the people who approve them is documented where the client requires it. The output is an approved release procedure and a dry run on a real change in a non-production environment, with the result recorded. Ask who holds the approval authority and how exceptions are logged.
Test the evaluation pipeline and AI application security
Where the scope includes it, DeployOpen runs security testing on the LLM applications and their agents, covering prompt injection, data leakage and unsafe output, and records the findings in a written report. Test results feed the regression datasets and the scoring rubrics. The evaluation tooling itself is also checked for access control and hardening. The output is a test report with findings, owners and remediation status. Confirm the test scope, rules of engagement and the handling of any sensitive data found during testing before testing starts.
Prepare evidence and hand over with ongoing support
DeployOpen assembles the evidence pack: configuration records, access reviews, retention settings, scoring specifications, review logs and test reports, mapped to the control or regulation the client names. This pack supports assessments run by the client's internal audit or an independent auditor, but DeployOpen does not issue audit reports, attestations or certifications. Handover covers runbooks, upgrade procedures and training for the client's team. Ongoing support, upgrades and continuous testing are available under a retainer agreed in writing. The output is the signed handover record and, if chosen, a retainer scope.
