Which open-source tools replace LangSmith evaluations?
Langfuse covers the closest set of LangSmith evaluation features, Opik covers evaluators and prompts under Apache-2.0, and promptfoo covers release-gating evals in CI. No single tool matches every LangSmith evaluation feature, so most teams that leave LangSmith for evaluation end up combining two of these or accepting a gap. If your renewal, an Enterprise self-hosting requirement or a data control concern is driving the move, start by listing which LangSmith evaluation features your team actually uses: datasets, experiments, online evaluators, annotation queues and CI checks.
Langfuse is the broadest match. Its evaluation layer includes LLM-as-a-judge scoring of live production traces, annotation queues for manual review, datasets with experiments run from the UI, SDK or OpenTelemetry, and CI/CD experiments that can block deploys on regressions. Its core features sit in the MIT-licensed code, but some administrative controls, such as project-level RBAC, data retention policies and audit logs, need an Enterprise license key. Confirm which of these controls your governance team requires before you assume the open-source version covers them. Langfuse evaluation overview lists the full set of evaluation features.
Opik, which Comet builds under Apache-2.0, covers LLM-as-a-judge metrics for hallucination, moderation and RAG, online evaluation rules and prompt management with a playground. Its README states the full platform, including evaluations, can be self-hosted without a commercial license. Opik does not match LangSmith's annotation workflow in the same way Langfuse does, so test it against your review process. promptfoo is different in kind: it runs evaluations and red teaming from the CLI, as a library or in CI/CD pipelines, and it runs locally against your models. It is strong for gating releases but is not a production tracing or annotation tool, and its self-hosted app stores data in SQLite and is not recommended for production use.
How LangSmith evaluation capabilities map to open-source options
This table maps each LangSmith evaluation capability, from datasets and online evaluators to annotation queues and CI gating, to the open-source option that covers it. Read the coverage column for gaps and for features that sit behind a Langfuse enterprise license key, and confirm current feature details with each vendor before you commit. The main takeaway is that Langfuse and Opik cover most trace-linked evaluation, while promptfoo covers release gating and red teaming, and both need some rebuilding.
| Capability | Langfuse | Opik and promptfoo |
|---|---|---|
| Datasets and experiments | Datasets with experiments run from the UI, SDK or OpenTelemetry in the MIT-licensed core | Opik provides evaluations on datasets; promptfoo runs test cases from config files and can be run as a CLI or library |
| LLM-as-a-judge on production traces | Scores live production traces with LLM-as-a-judge in the core product | Opik offers online evaluation rules and judge metrics for hallucination, moderation and RAG; promptfoo is oriented to pre-release evals, not live traces |
| Annotation queues and manual review | Annotation queues and scores through the UI for human review | Opik annotation coverage should be confirmed in its current release; promptfoo has no equivalent annotation queue |
| Code evaluators | Code evaluators and scores submitted via API or SDK | Custom code checks are possible in promptfoo configs; Opik supports custom metrics through its SDK, to be confirmed for your version |
| CI regression gating | CI/CD experiments can block deploys on regressions | promptfoo is built for CI/CD pipelines and can fail a build on failing assertions; Opik CI gating should be confirmed |
| Prompt-linked evaluation | Prompt versions and traces connect in one project, so regressions stay visible | Opik links prompts with its evaluations; promptfoo tests prompts defined in its config, so link results to prompt versions in your own process |
| Red teaming | Not a core focus; confirm current scope with Langfuse | promptfoo covers automated red teaming and references the OWASP LLM Top 10, NIST AI RMF and EU AI Act |
| Enterprise controls | RBAC roles, protected prompt labels, audit logs and data masking need an enterprise license key | Opik's full platform is Apache-2.0 without a key; promptfoo SSO, RBAC and continuous monitoring sit in its Enterprise tier |
| Self-hosted production hosting | PostgreSQL, ClickHouse, Redis or Valkey and S3-compatible storage, with Helm for production | Opik uses ClickHouse, MySQL and Redis with Helm; self-hosted promptfoo uses SQLite, so it is not recommended for production |
When Langfuse, Opik or promptfoo fits best
These three tools cover different parts of the LangSmith evaluation workflow. Langfuse and Opik keep evaluations attached to production traces, while promptfoo runs evaluations as a CLI gate in the release pipeline. The right choice depends on whether your team evaluates live traffic, reviews outputs by hand, or blocks regressions before deploy.
Langfuse for trace-linked evaluation
Langfuse attaches LLM-as-a-judge scores to live production traces, sends traces to annotation queues for manual review, and runs datasets with experiments from the UI, SDK or OpenTelemetry. CI/CD experiments can block a deploy on regressions. Its repository is MIT licensed except the ee directories, which fall under a separate enterprise license, so check which features your evaluation plan needs before you commit. Self-hosting requires PostgreSQL, ClickHouse, Redis or Valkey, and blob storage, so budget for operating that stack.
Opik for a fully open platform
Opik is Apache-2.0 licensed, and its README states the full platform, including tracing, evaluations and prompt management, self-hosts without a commercial license. It provides LLM-as-a-judge metrics for hallucination, moderation and RAG, plus online evaluation rules that score traces as they arrive. The stack is ClickHouse, MySQL and Redis, so confirm your team can run ClickHouse before you choose it over a lighter setup.
promptfoo for CI release gates
promptfoo is MIT licensed and works as a CLI, a library or a pipeline step, and its evals run on your own machine against the model provider directly. It suits teams that want a pass or fail check on every prompt or model change before release. The self-hosted app stores data in SQLite, cannot share data across replicas, and has no RBAC or SSO, so treat it as a runner for gates rather than a shared multi-team evaluation system.
Choosing between trace and gate tools
Pick Langfuse or Opik when evaluations must follow production traces and reviewers need queues and shared scores. Pick promptfoo when the main question is whether a change breaks a fixed test set before it ships. Many teams run both, with promptfoo in CI and a trace platform in production. Ask each vendor which evaluator types, retention settings and access controls your plan includes, since these vary by edition.
Which combination suits your team's workflow
Teams that score live traffic need a tracing backend that holds production traces, runs judges against them, and routes flagged outputs to people. Langfuse covers this in one self-hosted stack: LLM-as-a-judge scoring of production traces, annotation queues for manual review, and scores written back through the UI, API or SDK. Opik is a credible second option if you also want its hallucination, moderation and RAG metrics and online evaluation rules.
- Judge configuration: Recreate each LangSmith online evaluator as a Langfuse LLM-as-a-judge rule, and document the judge model, prompt and score name, since these do not transfer automatically.
- Annotation queues: Rebuild reviewer assignment and queue ownership in Langfuse, because the queues are part of the UI rather than a shared LangSmith workspace you can import.
- Judge cost ownership: Judge calls are billed by your model provider, so budget for them on your own account and confirm the sampling rate you intend to use.
- Operational load: A self-hosted Langfuse deployment runs PostgreSQL, ClickHouse, Redis or Valkey and blob storage, so someone must own backups and upgrades.
- Lost workflow: LangSmith's managed online evaluation and its hosted queue history are not carried over; past scores stay in the old system unless you export them.
Confirm that your judge provider terms and data handling allow production content to be sent to it before turning on sampling.
Teams that want evaluations to block a pull request or a deploy usually need a test runner, not a monitoring product. promptfoo runs locally from the command line, as a library or inside a CI/CD pipeline, and it covers LLM evaluation and red teaming. Pair it with Langfuse CI/CD experiments when you also want dataset runs and regression history recorded in the same tracing system that serves production.
- Gate definition: Express LangSmith dataset checks as promptfoo test cases and assertions in version-controlled config files, so the gate is reviewed like any other code change.
- Regression history: Langfuse experiments can record run results against datasets and fail the deploy on regressions, which gives you a comparison trail that promptfoo alone does not provide.
- Runtime requirements: The promptfoo release 0.124.1 on npm requires Node.js 22.22.0 or newer, so pin the runner image in your pipeline.
- Shared results: Team sharing, SSO and role-based access sit in promptfoo Enterprise. The self-hosted promptfoo app uses local SQLite and has no RBAC or SSO, so do not use it as a shared gate server.
- Lost workflow: LangSmith's dataset UI for editing examples in the browser is replaced by files in your repository, which some reviewers will miss.
Before relying on a gate, check which provider keys the CI runner holds and whether test prompts can contain personal data.
Teams that must keep traces, datasets and judge results inside their own network, and cannot accept a license-key dependency, should look at the Apache-2.0 platforms. Opik states that its full platform, including evaluations and prompt management, can be self-hosted without a commercial license. Langfuse's core features and APIs are also available under MIT without limits, but some administration features need an enterprise key, so list those against your requirements before you commit.
- Licensing check: Opik is Apache-2.0 end to end, while Langfuse keeps code under its ee directories on a separate commercial license, so confirm which files you will run.
- Features behind keys: Project-level RBAC, audit logs, data retention policies and SCIM are Langfuse Enterprise license key features, so plan compensating controls if you cannot buy the key.
- Egress and updates: Air-gapped installs need mirrored container images, model endpoints inside the network, and a manual update process.
- Storage sizing: Opik uses ClickHouse, MySQL and Redis, and Langfuse uses ClickHouse, PostgreSQL and Redis, so size ClickHouse memory and disk for your retention window.
- Lost workflow: LangSmith's hosted evaluation UI and vendor support are replaced by your own runbooks and, if needed, a paid support contract with the vendor or a services partner.
Ask each vendor to confirm in writing which features your deployment model includes.
What has to be rebuilt when LangSmith evaluators move
LangSmith evaluators, dataset definitions, judge prompts and annotation rubrics do not transfer automatically. Each target platform stores them differently, so plan to re-create every one of them and keep a written inventory of what LangSmith currently runs. For each evaluator, record the judge model, the judge prompt, the scoring scale, the input fields it reads from the trace, and whether it runs online against production traces or offline against a dataset. In Langfuse, LLM-as-a-judge evaluators are configured against live traces and scores are written back as typed values, so the mapping from LangSmith feedback keys to Langfuse score names needs to be decided up front. In Opik, online evaluation rules and the built-in LLM-as-a-judge metrics for hallucination, moderation and RAG cover some of the same ground, but custom judge prompts still need to be written and tested. In promptfoo, assertions live in the configuration file under version control, so the evaluator becomes code that a reviewer can read and change in a pull request.
Keep judge model access and secrets under your control before the first production run. A judge model is a second LLM call per evaluated trace, so decide which provider account pays for it, which key the evaluation service uses, and where that key is stored. Route judge calls through the same self-hosted gateway you use for production traffic if you have one, so rate limits, budgets and logging apply to evaluation traffic as well. Store provider keys in your existing secrets manager and confirm, for each platform, how it reads them: Langfuse and Opik deploy as containers on your infrastructure, while promptfoo runs locally and calls the model directly from wherever the CI runner or workstation sits. Confirm with your security team whether judge prompts, which often contain customer text from traces, may leave your network under your provider agreements.
Confirm score comparability before you cut over. Run the old and new evaluators side by side on the same traces or dataset items for a fixed period, and compare pass rates, score distributions and the cases where the two disagree. Differences often come from a changed judge model, a reworded rubric or a different temperature setting rather than from the application, so pin the judge model version and settings in both systems before comparing. Ask the team that owns each annotation queue to label a sample blind, then measure agreement between human labels and each automated judge. Only retire the LangSmith evaluator once the team accepts the new scores as a baseline, and record that baseline in the release process so regressions are measured against it.
Running these tools next to LangSmith tracing or a gateway
Most teams do not switch every LLM operations function on the same day. The four patterns below let you keep part of your LangSmith setup running while you add open-source evaluation, tracing or routing components one at a time. Each card names the component, the decision it forces and what to check before you rely on it.
Keep LangSmith tracing, gate releases in CI
Keep LangSmith as the tracing system of record and add promptfoo to your pipeline as a separate evaluation step. promptfoo runs evals on the build machine and calls the model providers directly, so the gate does not depend on a LangSmith dataset or experiment. Check that your CI runners meet the current Node.js requirement, and decide where provider API keys are stored and who can change the pass or fail thresholds.
Send traces to Langfuse over OpenTelemetry
Instrument the application with OpenTelemetry so spans go to a self-hosted Langfuse project while LangSmith callbacks stay in place for the transition. Both systems then hold the same production traces, which lets you compare scores and dashboards before you retire one. Confirm which span attributes your evaluators need, because LangSmith and Langfuse do not map fields identically, and agree on a date for turning off duplicate tracing.
Keep LangSmith datasets during transition
Leave existing LangSmith datasets and their reference answers in place while new experiments run from Langfuse datasets. Export the examples you still need in a neutral format and store them in your own bucket, so a later plan change or account closure does not remove the ground truth. Record which dataset version each release was scored against, because comparisons across two platforms only hold when the inputs match.
Route calls through a self-hosted LiteLLM gateway
Point applications at a self-hosted LiteLLM gateway that exposes one OpenAI-compatible endpoint and logs to Langfuse. Virtual keys, budgets and fallbacks then sit in your infrastructure, and Langfuse receives the traces without LangSmith in the request path. Check which LiteLLM features you need, because SSO, SAML, audit logs and multi-team management sit in the enterprise directory under a separate license.
Run Opik beside LangChain callbacks
Keep existing LangChain callbacks and add Opik's tracer so each chain run also writes to a self-hosted Opik workspace. This suits teams that need evaluations and prompt management under an Apache-2.0 license without an enterprise key. Plan for a second ClickHouse, MySQL and Redis stack, and decide which team owns its upgrades, backups and storage growth before production traffic is sent to it.
Plan the Enterprise LangSmith license
Self-hosted LangSmith is an add-on to LangChain's Enterprise plan and requires a license key, so it is not an option for teams that want to run it without an enterprise contract. If your goal is to avoid that contract, treat the patterns above as the replacement path, not as a LangSmith self-hosting path. Ask your LangChain account team in writing what the license covers, what is included in the Kubernetes deployment and how renewals are priced.
Moving datasets, experiments and evaluators off LangSmith
This sequence moves LangSmith datasets, experiments and online evaluators onto a self-hosted platform while keeping historical scores comparable, so CI gates and release decisions continue without a gap.
Export trace data while bulk export is available
Your team lead or platform owner confirms with LangChain whether your account is still on a plan that includes bulk export. Under the LangSmith data export documentation, customers who signed up after August 3, 2026 can use bulk export only on the Enterprise plan, while earlier Plus customers keep access until February 1, 2027. Schedule the export to an S3-compatible bucket in Parquet format and confirm the files contain the runs and feedback you need for historical comparisons. The step is done when you can open the Parquet files and count runs for a known project and date range.
Stand up the self-hosted evaluation platform
Your platform team deploys the chosen platform in your own cluster or VM. For Langfuse, this means Web, Worker, PostgreSQL, ClickHouse, Redis or Valkey and S3-compatible storage, using Helm on Kubernetes for production. For Opik, it means ClickHouse, MySQL and Redis with Helm or the opik.sh script for local testing. Check the minimum resource guidance for your chosen platform and confirm the version you deploy, since upgrade paths differ by release. The step is done when a test trace sent through the SDK appears in the UI and the storage backups run.
Recreate datasets from exported examples
Your evaluation owner rebuilds each LangSmith dataset as a dataset in the new platform. Use the Parquet export to pull the inputs and reference outputs, then write a small script that uploads them through the SDK or API. Keep the original dataset names and version notes in the new dataset description so reviewers can trace each one back. The step is done when example counts match the LangSmith dataset counts for every dataset you migrate, and a spot check of ten random examples shows matching inputs and outputs.
Port evaluators and judge prompts
Your evaluation owner translates each LangSmith evaluator into the target form. In Langfuse, this covers LLM-as-a-judge scoring of production traces, code evaluators and scores sent via API or SDK. In Opik, it covers the built-in LLM-as-a-judge metrics and online evaluation rules. Custom Python evaluators usually need rewriting against the new scoring interface, and any judge prompt must be copied with its model and temperature settings. The step is done when each evaluator runs on a sample of recent traces and writes scores that a reviewer has checked against the LangSmith results.
Run old and new scores side by side
Your evaluation owner runs the same dataset and the same candidate prompt or model through LangSmith and the new platform for a defined period. Compare scores per example and per metric, and record where judge models or scoring logic differ enough to shift results. Expect some differences, because a judge model change alters scores even when the system under test is unchanged. The step is done when a written note lists each metric, the agreement level your reviewers accepted, and any evaluator you decided to rebuild before cutover.
Switch CI release gates to the new platform
Your release engineer changes the pipeline stage that blocks deploys on regressions. Langfuse CI/CD experiments can block deploys on regressions, and promptfoo can run evaluations in CI for prompt and model changes that need a pass or fail gate. Store API keys in your secrets manager, not in pipeline files, and set the threshold values in version control. The step is done when a deliberately regressed test change fails the gate and an unchanged change passes it, and both runs appear in your records.
Retire LangSmith experiments and confirm retention
Your platform owner stops new experiments and online evaluators in LangSmith, then records the final export date and the location of the archived Parquet files. Confirm with your compliance lead how long the archive must be kept and who can read it, since retention in LangSmith depends on your plan and the new platform's retention is set by your own storage configuration. Remove LangSmith API keys from pipelines and rotate any that remain. The step is done when no pipeline or evaluator references LangSmith, and the archive location and retention owner are recorded in your change log.
What does replacing LangSmith evaluations cost and require?
What are the main cost components of self-hosting Langfuse or Opik for evaluations?
The cost comes from four components: the infrastructure that runs the data stores, the judge model tokens used by LLM-as-a-judge evaluators, the engineering time to port and rebuild evaluators, and an optional enterprise key and support contract. For Langfuse, the stack is ClickHouse, PostgreSQL, Redis or Valkey and S3-compatible blob storage. For Opik, it is ClickHouse, MySQL and Redis. Infrastructure is recurring and grows with trace volume and retention. Judge model tokens are recurring and scale with how many production traces or dataset rows each evaluator scores. Engineering time is mostly one-time. Ask any provider or partner to price each component separately in a written quote.
How much CPU and memory does self-hosted Langfuse need for evaluation workloads?
Langfuse's scaling guide lists minimum resources of 2 CPU and 4 GiB for Langfuse Web, Langfuse Worker and PostgreSQL, 1 CPU and 1.5 GiB for Redis or Valkey, and 2 CPU and 8 GiB for ClickHouse. The guide advises at least 16 GiB of ClickHouse memory for larger deployments. Those are starting points, not sizing for your load. Evaluations run in the worker, so heavier LLM-as-a-judge scoring raises worker CPU needs, and Langfuse recommends scaling workers on CPU load. Confirm sizing against your expected trace volume, retention period and number of evaluators before you buy infrastructure.
Can promptfoo run without a database server?
Yes, for local and CI use. promptfoo's documentation states that evals run on your machine and talk directly with the LLM, so the CLI and library need no database server. The self-hosted promptfoo app is different. It stores data in a local SQLite database that cannot be shared across replicas, and its documentation says it is not recommended for production. It has no RBAC, horizontal scaling, built-in authentication or SSO. Teams that need shared dashboards and access control should plan for the Enterprise on-premise option or confirm what they must build around the open-source app.
How long does a LangSmith evaluation migration take?
It depends on dataset volume and evaluator count, and no duration is set by the open-source projects themselves. A migration that moves a few datasets and one judge prompt is much smaller than one that rebuilds dozens of online evaluators, annotation queues and CI gates. Each evaluator needs its prompt, scoring logic and thresholds reviewed, and each dataset needs its examples checked after export. Ask for the timeline in a written scope that lists the datasets, evaluators, CI pipelines and environments in included, so you can compare what is covered against the estimate.
Can our team migrate LangSmith evaluations without outside help?
A team with Kubernetes, PostgreSQL and ClickHouse experience can deploy Langfuse or Opik using the Helm charts and documentation, and promptfoo can run locally in CI with a single command-line tool. The harder work is judgment: deciding which LangSmith evaluators still measure what matters, rewriting them for the new platform, validating scores against human labels and wiring release gates. Budget for upgrades too. Langfuse's v3 to v4 guide requires ClickHouse 25.12 or later (26.4 recommended), PostgreSQL 15 or later and Redis 7.0 or later, and advises about 3x ClickHouse disk headroom when backfilling historic data.
What does an implementation partner add to an evaluation migration?
DeployOpen plans, implements, customizes, integrates, hardens, backs up, upgrades, documents and supports self-hosted platforms inside your own infrastructure, and most engagements are fixed-scope with deliverables and timelines agreed in writing before work starts. For evaluations, that can cover deploying the stack, porting evaluators and datasets, wiring CI gates and writing runbooks. Ongoing support and upgrades are available as retainers. DeployOpen does not publish a price list, so a quote follows from the written scope. DeployOpen does not issue audit reports or certifications, and your evidence still needs independent review where a compliance assessment applies.
Which Langfuse features need an enterprise license key, and do evaluations depend on them?
The core evaluation features do not. Langfuse states that all core features and APIs are available in the MIT-licensed open source version without limits, including LLM-as-a-judge scoring, annotation queues, datasets with experiments and CI/CD experiments that can block deploys. Features that need an enterprise key include project-level RBAC roles, protected prompt labels, data retention policies, audit logs, server-side data masking and the SCIM and instance management APIs. Those matter if you must control access to evaluation data across teams. The Self-Hosted Enterprise plan adds a named support engineer and a support SLA, priced by Langfuse on request.
How DeployOpen scopes an LLM evaluation migration
DeployOpen scopes an LLM evaluation migration as a fixed-scope engagement, with the scope, deliverables and timeline agreed in writing before any work starts. For this topic, the written scope usually names which evaluation paths move: LLM-as-a-judge on live traces, annotation queues, dataset experiments, and CI gates in promptfoo. It also records which LangSmith datasets, online evaluators and annotation queues must be rebuilt, which are exported, and who signs off on the evaluation criteria. DeployOpen plans and implements the self-hosted target, whether that is Langfuse, Opik or promptfoo, inside your own cloud account, data center or air-gapped network. Each part is written down so that the decision on which platform covers which workflow is made before deployment begins.
Implementation covers the parts that determine whether the evaluations can be trusted after cutover. For self-hosted Langfuse, that means the Web and Worker containers, PostgreSQL, ClickHouse, Redis or Valkey, and S3-compatible blob storage, sized against the scaling guidance. For Opik, it means the ClickHouse, MySQL and Redis stack. For promptfoo, it means deciding whether a local SQLite-backed app is acceptable or whether the Enterprise on-premise route is needed, since the self-hosted app is not recommended for production. DeployOpen hardens these components, configures backups for the databases and blob storage, and documents the runbooks and evidence that your compliance and security teams will request. DeployOpen prepares that evidence but does not issue audit reports or certifications; independent auditors and certification bodies do that work.
Ongoing work is handled through retainers for upgrades, support and continuous testing, which fit well once the platform carries release gates. Upgrade paths such as Langfuse's v3 to v4 migration are scheduled and tested separately from the main rollout. DeployOpen can also run security testing on the AI systems and agents that these evaluations judge, including red teaming against the risks promptfoo covers. To request a quote, send your list of evaluation workflows, the platforms you are considering, your hosting constraints and any license requirements, and DeployOpen will return a written scope and quote based on it.
