Which open-source options replace Braintrust evals and experiments?
If a renewal, a licensing change or a residency rule is pushing your evaluations off Braintrust, split the job in two. promptfoo covers evaluations run on a laptop or in CI, while Opik or self-hosted Langfuse cover tracked experiments, datasets and LLM-as-a-judge metrics that a team reviews over time. Most teams need both halves, and the choice of tracked platform decides how much of Braintrust's experiment workflow you keep.
Opik is the most complete drop-in for the tracked half. Its Apache-2.0 repository includes the full platform, covering the server backend, web application, tracing, evaluations and prompt management, and the README states it can be self-hosted without a commercial license. Langfuse's core features and APIs are MIT licensed, but its ee directories fall under a separate enterprise license, and project-level RBAC, audit logs, data retention and SCIM are among the features that need an enterprise license key. Check that list against the roles and audit requirements your reviewers expect before you commit.
Treat promptfoo's self-hosted app with care. It is an Express server with a web UI and API, but its documentation stores data in a local SQLite database that cannot be shared across replicas, and it lacks RBAC, built-in authentication and SSO, and is not recommended for production. Run promptfoo from the CLI or CI for evaluation gates, and put shared results in the tracked platform. Confirm the current promptfoo release and Node.js requirement with the project before you pin versions in your pipeline.
Mapping Braintrust evaluation capabilities to open-source options
This table maps each Braintrust evaluation capability to promptfoo, Opik and Langfuse, and flags where the open-source option is partial or depends on a license. Read each row as a question to answer in your own environment: does the open-source piece cover the workflow your team actually runs, and what does it leave for you to build or operate?
| Capability | promptfoo | Opik | Langfuse |
|---|---|---|---|
| Experiments and comparisons | Local and CI eval runs compare prompts and models; self-hosted app stores results in SQLite and is not recommended for production | Experiments tracked in the Apache-2.0 platform with comparison views; confirm report depth against your current Braintrust views | Experiments run against datasets and compared in the UI; confirm which experiment views your team uses on the MIT core |
| Datasets | Test cases defined in config files and version-controlled alongside code | Datasets managed in the platform and linked to experiments; confirm import from your Braintrust export | Datasets managed in the MIT core; confirm dataset versioning needs against your current workflow |
| Scorers and custom code | Assertions and custom scoring code run locally; confirm your scorer code runs unchanged in the promptfoo format | Custom metrics via the SDK and @track decorator; port scorer functions and confirm their inputs match | Scores attached to traces and experiments via the SDK and API; port scorer logic and confirm score schemas |
| LLM-as-a-judge metrics | Model-graded assertions supported; confirm judge model choice and cost per run with your provider | Built-in judge metrics for hallucination, moderation and RAG; confirm they match your existing baselines | Evaluators can call an LLM judge; confirm judge prompts and calibration against your baseline scores |
| Online scoring on production traffic | Not applicable | Online evaluation rules score live traces; confirm rule limits and sampling in your deployment | Scores can be recorded on production traces; automated online evaluators depend on your setup, so confirm them |
| CI gates | Designed to run in CI/CD pipelines; the main fit for pull-request gates | Can run evaluations from code; confirm how pass or fail thresholds are wired into your pipeline | Evaluation runs can be scripted through the API; threshold gating is usually custom work |
| Red teaming | Automated red teaming with 10k probes per month on Community; references OWASP LLM Top 10, NIST AI RMF and EU AI Act | Guardrails and moderation metrics; dedicated red teaming is not a core feature, so confirm before relying on it | Not applicable |
| Prompt management | Not applicable | Prompt management with a playground is in the platform; confirm versioning against your release process | Prompt management with labels; protected prompt labels need an Enterprise license key |
| Access control and SSO | Community has no built-in SSO or RBAC; Enterprise and On-Premise add them | Confirm user roles and SSO options in the self-hosted build before deployment | Project-level RBAC, audit logs and SCIM need an Enterprise license key; SSO documentation is separate |
Which option fits which evaluation workflow?
Braintrust combines experiments, scorers and datasets in one hosted control plane. The three options below cover the same ground in different shapes: a local and CI-first test runner, a full self-hostable tracing and evaluation platform, and an open-core observability stack with experiments layered on top. Pick the one that matches where your evaluations run and where your results need to be kept.
Local and CI evaluations with promptfoo
promptfoo runs evals on the machine or CI runner that executes them, and it talks directly to the model providers, so prompts and outputs do not pass through a vendor service. It ships as a CLI, a library and a CI/CD component, which makes it the natural home for regression gates on pull requests. The trade-off is that it is a test runner rather than a results warehouse: the self-hosted app stores data in a local SQLite database that cannot be shared across replicas and has no RBAC or built-in SSO. Check how your team will share results before you rely on it for governance. Note that OpenAI acquired promptfoo, and its founders state it will remain open source.
Full platform with Opik
Opik is Apache-2.0 licensed, and its README states the full platform, including tracing, evaluations and prompt management, can be self-hosted without a commercial license. It offers LLM-as-a-judge metrics for hallucination, moderation and RAG, plus datasets and experiments tied to traces. The cost of ownership sits in its data layer: ClickHouse, MySQL and Redis must be deployed, sized, backed up and upgraded by your team. Confirm the Helm chart and Docker Compose options against your Kubernetes standards before committing to the stack.
Open-core observability with Langfuse
Self-hosted Langfuse keeps its core features and APIs under the MIT license, with code in the ee directories under a separate enterprise license. Datasets and experiments sit beside traces and prompt management, so teams already tracing in Langfuse can evaluate in the same place. Plan for PostgreSQL, ClickHouse, Redis or Valkey and S3-compatible storage, plus the Web and Worker containers. Advanced controls such as project-level RBAC, audit logs and SCIM require an enterprise license key, so list those features against your access requirements early.
How engineering, ML and compliance teams split the work
Application engineers usually own the prompt regression suite that runs on every pull request. In the open-source split, promptfoo fits this role because it runs evaluations on a developer machine and in CI/CD pipelines, and its documentation states that evals run locally and talk directly with the model provider. Engineers keep test configurations in the repository next to the prompts they test, so a failing assertion blocks a merge the same way a failing unit test would.
- Test definitions in Git: Prompts, test cases, assertions and provider settings live in version-controlled files that code review can inspect.
- Local and CI runs: Developers run the CLI or library on their own machine, and the same command runs in the pipeline with provider keys stored as CI secrets.
- Node.js runtime: The current promptfoo release requires Node.js 22.22.0 or newer, so pin the runtime in CI images before upgrading.
- Results handed off as artifacts: Pipelines export results as files that can be attached to a pull request or sent to a tracking tool when a team wants history.
- Scoring functions in code: Custom assertions written in JavaScript or Python stay in the repository and are reviewed like any other code.
Confirm with your own pipeline that provider API keys never appear in logs and that the Node.js version in your CI image matches the release you pin.
ML teams own the question of whether a change improved quality over time, which is where Braintrust's experiments view was most used. Opik and Langfuse both record experiments, datasets and LLM-as-a-judge scores as tracked results, so this group compares runs against a stored baseline rather than against the last local run. Ownership here includes the dataset versions and the judge prompts, because a score is only comparable when those stay fixed.
- Datasets as versioned assets: Golden datasets are created in the tracking platform, and each experiment references the dataset version it used.
- Judge metrics: Opik provides LLM-as-a-judge metrics for hallucination, moderation and RAG, and Langfuse supports LLM-as-a-judge evaluators, so the team picks one platform for judge scores and documents the judge model.
- Baselines: The baseline experiment is named and frozen so new runs show a delta against a known reference.
- Hand-off from promptfoo: Results generated in CI can be pushed into the tracking platform when the team needs shared history across releases.
- Infrastructure owned elsewhere: The platform team runs the database and storage behind the tracker, so ML owners need an agreed way to request capacity and retention changes.
Confirm which tracking platform your team will use before moving any baseline, because scores are not portable between the two products without an export step.
Security and compliance leads review two things: the red teaming evidence that shows the model was tested before release, and the location of the prompts, responses and judge calls. promptfoo's red teaming documentation references the OWASP LLM Top 10, the NIST AI Risk Management Framework and the EU AI Act as frameworks that support systematic testing. This group decides which framework mapping is acceptable evidence for their own program and does not treat a tool's label as proof of compliance.
- Red teaming scope: Ask which plugins and strategies were run, against which model versions, and which findings were accepted or remediated.
- Data location: Traces and judge outputs held in Opik or Langfuse stay where the team deploys them, while evals run by promptfoo send requests directly to each model provider, so the provider's data terms apply.
- Access control: The self-hosted promptfoo app stores data in SQLite and has no RBAC or built-in SSO, so shared deployments need an access layer in front of it. Langfuse keeps project-level RBAC and SSO behind an Enterprise license key.
- Audit trail: Record who approved each baseline and each red teaming exception, because auditors ask for that history.
- Licensing review: Check the license of each component, since promptfoo is MIT licensed and Langfuse's enterprise directories fall under a separate license.
Confirm with your independent assessor which evidence format they accept. DeployOpen prepares controls and evidence, but audit opinions and certifications come from the assessor.
How to carry scoring functions and baselines across
The scoring logic and baseline results are the assets to carry across, because they define what "better" means for your application. Start by inventorying every Braintrust scorer, whether it is a built-in LLM-as-a-judge prompt, a heuristic code scorer or an external scorer, along with the prompt versions and the experiment that each one produced the baseline from. Confirm the exact export options against Braintrust's current documentation before you plan the work, since the formats and fields available for scorer definitions and experiment results may differ from what you expect. Record, for each scorer, its input fields, its output range and the threshold a release had to meet, because those thresholds are what the new stack has to reproduce.
Rewrite each scorer in the target tool's native form. Deterministic checks such as JSON validity, required fields or regex rules usually map to promptfoo assertions that run locally and in CI. Rubric-based judges map to promptfoo's model-graded assertions or to Opik and Langfuse LLM-as-a-judge metrics when results must be tracked over time. Expect some scorers to need manual rewriting, since prompt wording, judge model choice and output parsing may not translate one to one. Store the rewritten scorers in version control next to the prompts they evaluate, and keep a mapping table from each old Braintrust scorer name to its new definition.
Before you retire anything, re-run the baseline experiment on the new stack with the same dataset, the same prompt versions and the same model settings, then compare scores case by case. Investigate every disagreement, because a difference can come from the judge model, a parsing change or a real regression in the scorer. Keep both systems running in parallel until the scores agree within a tolerance your team has agreed in writing, and keep the Braintrust export as an archived record of the original baseline. Only then remove the old scorers from your release gates.
How to move evals off Braintrust without losing baselines
This sequence keeps your scoring logic and historical baselines intact while moving evaluations to self-hosted tooling, so each new result can be checked against what Braintrust produced before you cut over.
Inventory datasets, scorers and experiments
Engineering and the evaluation owners list every Braintrust project, dataset, scorer (code-based and LLM-as-a-judge), prompt version and experiment that feeds a release decision or CI gate. Record the scorer source code, model and judge prompt versions, thresholds and which experiment serves as the baseline for each application. Done means each evaluation has a written owner, a named baseline experiment and a list of the CI jobs that read its results.
Export datasets, experiments and scores
The team exports datasets, experiment results and per-row scores from Braintrust before the contract or access changes, using the export paths documented on the sibling migration and export pages, which hold the detailed steps. Store exports with their Braintrust project and experiment IDs and timestamps. Done means every baseline experiment you listed has a complete export that someone other than the exporter has opened and row-counted against the Braintrust UI.
Stand up the chosen self-hosted platform
Platform engineering deploys the target in your own infrastructure. Promptfoo runs locally and in CI with a Node.js 22.22.0 or newer runtime, and its self-hosted app stores data in SQLite, which it notes is not recommended for production. Opik and Langfuse run tracked experiments on their own stacks: Opik uses ClickHouse, MySQL and Redis, while Langfuse uses PostgreSQL, ClickHouse, Redis or Valkey and S3-compatible storage. Done means the platform answers health checks, has authentication configured and is backed up.
Recreate scorers and datasets
Engineers port each scorer into the target's format, whether a promptfoo assertion, an Opik LLM-as-a-judge metric or a Langfuse evaluator, and import the exported datasets. Judge prompts and thresholds are copied verbatim before any tuning. Done means each scorer runs on a fixed sample and returns the same pass or fail decision as the Braintrust version on the rows you check by hand.
Run old and new evaluations in parallel
For a defined period agreed by the team, both systems score the same prompts, model versions and datasets. The team compares aggregate scores, per-row disagreements and latency for each scorer, and investigates every difference by reading the traces. Done means the team signs off in writing that the new results match the old baseline within the tolerance set in step one, or that each gap is explained and accepted.
Move CI gates to the new platform
DevOps changes pipeline jobs so the release gates call the new evaluation runner and read thresholds from the same versioned configuration. The team tests a failing and a passing build on a branch to confirm the gate blocks and passes as intended. Done means the pipeline records the new run IDs, and the old Braintrust gate is disabled in every repository that used it.
Retire Braintrust access
Security and the platform owner remove Braintrust API keys from CI secrets and developer environments, revoke team members and confirm the final exports are archived under your retention policy. Confirm with Braintrust what happens to your data at contract end and request written deletion if required. Done means an access review shows no active Braintrust credentials and the archive location and retention owner are documented.
Running the new evals next to the rest of your stack
Moving evaluations off Braintrust rarely happens in one cut. The cards below cover the parts that keep running during the transition: tracing, model routing, and who signs in. Each one is a decision to make before the old system is switched off.
Keep Braintrust tracing during the overlap
Evaluation results and production traces can be moved on different schedules. Many teams keep Braintrust for live tracing for a period while Langfuse or Opik runs the new experiments, so that production visibility does not drop during the switch. Check which dashboards, alerts and on-call runbooks read from Braintrust before you decide when to stop sending traces there. Ask both vendors how they handle duplicate spans if you send to two backends at once.
Send traces with OpenTelemetry
Opik supports OpenTelemetry, and Langfuse documents an OpenTelemetry path for its trace ingestion. Sending spans through an OpenTelemetry Collector lets one instrumentation layer feed more than one backend, which reduces rework when you change tools again. Confirm which span attributes each platform maps to its own trace and score fields, because evaluation scores and metadata often need explicit mapping. Test this with a non-production workload first.
Route model calls through a gateway
A gateway such as LiteLLM gives applications one OpenAI-compatible endpoint, so evaluation runs and production traffic use the same keys, budgets, rate limits and fallbacks. Without this, spend for the same model can appear under different accounts and evaluation runs can hit different provider limits than production. LiteLLM can send logs to Langfuse, which keeps cost data in the same place as traces. Decide which team owns virtual keys and budgets before you point evaluation jobs at the gateway.
Plan identity and SSO integration
Evaluation tools hold prompts, datasets and sometimes customer data, so access should follow your identity provider. Langfuse documents SSO in its self-hosting security section, but some SSO and role features need an Enterprise license key, so confirm with the vendor which methods your deployment would use under the open-source license. Opik and LiteLLM have their own access controls, and LiteLLM's SSO and SAML features sit in its enterprise offering. Map roles from your identity provider before inviting evaluation teams.
Run evaluations from CI
promptfoo runs locally and in CI/CD pipelines, which lets a pull request fail when a prompt change lowers a score against a stored baseline. Keep the baseline file under version control so that the threshold is reviewed like any other code change. Results that matter for audit should also be recorded in the tracked platform, so that a reviewer can see the run history. Confirm where the CI runner stores model credentials and whether test prompts contain regulated data.
What does replacing Braintrust evals cost and take?
What are the cost components of replacing Braintrust evals with open source?
The cost has four parts: infrastructure, people, software licenses and external services. Most open-source options in this group carry no license fee for the core evaluation features, so the spend shifts to compute, storage and the staff who run and change the system. For Opik, the README states the full platform, including evaluations, can be self-hosted without a commercial license, so the line item is hosting and operations. For Langfuse, core features are MIT licensed, but some advanced features need an Enterprise license key, which is a separate contract. For promptfoo, the Community edition is free and runs locally, while Enterprise is custom-priced. Ask each vendor which features you need and whether they sit behind a paid tier.
Which costs are one-time and which recur after go-live?
One-time costs cover discovery, mapping existing Braintrust datasets and scorers, building the evaluation harness, wiring CI gates and the initial hardening and testing of the deployment. Recurring costs cover cloud compute and storage for the platform, backups, upgrades, on-call time, and any subscription for an enterprise license or support plan. Upgrades recur more than many teams expect. The Langfuse upgrade guidance notes that major versions follow dedicated migration guides, and the v3 to v4 guide requires ClickHouse 25.12 or later and advises about 3x ClickHouse disk headroom when backfilling historic data. Budget that work as a recurring cost, not a single project.
How many people does a self-hosted evaluation platform need?
The answer depends on whether you run one tool or several, and on how much you self-host. A promptfoo CLI run in CI needs an engineer who owns the eval suites and a pipeline job, but no platform team. A tracked platform such as self-hosted Langfuse or Opik adds an operator who understands the data stores behind it. Langfuse lists PostgreSQL, ClickHouse, Redis or Valkey and S3-compatible storage as dependencies, and Opik uses ClickHouse, MySQL and Redis. Someone must patch, back up and scale these. Confirm with the vendor and your own environment whether your team already runs these stores, because that decides whether the work is new skill or extra load.
What drives the effort of replacing Braintrust evals up or down?
Effort scales with four factors. The first is the number of datasets, scorers and experiments to migrate, and whether their results must match historic baselines. The second is how many applications send traces, since each instrumentation change is separate work. The third is the security and identity requirements. The Langfuse license key docs list project-level RBAC, audit logs, data retention and SCIM as Enterprise features, and self-hosted promptfoo has no RBAC, SSO or built-in authentication, so teams with access control needs must plan for it. The fourth is scale. Langfuse's scaling guide lists minimum resources and recommends splitting langfuse-web under heavy ingestion, which adds deployment work. Narrowing scope early usually lowers effort the most.
Can our team replace Braintrust evals without outside help?
A small team can do it alone if it has Kubernetes or Docker experience, owns its eval suites, and can run a database-backed service on a schedule. Starting with promptfoo locally and in CI is the lowest-risk first step, because the documentation states evals run on your machine and talk directly with the LLM. Moving tracked results and LLM-as-a-judge metrics into a self-hosted platform takes more operations skill. Many teams bring in a partner for the deployment, hardening, backup design and upgrade plan, then keep the evaluation logic in-house. DeployOpen plans, implements, hardens, backs up, upgrades and documents self-hosted platforms in your own infrastructure, and most engagements are fixed-scope.
How long does replacing Braintrust evals take at different scopes?
No researched duration is available for this migration, so treat any figure you hear with caution and ask for a scope-based plan. Timing depends on the number of evaluations, the volume of trace data, whether historic results must be reproduced, how many CI gates depend on the current scores, and whether the platform needs high availability. A pilot that runs promptfoo in one pipeline can be scoped separately from a production tracked platform. Ask the vendor or partner to break the timeline into discovery, build, parallel run, cutover and handover, and to state which assumptions each phase depends on.
What should a fixed-scope quote for this migration include?
A fixed-scope quote should list each deliverable in writing before work starts. Ask for the inventory of datasets, scorers and experiments in scope, the target platform and version, the deployment method (Docker Compose for testing, Kubernetes with Helm for production), and the data stores and their sizing. Require a plan for exporting baselines and proving that new scores match old ones on a sample. Confirm the handling of secrets, access control, backups, and who operates each component after handover. Ask which Enterprise features, if any, are needed and who pays for those licenses. Also ask for the parallel run period, acceptance criteria and the support model after go-live.
Does a self-hosted evaluation setup help with compliance evidence?
Self-hosting keeps evaluation data, traces and red teaming results inside your environment, which supports data residency and review. However, a self-hosted platform does not make you compliant by itself. The promptfoo red teaming documentation references the OWASP LLM Top 10, the NIST AI Risk Management Framework and the EU AI Act as frameworks for systematic pre-deployment testing. DeployOpen prepares systems, controls and evidence for compliance assessments but does not issue audit reports or certifications; independent auditors and certification bodies do. Ask your assessor which evidence they need, such as access logs, retention settings and test records, and plan for those outputs in the quote.
What your team must provide before a migration starts
Before any scorer or dataset moves, your team must secure a complete export of Braintrust data and a named owner for every scorer, dataset and experiment that feeds a release decision. Ask the Braintrust account owner to confirm that your organization can export experiments, datasets and logs in the formats you need, and that the export user has the permissions the export actions require. Then assign one engineer per scorer to document its inputs, prompt or code, thresholds and the baseline experiment it is compared against. Scorers written as Braintrust code will need rewriting as promptfoo assertions, Opik or Langfuse LLM-as-a-judge metrics, or plain functions in your CI, so the owner must be able to explain what each one measures before it is rebuilt.
Next, confirm the infrastructure. Opik and Langfuse both self-host on Kubernetes with Helm for production, and Opik depends on ClickHouse, MySQL and Redis, while Langfuse depends on PostgreSQL, ClickHouse, Redis or Valkey, and S3-compatible storage. Your platform team needs a cluster with capacity for those stateful services, cloud accounts or data center space where they will run, and object storage with a bucket policy you control. Confirm that someone on the team can operate ClickHouse, which covers backups, disk headroom, upgrades and query tuning, and that someone can run PostgreSQL. If neither skill exists in-house, the gap should be filled by a hire, a training plan or an implementation partner before cutover, not after.
Finally, make two decisions in writing. The first is data residency: name the region and the accounts where traces, prompts and datasets may be stored, and confirm that evaluation runs calling external model APIs send only what your policy allows. The second is retention: set how long traces, scores and datasets are kept, since Langfuse documents data retention policies as an Enterprise license key feature, and Opik's retention behavior should be confirmed with its documentation for your version. Record these decisions in the migration plan so that the export, the new deployment and the audit evidence all reflect the same rules.
How DeployOpen scopes and delivers an evals replacement
DeployOpen plans, implements, integrates, documents and supports self-hosted open-source platforms inside your own infrastructure, so evaluation runs, scorers and results stay under your control. Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
How we scope
- Inventory the Braintrust experiments, datasets, scorers and CI gates you actually use today.
- Decide which evaluations move to promptfoo locally or in CI and which move to Opik or Langfuse tracked experiments.
- Write down the deliverables, environments, hosting targets and acceptance criteria in a written, fixed-scope agreement.
How we build
- Deploy the chosen platform in your cloud account, data center or air-gapped network using its documented install path.
- Port scoring functions and LLM-as-a-judge metrics, then rerun baseline datasets to confirm results match previous runs.
- Wire evaluations into your CI pipeline and integrate with identity, secrets and storage you already operate.
How we hand over
- Document architecture, configuration, dataset and scorer ownership, and the steps to add a new evaluation.
- Provide runbooks for upgrades, backups and restores of the database and object storage the platform depends on.
- Deliver evidence of test results and configuration to support your internal review or a compliance assessment by an independent auditor.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers after handover.
- Retainers can cover version upgrades, including major migrations that follow the vendor's dedicated upgrade guides.
- Ask any quote to name who operates the database, storage and workers after go-live, since that drives recurring effort.
