What a promptfoo migration involves
A promptfoo migration converts the evaluation work your teams already run into promptfoo configurations and test cases, then moves the results of those runs from individual machines into a setup the whole team can use. The source material is usually one of three things: prompt tests kept in code or notebooks, evaluations tracked in spreadsheets with scores pasted by hand, and custom eval scripts written for a single model or application. Each needs a different conversion path, and the first task is an inventory that records what each suite checks, which models it calls and who runs it.
The usual triggers are a team that has outgrown spreadsheets, scripts that only one person knows how to run, and a need for CI gating or shared results that someone can review. Small suites with a single owner are often converted in-house, since the work is mostly rewriting test cases into YAML and checking the output matches. Shared self-hosted or Enterprise setups, assertion mapping at volume and CI integration are where partners typically add the most, because they involve access design, pipeline changes and many judgment calls about what a passing assertion actually proves.
promptfoo runs locally and calls model providers directly, so the migration must settle where provider API keys and eval data live before any shared setup goes live. Decide whether keys sit in CI secret stores, a vault or developer environments, who may rotate them, and which prompts, datasets and red team outputs may leave a given machine. Confirm these answers with your security team and with the provider terms your organization has accepted, because a migrated suite that sends sensitive data to an external model is a policy decision, not a configuration detail.
What the work includes
The migration breaks into a small set of work packages, each ending in a reviewable artifact. The exact list, quantities and acceptance checks are fixed in the written scope before work starts.
Inventory of existing tests
We catalog every prompt test, spreadsheet, notebook and custom eval script in scope, recording its inputs, expected outputs, model targets and owners. This inventory sets the boundary of the migration and shows which suites depend on undocumented behavior. Check that it includes the scripts your team forgets about, such as ad hoc notebooks and CI jobs that call models.
Conversion into promptfoo configuration
Each in-scope suite is rewritten as promptfoo configuration files covering prompts, providers, test variables and default test settings. Where logic lives in custom scripts, we decide whether it becomes a provider, a transform or a separate step. Ask which items stay in code and which move into declarative configuration.
Mapping expected outputs to assertions
Exact-match checks, regex rules, JSON schema checks, similarity scores and model-graded rubrics each replace a different kind of expected output. We record why each assertion type was chosen so reviewers can challenge weak checks. Confirm with your domain owners which outputs are truly deterministic and which need graded criteria.
Validation against known results
Converted suites are run and compared with results your team already trusts, such as a previous spreadsheet run or a reviewed batch of outputs. Differences are logged as either conversion defects or real behavior changes that need a decision. Agree the tolerance for model variability in writing before the comparison begins.
CI/CD wiring where in scope
When pipeline integration is included, we add promptfoo runs to the pipelines you name, with pinned versions, stored secrets and pass or fail thresholds. Pipeline wiring is optional and must be listed in the written scope. Check which runner, secrets store and branch rules the pipelines must follow.
Shared setup and access model
Where a shared instance is in scope, we deploy it as self-hosted promptfoo or as Enterprise or On-Premise, and document who can view, run and change evals. Self-hosted promptfoo uses a local SQLite database that cannot be shared across replicas and has no built-in authentication or RBAC, so access controls must be designed around it. Confirm the edition and access model with your security team first.
Handover documentation
You receive the configuration files, a conversion log that traces each old test to its new form, runbooks for running and updating suites, and handover notes for the owning team. The conversion log is the record your reviewers will use, so ask for it to cover every item in the inventory, including items that were retired or left out.
What drives effort and duration
This table lists the factors that move migration effort for promptfoo eval suites, why each one matters, and how to keep it contained before conversion starts.
| Driver | Why it moves effort | How to keep it down |
|---|---|---|
| Number and format of source suites | Each spreadsheet, prompt file, notebook or custom script needs its own mapping into promptfoo test cases and assertions, and inconsistent formats multiply that mapping work. | Freeze a representative subset of suites first, convert those end to end, and agree a single target format before converting the rest. |
| Number of providers and models | Each provider needs its own configuration, credentials handling and output checks, and results from different models must be compared on equal terms. | Agree the provider list in writing before conversion starts and drop models that no longer serve the evals. |
| Depth of custom eval logic | Scripts that compute scores, call external tools or hold state must be reworked into promptfoo custom assertions or providers, which need review and testing. | Separate simple string checks from logic that truly needs code, and retire checks that nobody uses before porting them. |
| Subjective versus deterministic assertions | Deterministic checks port quickly, while rubric-based or model-graded assertions need rubric design, judge model choice and calibration against human labels. | Start with deterministic assertions, and limit model-graded checks to a small set with documented rubrics and a sample of human-reviewed results. |
| Number of users and teams sharing results | Shared results need agreed naming, tagging, ownership and access expectations, and Community edition runs locally, so sharing needs a separate setup decision. | Name the teams, owners and result locations before migration, and decide whether a shared self-hosted or Enterprise setup is in scope for this phase. |
| Target edition and deployment | Moving from local runs to a self-hosted app involves database, authentication and hosting choices, while the self-hosted SQLite store is not recommended for production and lacks built-in authentication and SSO. | Confirm the target edition and deployment method early, and ask whether Enterprise or On-Premise features such as SSO are required before design begins. |
| CI integration | Pipelines must run promptfoo with pinned versions, secrets for provider keys, and pass or fail rules that match the team's release gates. | Choose one CI pipeline to convert first and agree the pass or fail criteria with the owners of the release process. |
| Compliance evidence needs | Assessors may expect records of test sets, versions, assertions, results and red team coverage mapped to frameworks such as the OWASP LLM Top 10 or NIST AI RMF, which means more documentation and retained outputs. | Ask the compliance lead which evidence is required, then design result retention and naming around those needs, noting that DeployOpen prepares evidence but auditors issue any assessment. |
Migration phases and outputs
The migration runs in six ordered phases, and each phase closes with a named artifact the client can review before the next one starts.
Inventory and classify existing tests
The team reviews every prompt test, spreadsheet, and custom eval script in scope and records its inputs, expected outputs, model targets, and owners. Each item is classified by how it maps to promptfoo, such as direct conversion, assertion rewrite, or retirement. The step closes with a signed-off inventory that lists what moves, what changes, and what is dropped.
Design the target deployment
Together, the client and DeployOpen choose where evals will run: local-only use, a self-hosted promptfoo app, or the Enterprise or On-Premise tier. The design records the reasons for the choice, the data handling rules for eval inputs and outputs, and the providers in use. The step closes with a written target design that the client approves.
Convert suites into promptfoo configurations
DeployOpen rewrites each approved item as a promptfoo configuration with providers, prompts, test cases, and assertions, and moves shared values into variables and environment settings instead of hard-coded text. Secrets are kept out of configuration files. The step closes with a versioned repository of configurations that runs locally without errors.
Validate parity against known results
Converted suites are run against the outputs that the original tests produced, and every difference is either fixed or documented as an accepted change. Assertion results are compared case by case, and flaky cases are identified and marked. The step closes with a parity report that lists matched, changed, and rejected results for each suite.
Set up shared deployment and access
For shared use, DeployOpen deploys the chosen promptfoo setup in infrastructure the client controls, configures persistent storage, and connects identity and access where the chosen edition supports it. Self-hosted instances keep their SQLite data on persistent storage that is not shared across replicas, so the design must account for that limit. The step closes with a running environment, tested access paths, and a recorded list of who can view and run what.
Hand over with runbooks
DeployOpen delivers runbooks for running suites, adding tests, rotating provider keys, upgrading promptfoo, and restoring the data store, and the client team runs a supervised session on each. Open items and ownership are recorded in the handover pack. The step closes when the client team confirms it can run and maintain the suites without DeployOpen.
What teams keep in-house and what they outsource
This table lists each migration part of a promptfoo move, says whether teams usually keep it in-house or outsource it, and names the skills it needs, so you can see where your own people must stay involved.
| Migration part | Usually | Skills it needs |
|---|---|---|
| Suite inventory | In-house, with partner help to catalog | Someone who knows each spreadsheet, script and test owner, plus a record of which suites still matter |
| Configuration conversion | Outsourced or shared | Working knowledge of promptfoo config files, YAML, and the scripts or formats being replaced |
| Assertion design | In-house, always with partner support | Domain experts who define pass criteria; the converted assertions encode their judgment, so they must review every one |
| Provider and secret handling | Outsourced, with in-house sign-off | Model API administration, secret management, and least-privilege key handling in your environment |
| Self-hosted or Enterprise deployment | Outsourced or shared | Docker, Compose or Kubernetes with Helm, plus database and storage planning; Enterprise licensing is a separate commercial decision |
| CI wiring | Outsourced, owned in-house afterward | CI pipeline engineering, Node.js version pinning, and gating eval runs on your build system |
| Access model | In-house decision, partner implementation | Identity administration and SSO; self-hosted instances lack built-in authentication and RBAC, so plan access before sharing |
| Handover training | Outsourced delivery, in-house ownership | Documentation, run books and teaching the team to add, review and retire test cases |
What usually goes wrong
Most migration problems come from decisions made while moving tests, not from the tool itself. Check each item below before the first shared run.
Converted assertions that behave differently
Assertions copied from spreadsheets or scripts often change meaning when rewritten, for example through different string matching, whitespace handling, thresholds or regular expression anchors. A case can pass in the original suite and fail in promptfoo, or the reverse, with no model change. Run the old and new suites against the same recorded outputs and review every mismatch before retiring the original.
Local SQLite treated as production
Self-hosted promptfoo stores data in a local SQLite database that cannot be shared across replicas. The self-hosted build also lacks RBAC, horizontal scaling, built-in authentication and SSO, and the documentation does not recommend it for production use. Decide the shared target before migration, and confirm with the vendor which setup your team will operate.
Provider keys in shared configs
Eval configurations reference model providers, and it is easy to paste API keys directly into YAML that is committed, copied into tickets or shared with contractors. Keep keys in environment variables or a secrets manager, reference them by name in configs, and scan the repository and its history for literal keys before the suite is shared.
Losing spreadsheet owner context
Spreadsheet tests often carry context that exists nowhere else: who wrote the expected answer, which product rule it reflects and when it was last reviewed. Once rows become test cases, that context disappears and nobody can judge a failure. Record an owner, source and reviewer for each migrated case, and keep a mapping from each original row to its test ID.
Node.js version below the minimum
The promptfoo npm release 0.124.1 requires Node.js 22.22.0 or newer. Developer laptops, CI runners and base container images often pin older runtimes, which can cause installation failures or errors that look unrelated to the evals. Set the Node.js version explicitly in CI and images, and print the runtime version as the first pipeline step.
How the work changes by starting point
Prompt tests kept in spreadsheets or notebooks usually hold inputs, expected outputs and informal pass or fail judgments. Conversion turns each row into a test case with explicit assertions.
- Conversion: map input columns to test variables and each pass or fail judgment to an assertion, such as an exact match, a contains check or a model-graded check with the criterion written down.
- Data handling: sheets often contain customer or internal text. Decide which rows may be sent to a model provider before the first run.
- Deployment: a local Community run is usually enough for the first pass. Community is free under the MIT license and includes all model providers.
- Confirm: whether the Community red teaming allowance of 10k probes per month fits your planned runs, and which providers your team needs.
Custom scripts often hold logic that promptfoo does not express directly, such as custom graders, data loaders and retry handling. Migration keeps that logic and moves it behind promptfoo's provider and assertion interfaces.
- Conversion: port custom graders into custom assertions and custom model calls into custom providers, keeping the business rule unchanged and reviewed by the original author.
- Data handling: scripts often read files or call APIs directly. Move credentials into environment variables or a secrets manager, not into configuration files committed to version control.
- Deployment: run the suite in CI with a pinned promptfoo version. The current npm release requires Node.js 22.22.0 or newer.
- Confirm: with the vendor, whether the custom provider and assertion interfaces in your target version match the scripts you are porting.
A local setup keeps configuration and results on one developer's machine. A shared instance needs configuration in version control and results stored in one place that the team can reach.
- Conversion: move configuration files into a repository, reference them from a single pipeline job and standardize output locations.
- Data handling: local results may contain API keys or sensitive model outputs. Review and scrub them before importing anything into a shared instance.
- Deployment: the self-hosted app runs with Docker, Docker Compose or Kubernetes with Helm. Its SQLite store cannot be shared across replicas, so plan for a single replica with a persistent volume.
- Confirm: whether the self-hosted app's lack of built-in authentication, RBAC and SSO is acceptable for your users, since that gap usually points toward Enterprise.
Enterprise adds team sharing, teams-based access control, SSO, custom roles and permissions, and continuous monitoring. Migration maps existing suites to teams and roles that you define before the first import.
- Conversion: group existing suites by owning team, then map each team to a role, so access is set before any results are imported.
- Data handling: decide where evaluation and red teaming results are stored. On-Premise runs the Enterprise functionality on your infrastructure with complete data isolation.
- Deployment: identity provider integration and role mapping usually take the most planning. A dedicated deployment engineer is included with On-Premise.
- Confirm: in writing, which SSO protocols are supported, how roles map to your directory groups, and what support terms apply. Enterprise pricing is custom, so request a quote that lists the migration scope.
In a restricted network, model providers must be reachable from inside the boundary, and every artifact must be brought in deliberately. Community runs evaluations on the local machine and talks directly to the model, so the network path to each provider is a design decision.
- Conversion: confirm which model endpoints can be reached through approved egress or a proxy. If only private or local models are allowed, map the suite to those providers first.
- Artifacts: mirror the npm packages, container images and Helm charts into the boundary, and agree how updates enter it.
- Data handling: eval inputs and outputs may contain regulated data. Define retention, export and deletion rules before the first run.
- Confirm: whether On-Premise fits your environment, and whether the vendor supports offline installation and updates for your target version.
What the client needs to provide
Most delays in a promptfoo migration come from missing inputs, not from the configuration work itself. Before the engagement starts, the client should be ready to supply the following: access to the existing prompt tests, spreadsheets, notebooks and custom eval scripts; a sample of expected outputs or written sign-off criteria for each suite; and a list of model providers and API accounts, with one named person accountable for each set of keys. Without the expected outputs, the team can convert the files, but nobody can say whether the converted assertions still mean what the old checks meant.
The target environment needs the same attention. Confirm where the shared instance will run, whether it is a Docker, Docker Compose or Kubernetes deployment, which network paths it needs to reach model providers and identity systems, and which egress rules the security team will approve. Also decide, in advance, which edition the team will use. The Community edition is free and covers evaluations and red teaming, while Enterprise adds team sharing, access control and SSO. Identity decisions, such as which directory will supply users, should be settled before migration begins, because they shape how the shared instance is set up.
Two named owners matter more than any tool setting. The eval criteria owner decides what a passing answer looks like, signs off the converted assertions and resolves disagreements about old results. The shared instance owner runs the deployment, keeps the service current and controls who gets access. The criteria owner matters more because a promptfoo configuration can run cleanly and still test the wrong thing. If that person is unavailable, the migrated suites will produce numbers that look authoritative but have no agreed meaning.
Ask the client to confirm each item in writing at kickoff. Useful questions include: who holds each provider key and how it is rotated, which existing suites are retired rather than migrated, and who approves changes to shared criteria after handover. An illustrative example: a team with four spreadsheets and two scripts may find that one spreadsheet's pass rules were never written down, and that gap should be closed before any conversion work is quoted.
Common questions about promptfoo migration
How much does a promptfoo migration cost, and what drives the price?
The cost follows from the written scope, and DeployOpen does not publish a price list. The main cost components are the migration work itself, which covers converting prompt tests, spreadsheets and custom eval scripts into promptfoo configurations, plus any CI/CD wiring, shared setup and handover. The size and messiness of your existing suites, the number of model providers and whether a shared self-hosted instance is included move the cost most. Separately, model API spend is paid to your providers and recurs with every run. Ask for a quote that lists deliverables, assumptions and what your team provides.
How long does a promptfoo migration take to complete?
The duration depends on the scope, and the schedule is agreed in writing before work starts. The biggest schedule drivers are how many suites exist, how inconsistent their formats are, how many providers they call and whether results must feed into CI pipelines. Access approvals in your environment, such as cloud accounts, secrets stores and runner access, often set the pace more than the conversion itself. Reviewer availability matters too, because someone on your side needs to confirm that migrated assertions still test what the originals tested. Confirm the timeline against your own constraints during scoping.
Can our team migrate existing evals to promptfoo without outside help?
Yes, a team with engineers who know the LLM applications can do this in-house. promptfoo is MIT licensed and runs locally, so there is no license purchase needed to start. The work requires learning the configuration format, translating each original test into an assertion that checks the same behavior, and wiring runs into CI. Teams with limited time or no owner for eval quality often find that the validation step takes longer than expected. A partner can take on conversion and validation while your engineers keep ownership of the suites.
What does DeployOpen need from our team during a migration?
The engagement needs access to the source material, including spreadsheets, prompt files, custom scripts and any existing results. You also need to provide provider accounts and API keys, which should stay in your own secrets system rather than moving into migration deliverables. A named owner should review expected outputs for each suite, because only your team knows what a correct answer looks like. Access to the CI runners, repositories and the environment where the shared instance will run is also required. Your written scope should list each of these items with a responsible person and a date.
Can we share a local promptfoo setup without buying Enterprise?
You can deploy the self-hosted promptfoo app with Docker, Docker Compose or Kubernetes with Helm, but it has real limits for shared use. It stores data in a local SQLite database that cannot be shared across replicas, and it has no RBAC, horizontal scaling, built-in authentication or SSO. The promptfoo documentation says it is not recommended for production use. Sharing without Enterprise therefore means building and operating your own access controls around the instance. Enterprise adds team sharing, teams-based access control and SSO, and On-Premise runs that functionality in your infrastructure. Confirm current feature details with promptfoo before you decide.
What do 10k monthly red teaming probes mean for migrated suites?
The Community edition includes red teaming with 10k probes per month, along with all LLM evaluation features and model providers. Any migrated suite that runs red team scans draws from that monthly allowance, so scan frequency and scope should be sized before you rely on it. A practical approach is to run a defined set of red team scans in CI and schedule broader scans less often. Check with promptfoo how probes are counted across your configurations before you finalize the plan. If continuous monitoring or a centralized dashboard becomes necessary, those features are part of Enterprise.
How DeployOpen delivers promptfoo migrations
DeployOpen moves existing prompt tests, spreadsheets and custom eval scripts into promptfoo configurations, then helps teams move from local runs to a shared, self-hosted setup in infrastructure they control. The work follows a fixed scope agreed in writing before it starts.
How we scope
- The written scope lists each existing eval source, such as prompt test files, spreadsheets and custom scripts, and the promptfoo configuration it will become.
- Scope names the model providers and API keys each suite uses, so credential handling is agreed before any suite is moved.
- Scope states whether the target is local runs, CI/CD execution or a shared self-hosted or Enterprise instance, and what that choice changes.
How we build
- Each existing suite is converted into promptfoo test cases and assertions, with outputs compared against the original results to confirm coverage.
- Custom eval scripts that cannot map directly are rewritten as custom providers or assertions, and the changes are reviewed with your team.
- Shared deployments are built on the self-hosted app with Docker, Docker Compose or Kubernetes with Helm, and the database and access setup are documented before cutover.
How we hand over
- Handover includes the configuration files, a run guide for local and CI/CD use, and a record of which original tests were mapped or retired.
- Your team receives documentation on where data is stored, including the SQLite database of a self-hosted instance and its backup location.
- A walkthrough with your engineers covers running evals, reading results and adding new test cases without DeployOpen involved.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers, and they can cover promptfoo version changes after handover.
- Support can include checking a pinned promptfoo version against the Node.js requirement of each new release before your pipelines adopt it.
- Ask in the quote whether support covers the shared instance, the CI/CD pipelines and the model API configuration, and who owns each.
