What a promptfoo rollout involves
Implementing promptfoo for evals and red teaming is mostly test design, model access and pipeline work. The software is open source under the MIT license and runs completely locally: evals run on your machine and talk directly with the LLM. That means the rollout does not start with a server build. It starts with deciding what the application must do, how each behavior will be checked, and which model endpoints the tests will call. The core components are the test suite (prompts, test cases and variables), the providers that point at each model or application endpoint, and the assertions that decide pass or fail. Assertions usually take the most effort, because generic checks rarely catch the failures that matter to your application. Red team scans add a second layer: you scope which plugins and strategies run, and map results to recognized frameworks such as the OWASP LLM Top 10, the NIST AI Risk Management Framework and the EU AI Act, as the promptfoo red teaming documentation references.
Pipeline work covers running evals in CI/CD on each prompt or model change, storing results, setting thresholds that fail a build, and deciding who reviews red team findings. Confirm with your engineers which CI system and secrets store will hold provider keys, and which Node.js version your runners provide, since the current npm release requires Node.js 22.22.0 or newer.
Teams usually trigger this work when a new LLM feature ships, when a model or prompt changes, when an audit or customer questionnaire asks for evidence of testing, or when ad hoc spot checks stop scaling. Most teams build the first suites in-house because the test cases depend on knowledge of the product. A partner is worth considering when the scope spans several applications, when red team scope must be defined against a framework quickly, or when internal engineers lack time for assertion design. Migration of existing eval scripts, upgrades, SSO and hardening are covered on their own pages.
What the work includes
Each work package below ends with a deliverable the client keeps. The scope, deliverables and timeline are agreed in writing before work starts, so the items listed here are the starting point for that agreement.
Discovery of LLM applications
Discovery inventories each LLM application, its prompts, model calls, tools, data sources and owner. The output is a scoped list of test targets and a written decision on which applications enter the first phase. Check that each target has a named owner, a non-production environment and a person who holds its provider API keys before work begins.
Test suite and assertions
The deliverable is versioned promptfoo configuration files with test cases, datasets and an assertion library tied to the behavior the application must show and must avoid. Assertion design is often the largest effort driver, because useful checks depend on business rules that someone must define. Ask who approves expected outputs and which assertions use a grader model.
Provider and model configuration
This package sets up provider definitions for each model endpoint under test, with credentials read from environment variables and model versions pinned in configuration. The deliverable is a providers file and a secrets note that lists where each key is stored and who can rotate it. Confirm whether any endpoints are private or on-premises, since model API spend is billed by each provider and recurs with every run.
CI/CD integration
The deliverable is a pipeline template that runs evals on pull requests or model changes, fails the build when assertions regress and stores results as artifacts. Decide which pipeline platform hosts the runners, where API keys are injected, and which branches trigger full red team runs instead of smaller smoke evals. Confirm that the runner image pins the Node.js version the promptfoo release requires.
Red team baseline scan
The deliverable is a baseline red team scan of the agreed targets, using plugins mapped to the OWASP LLM Top 10 and any other frameworks the client selects, along with the raw results. Probe volume matters for planning, because the Community edition includes 10k probes per month. Agree the scan size, frameworks and targets in the written scope.
Reporting and documented handover
The deliverables are a scan report that summarizes findings and test coverage, a runbook for running evals and scans, and a handover session with the team that will own the configurations. The report describes what was tested and what the results show. It does not serve as an audit opinion or certification. Ask that the runbook covers upgrades, key rotation and regression triage.
What drives effort and cost
Each row names one driver of promptfoo rollout effort or spend, why it moves the total, and one way to keep it under control.
| Driver | Why it changes effort or cost | How to keep it down |
|---|---|---|
| Number of LLM applications and prompts | Each application needs its own test cases, expected behaviors and maintenance as prompts change, so effort grows with the count of applications and prompt versions. | Start with the one or two applications that carry the most risk and reuse shared assertion templates across the rest. |
| Providers and models in the test matrix | Every provider and model multiplies the number of eval runs and the model API calls billed to your account, and each one needs its own configuration and credentials. | Compare only the models you are choosing between or have in production, and keep a smaller matrix for pull request runs. |
| Deterministic versus model-graded assertions | Deterministic checks such as regex, JSON schema or exact-match tests are cheap to run and quick to write. Model-graded checks need a grader model, add API spend on every run and take more time to tune so their verdicts agree with reviewers. | Use deterministic assertions wherever the expected output allows, and reserve model-graded rubrics for qualities that cannot be checked mechanically. |
| Red team scope and probe volume | Red teaming cost follows the plugins, strategies and probes you enable. The Community edition includes 10k probes per month, so a broad scan can reach that limit quickly, and larger volumes need an Enterprise conversation. | Plan probe budgets per application and per month before the first scan, and prioritize plugins that map to your threat model. |
| CI pipeline count and run frequency | Each pipeline that runs evals on every commit or pull request adds compute time and model API calls. Frequent runs across many repositories raise both runner load and spend. | Run the full suite nightly or on release branches, and run a small smoke subset on each pull request. |
| Need for shared access and team features | A single engineer running the CLI locally needs little setup. Shared dashboards, team sharing, teams-based access control, SSO and custom roles are Enterprise features, and a shared self-hosted instance needs its own authentication and access design. | Confirm which people genuinely need a shared view before buying or building access controls, and keep local CLI runs for everyone else. |
| Data sensitivity and deployment location | promptfoo runs on your machine and calls providers directly, so prompts, outputs and red team results may contain regulated or confidential data. Keeping that data inside your infrastructure requires self-hosting or On-Premise, with added hardening and backup work. | Classify eval datasets first, use synthetic or redacted data where possible, and confirm with your security team which deployment location each class of data may use. |
| Compliance mapping | Mapping red team results to frameworks such as the OWASP LLM Top 10, the NIST AI Risk Management Framework or the EU AI Act requires evidence, traceability and a documented review process. This is mostly staff time for your compliance and security leads. | Agree the framework list and evidence format at the start, and generate reports from a fixed set of configurations so mapping is repeated, not rebuilt. |
| Enterprise pricing | Enterprise and On-Premise pricing is custom and is set from the agreed scope, so it cannot be estimated from the Community edition. Budget for it as a quote item, not a fixed fee. | Ask for a written quote that lists seats, probe volume, support level and deployment model, and compare it against the Community edition limits for your planned workload. |
Project phases and outputs
The sequence moves from agreed scope to a tested evaluation suite, a CI gate, a red team baseline and a documented shared deployment that your team can run.
Confirm scope and edition
We review your applications, model providers, data sensitivity and who will use the results, then document scope in writing. The edition decision between Community, Enterprise and On-Premise is made here, because it determines whether you need team sharing, SSO and role-based access, and it shapes the shared deployment design. Output: a signed scope document and an edition recommendation for your confirmation.
Build a pilot suite on one application
We configure promptfoo for one application, covering its providers, prompts, test cases and first assertions, running locally against your model endpoints. This pilot checks that the configuration and provider access work as expected. Output: a version-controlled configuration with a documented baseline run and a short list of issues your team has reviewed.
Build out assertions
We work with your domain experts to turn expected behaviour into assertions, such as deterministic checks, rubric-based grading and regression cases drawn from real failures. Effort here depends on how well your expected outputs are defined, so this phase often takes the most review time. Output: an assertion library with ownership notes and a record of which checks need human judgement.
Integrate evals into CI/CD
We add the evaluation run to your pipeline with pinned promptfoo and Node.js versions, secrets handling for provider keys and a defined pass or fail threshold. Output: a pipeline that runs on each change, a documented gate definition and a test run showing both a passing and a failing build.
Establish a red team baseline
We run promptfoo red teaming against the pilot application using the plugins and strategies agreed in scope, referencing the OWASP LLM Top 10, NIST AI RMF or EU AI Act where your governance requires it. The baseline records what was tested and what was found. Output: a baseline report, triaged findings and a schedule your team can repeat.
Decide and roll out the shared deployment
The self-hosted app stores data in a local SQLite database that cannot be shared across replicas and lacks built-in authentication, RBAC and SSO, so the design must address storage, access and backups before any team uses it. We document the chosen architecture, deploy it in your infrastructure with Docker, Docker Compose or Kubernetes, and roll it out to the first teams. Output: a running shared instance, access rules and a rollout log.
Hand over and document
We deliver runbooks for configuration changes, version pinning, red team reruns and restore from backup, then walk your engineers through them. Output: documentation, a handover session with recorded attendance and a list of open items with owners.
What to keep in-house and what to outsource
This table splits promptfoo rollout work into parts teams usually keep and parts they often outsource, with the skills each part needs, so a team lead can decide where to start.
| Work area | Usually keep in-house | Usually outsource | Skills needed |
|---|---|---|---|
| Test case design | Writing domain-specific prompts, expected behaviors and test data from real product workflows | Reviewing coverage gaps against the test plan | Prompt and product knowledge, domain expertise, YAML configuration |
| Acceptable behavior | Deciding what the model may say, refuse or escalate for your users and regulators | Documenting decision criteria as testable rules | Product ownership, risk and compliance judgment, legal input |
| Assertion framework design | Agreeing which outcomes are measured and which thresholds count as a pass | Designing reusable custom assertions, graders and helper scripts | Node.js or Python, LLM evaluation methods, test engineering |
| Provider configuration | Choosing which approved model APIs and keys the team may use | Setting up providers, routing, retries and rate limits per environment | API integration, secrets handling, cloud and vendor account administration |
| CI/CD patterns | Owning pipeline gates, release criteria and who can override a failing eval | Building pipeline jobs, caching, reporting artifacts and pinned versions | CI tooling, Node.js 22 or newer runtime management, shell scripting |
| Red team scan design | Defining which risks, user roles and data sets matter for the application | Configuring plugins, strategies and scan schedules within probe limits | Application security, OWASP LLM Top 10 knowledge, adversarial testing |
| Self-hosted deployment | Approving where eval data and red team outputs may be stored | Deploying the web app with Docker, Compose or Helm and hardening the host | Docker or Kubernetes operations, Helm, database administration, network security |
| Results and triage | Owning findings, severity decisions, remediation tracking and sign-off | Producing summary reports and trend data for review meetings | Engineering judgment, incident handling, risk reporting |
What usually goes wrong
Most promptfoo rollouts fail on assertion quality, model spend, version pinning, credential handling, ownership or hosting choices. Each risk below comes with the step that prevents it.
Keyword-only assertions
Assertions that check for a required word or a banned phrase pass answers that are fluent and confident but wrong, or that repeat harmful guidance in softer language. Prevent this by pairing keyword checks with assertions on facts, output schema and refusal behavior. Review a sample of passing outputs at each release.
Hidden spend in the model bill
Model-graded assertions call a judge model for every test case, so each added case adds judge calls on top of application calls. Test matrices multiply prompts by models, variables and repeats, and red team probes can use up a monthly allowance early. Set a probe and token budget per run, cap matrix size in CI, and ask for a cost estimate per suite before expanding coverage. The Community edition includes 10k red team probes per month (promptfoo pricing page).
Unpinned versions and ownership drift
A floating promptfoo version in CI can change assertion behavior, output formats or runtime requirements between runs. The npm release 0.124.1, published 8 October 2026, requires Node.js 22.22.0 or newer. Pin the version, test upgrades on a branch, and confirm the MIT license and roadmap you rely on, since OpenAI announced its acquisition of promptfoo on 9 March 2026.
Provider keys in config files
Provider API keys written into promptfoo configuration files end up in version control, CI logs and shared eval outputs. Keep keys in your secrets manager, inject them as environment variables at run time, and scan repositories for key patterns before merge. Rotate any key that has appeared in a committed config, and limit which pipelines can read provider credentials.
Suites nobody owns after handover
Eval suites and red team configurations often leave with the people who wrote them, and failing tests then get skipped or edited until they pass. Assign a named owner to each suite, document which assertions protect which behavior, and require review for changes to thresholds. Agree before handover who triages failures and who approves new providers.
SQLite treated as production storage
The self-hosted promptfoo app stores data in a local SQLite database that cannot be shared across replicas and has no RBAC, horizontal scaling, built-in authentication or SSO. The documentation says it is not recommended for production use. Treat it as a single-user or evaluation instance, or scope Enterprise or On-Premise for shared use, and confirm the access model before inviting a team.
How the work changes by starting point
Effort is lowest here. One engineer installs promptfoo locally, writes a configuration and runs evals against the providers they already use. The software runs on the developer's machine and talks directly with the model, so nothing is hosted for you.
- Setup: install on a Node.js version that meets the 22.22.0 minimum, then write the first suite.
- Effort driver: writing assertions that reflect actual failure modes, which takes more time than installation.
- Main constraint: results stay on individual machines, so sharing them depends on manual exports.
- Confirm first: which providers and API keys each developer will use, and whether prompts may leave the developer's machine.
Effort rises because pipelines need pinned versions, secrets handling and a stable definition of pass and fail. promptfoo can run in CI/CD pipelines, so the work is mostly pipeline design and agreement on which tests block a merge.
- Version pinning: fix the promptfoo version in the pipeline so a release does not change results mid-sprint.
- Secrets: provider keys belong in the CI secret store, scoped to the jobs that call models.
- Spend: every run that calls a model incurs API cost, so limit which suites run on each change and track red team probe use against the Community allowance of 10k probes per month.
- Confirm first: the runner Node.js version, the gating policy for failed evals, and who reviews a red team finding before release.
A shared instance changes the work from tooling to access management. Team sharing, teams-based access control, SSO and custom roles are Enterprise features, and On-Premise delivers the Enterprise functionality on your infrastructure. Pricing is custom, so scope and quote come first.
- Identity: map your identity provider groups to promptfoo teams and roles before users are onboarded.
- Hosting choice: decide between Enterprise and On-Premise, since that decision sets who operates the platform and where data sits.
- Support: Enterprise includes priority support with SLAs, so confirm the response terms in writing.
- Migration: existing local suites must move into a shared project structure, which adds effort.
- Confirm first: the identity provider, the administration model and the data residency requirement.
The self-hosted app is an Express server that serves the web UI and API, deployable with Docker, Docker Compose or Kubernetes with Helm. Packaging is the easy part. The app is not ready for multi-user production on its own, so the work includes building the controls around it.
- Storage: data sits in a local SQLite database that cannot be shared across replicas, so the app cannot scale horizontally.
- Access: the app has no RBAC and no built-in authentication or SSO, so access control must sit in front of it.
- Persistence: plan a durable volume and a backup routine for the database file.
- Production status: the project documentation states that self-hosted use is not recommended for production.
- Confirm first: whether the goal is a single-user or small internal instance, and which authentication layer your platform team will operate.
In a restricted network the central question is reachability. Evals run locally and call the model directly, so the runner must reach every endpoint in the suite, including private ones. Each dependency must be available inside the boundary before the first run.
- Packages: the npm package and the Node.js runtime need an internal mirror or an approved transfer path.
- Endpoints: private model endpoints must be reachable from every machine or runner that executes evals and red team probes.
- Data isolation: On-Premise runs the Enterprise functionality on your infrastructure with complete data isolation, which changes the team and hosting model.
- Probe budget: Community red teaming includes 10k probes per month, so plan probe use for the restricted environment.
- Confirm first: which endpoints and packages are reachable, whether runners may make outbound calls, and whether a shared instance is needed.
What the client needs to provide
The schedule for a promptfoo rollout depends mostly on what the client supplies in the first two weeks. Start with access to every LLM provider and approved model endpoint the evaluations will call, with API keys issued specifically for testing so that eval spend and rate limits can be tracked separately from production traffic. Without those keys, the first suites cannot run against real models, and the team loses time to access requests rather than to test design.
Next, provide sample prompts and the expected behavior for each application, along with representative inputs that include edge cases such as malformed text, ambiguous requests and known failure modes. Assertions are written from these expectations. If they arrive late or describe behavior only in general terms, the first test suite will be built on guesses and will need rework. Name one owner for each application who can answer questions about expected behavior, and one owner for the evaluation results who decides what a failure means and who acts on it.
On the technical side, confirm a build environment with a Node.js version that the promptfoo release in use supports. The npm release 0.124.1, published on 8 October 2026, requires Node.js 22.22.0 or newer, so confirm this against the version you plan to pin. Also provide a CI system with access to pipeline configuration, since evaluations are usually wired into merge checks. Access to the pipeline definitions should be granted before the integration sprint, not during it.
Two decisions gate the rest of the plan. The first is budget approval for model API spend, because red teaming and large test matrices generate many paid calls and the team needs a spending ceiling before the first full run. The second is the edition and hosting model: Community, Enterprise or On-Premise, and whether the instance runs locally or as a shared service. Confirm the edition choice with promptfoo's current pricing page, since it determines whether team access control, SSO and a shared database are in scope from the start.
Common questions about promptfoo implementation
How much does a promptfoo implementation cost?
The fee depends on the written scope, not on a standard package. The main inputs are how many eval suites and providers need configuring, whether red teaming and CI/CD integration are included, how many assertions must be written against your use cases, and whether you need a shared self-hosted instance with SSO or only local runs. DeployOpen does not publish a price list, and a quote follows from the written scope. Ask for the fee broken into scoping, build, handover and any retainer, and for each item to name its deliverables. Software licensing is separate: the Community edition is free, while Enterprise pricing is custom.
How long does a promptfoo rollout take?
Duration varies with the number of applications under test, the state of your existing prompt tests, and how many assertions need to be designed from scratch. A single team running local evals in one pipeline is a much smaller job than several teams moving onto a shared self-hosted instance with authentication, backups and CI gates. Your timeline should come from the agreed scope, and you should ask for milestones tied to deliverables. Plan for the time your own reviewers need to validate assertions, since that step often sets the pace.
Can our own team run a promptfoo rollout without a partner?
Yes, many teams do. The software is open source under the MIT license and can run as a CLI, a library or in CI/CD. The harder work is usually designing useful assertions, deciding what to red team, and standing up a shared deployment that is secure enough for production. If your team has engineers familiar with Node.js, CI pipelines and LLM application behavior, a partner can be limited to reviewing the design or hardening the shared instance. Be realistic about staff time, and confirm who will own test cases after handover.
What does promptfoo need from our staff during implementation?
You will need a product owner who can describe expected LLM behavior, engineers who can provide access to test environments and CI runners, and someone who can approve provider accounts and data handling rules. Security or compliance staff should review red team scope and where outputs are stored. Expect reviewers to check assertions against real expected outcomes. Ask the partner to list these roles and hours in the written scope so your staff time is planned, not assumed.
What ongoing spend should we expect after promptfoo goes live?
Ongoing spend has two main parts. The first is model API usage, because evals and red team probes call the LLM provider you configure, so cost scales with suite size, run frequency and the models you test. The second is edition choice: Community is free and includes all evaluation features, providers and 10k red team probes per month with community support, while Enterprise and On-Premise use custom pricing. Ask for a projection of API calls per pipeline run and confirm which edition covers your probe volume.
Does test data leave our environment when we use promptfoo?
The software runs locally and calls the LLM provider directly, so evaluation prompts and responses go wherever that provider processes them. Whether test data leaves your environment depends on the provider you choose, including hosted APIs versus models you run in your own infrastructure. Confirm each provider's data handling terms with its vendor, and decide which datasets may be sent to external services before the build starts. DeployOpen can design the provider configuration and data rules to match that decision.
How DeployOpen delivers this service
DeployOpen scopes a promptfoo rollout in writing before work starts, then builds the eval suites, provider connections and CI/CD integration inside your own infrastructure. Each step ends with deliverables your team keeps and can run without us.
How we scope
- Engagements are fixed-scope, with the promptfoo use cases, providers and eval targets agreed in writing before work starts.
- We map which applications, prompts and models need evals and red teaming, and which frameworks such as the OWASP LLM Top 10 apply.
- The written scope states whether promptfoo runs as a local CLI, a shared self-hosted app or an Enterprise setup, and why.
How we build
- We write promptfoo configurations with test suites, providers and assertions that reflect your application's expected behavior.
- We connect evals and red team runs to your CI/CD pipeline, with provider API keys stored in your secrets handling, not in configs.
- We confirm model API usage per run with you during build so spend is visible and tests stay within an agreed budget.
How we hand over
- You receive the configurations, assertion logic, pipeline definitions and a runbook describing how to add new test cases.
- Documentation covers how to read eval results and red team outputs, and who on your team reviews failures and findings.
- Handover includes a walkthrough with your engineers so they can extend the suites and pin the promptfoo version in CI.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer, covering promptfoo releases and Node.js requirements.
- If your team needs a shared self-hosted instance, we plan its database, backups and access model as a separate scoped item.
- Confirm with us in the quote which support tier, response expectations and upgrade cadence apply to your environment.
