promptfoo in GitHub Actions

The comparison answers one question: did this pull request change what the LLM application produces for the same inputs? To answer it, the eval runs against the base branch and against the pull request head using the same test cases. The results then show which outputs changed, which assertions now fail, and which passed before and still pass. Reviewers read that diff alongside the code change, so the prompt, model or retrieval edit is judged on observed behavior rather than on the diff alone.

A fair comparison needs three things to stay fixed. The dataset must be identical on both sides, so the same prompts, variables and expected cases run each time. The providers must be identical, meaning the same model names, parameters and endpoints, or any output change may come from the model setting rather than the prompt change. The assertions must be identical too, because a loosened check can turn a regression into a pass. If any of these differ between base and head, the comparison measures the difference in configuration, not the difference in the application.

Changing the eval config inside the pull request can hide regressions. The promptfoo configuration file is part of the repository, so a pull request can edit the test cases, remove a failing assertion, or switch to a cheaper model, and the head run will look clean. The base run then compares against a baseline the pull request has already rewritten. Teams should treat changes to eval configuration, datasets and assertion files as a separate review item, and consider requiring that those files come from the base branch for the comparison run, or that a reviewer approves them explicitly.

Confirm how the action selects the base revision before you rely on it. The official GitHub Action documentation describes how it determines the base for a pull request, including which commit it checks out, how it handles forks, and whether it reads the configuration from the base or the head. Read that section against your branching model and your repository permissions. Then run a test pull request that deliberately changes one prompt and one expected output, and confirm that the report shows the change as a regression against the base you intended. Do not assume the default behavior matches your workflow until you have seen it work on your own repository.

The first decision is to separate two layers. Action inputs control how the job runs: which promptfoo version executes, which directory the job starts from, which environment values are passed in, and whether results are reported back to the pull request. The promptfoo config controls what is tested: the prompts, the providers (the model endpoints each prompt is run against), the test cases, and the assertions that decide pass or fail. Keep the two layers separate in your repository so that a change to one does not force edits to the other. Confirm the current input names, their defaults and which inputs are optional against the action's own documentation. Older blog posts and forks often show inputs that have since been renamed or removed, and copying an old workflow is a common reason a job runs successfully while testing nothing useful.

Next, decide where the config file lives and who owns it. A single root-level config is easy to find, but it becomes crowded when several applications share a repository. A config in each application directory keeps prompts and assertions next to the code they describe, but it requires a rule for which job reads which file. Write the config path explicitly in the workflow so a reviewer can see from the workflow alone which file is being evaluated, instead of relying on a default. Also decide whether prompts are written inline in the config or kept in separate files that the config references. Separate prompt files are easier to review in diffs and make it possible to limit runs to the files that changed, which is covered below.

Then settle the scope: which prompt files, which providers and which test cases run on every pull request. Not every prompt needs every model on every change. Provider calls are billed by the provider, so a matrix of many prompts across many models multiplies both cost and run time. Ask the product and risk owners which model versions are approved for production, make those providers the default for pull request runs, and move broad comparisons to a scheduled job. Promptfoo runs on the machine where the job executes and calls the LLM directly, so the runner needs outbound network access to each approved provider endpoint. Confirm that access with the network team before the first run, especially if runners sit in a restricted subnet.

Finally, decide whether eval runs are limited to changed prompt files. Limiting runs to changed files reduces cost and time, but it can miss regressions caused by shared inputs: a common template, the config itself, a shared dataset, or a provider setting that many prompts depend on. Define the trigger paths up front and include the config file and any shared test data. A practical rule is that a change to the config or to a file it references widens the run to the full suite. Prove the rule with a test pull request that changes only a shared file, and confirm that the job starts. Also pin the runtime. The promptfoo npm release 0.124.1, published on 8 October 2026, requires Node.js 22.22.0 or newer, so set that Node.js version on the runner and confirm which promptfoo version the action installs before you rely on its results.

Every pull request that touches a prompt, a test file or a provider setting can trigger a full promptfoo run, and each run sends every test case to every configured provider. The model spend recurs with each push, not with each change. A team that pushes five commits to one pull request pays for five complete matrices unless something prevents repeat calls. The main cost is model provider billing for input and output tokens. Judge models used by some assertions are billed separately and are easy to overlook, so ask which assertion types make additional model calls. CI runner minutes are a smaller recurring cost that grows with the same matrix.

promptfoo offers caching of provider responses, so an identical request made again can be answered from a stored result instead of a new provider call. Confirm in the promptfoo documentation where the cache is stored, for example on the runner's local disk or in a location that persists between workflow runs, and how entries are invalidated. The cache key is derived from the request inputs. In practice, the prompt text, the model name and the provider parameters should all influence the key, so changing any of them should produce a miss. Test this before relying on it: change one parameter in a branch and confirm that a fresh provider call occurs. If you use GitHub's own cache mechanisms to persist files between runs, confirm how retention and eviction apply to those files as well.

Caching lowers cost, but it can hide provider changes. If a provider updates a model behind the same name, a cached response will keep the old output, and an eval may pass on behavior the provider no longer delivers. Cached results therefore answer a narrower question: did the inputs you control change in a way that alters the outputs? To answer whether the provider itself has changed, schedule uncached runs, for example on the default branch at a fixed interval, with caching disabled and results recorded separately from pull request runs. The schedule is a decision to make with the owners of the evaluation suite. Those scheduled runs carry the full per-run cost, so include them in the budget from the start.

Before enabling the cache, decide who owns the cache configuration, which branches may read or write it, and how often uncached runs happen. Check the workflow logs for cache hits and misses after a few pull requests, and confirm with the promptfoo documentation where those counts are reported. When you request a quote for this work, ask how the implementation partner will configure cache storage, the uncached schedule and the reporting that shows whether the cache is being used. Ask also who pays for the model calls in scheduled runs and whether those runs use a separate API key from pull request runs.