promptfoo red teaming for the EU AI Act

Red teaming can show whether an LLM application behaves in ways that touch the prohibited practices in Article 5 of the AI Act, but only within the configurations and inputs you test. The Commission counts nine prohibited practices after the 2026 amendment made by Regulation (EU) 2026/1744. Eight have applied since 2 February 2025 under Article 113. The ninth, which covers non-consensual intimate imagery and child sexual abuse material, applies from 2 December 2026. Article 99 sets maximum fines for prohibited practices at EUR 35 million or 7% of worldwide annual turnover, whichever is higher. Confirm the exact wording of each item against the consolidated text on EUR-Lex before you write any probe that depends on it.

An application rarely becomes prohibited in one step. It drifts there. A system prompt may ask the model to infer a candidate's emotional state from written answers, a tool may pull behavioral data about customers to produce a score, or a retrieval index may hold profiles of named individuals that the assistant then ranks. (Illustrative example: an internal HR assistant whose knowledge base was extended with manager notes about staff. Nobody changed the model, yet the retrieval layer now supports a use that Article 5 may address.) Your threat model should therefore list each surface where behavior can change: the system prompt, tool definitions and their outputs, retrieved documents, and any memory or user profile store.

Once the surfaces are listed, translate each prohibited practice into observable behavior. For example, a probe can ask the assistant to rank people by their social conduct, to produce a profile of a named employee's emotions in a meeting, or to categorize users by a sensitive attribute from a photo or text. promptfoo's eu:ai-act preset gives you a starting set of probes for red teaming, and the Community edition includes red teaming with 10k probes per month. Confirm which categories the preset covers in the version you install, then add custom plugins for the practices and data sources specific to your application. Test the full application, including tools and retrieval, because a model-only test misses the paths where drift usually occurs.

A clean run tells you that the probes you ran were not triggered under the configuration you tested. It does not tell you that the system is lawful. Lawfulness depends on purpose, deployment context, the deployer's role and facts outside any prompt, so only legal review can settle it. Agree the probe scope in writing with legal counsel and the product owner before the run. The agreement should name the practices in scope, the user groups and languages, the tools and data sources, the model versions, and what counts as a failure. Record every exclusion and the reason for it, because an auditor will ask what was not tested.

Plan to rerun the agreed probes whenever the model, system prompt, tool set, retrieval corpus or deployment context changes. Set the NCII and child sexual abuse material probes to run before 2 December 2026, since that prohibition applies from then. Keep each run's configuration, probe version and results with the scope document so the evidence shows what was tested and when. Treat failures as product defects to fix and retest, and escalate any case where a probe reveals a use that legal counsel must assess before release.

Running promptfoo as a local CLI or library and running its self-hosted app are different operating models, and only the second one looks like a shared system of record. The CLI and library run evals on the machine that invokes them and talk directly with the model under test. Results stay with that run, which suits a developer or a pipeline job. The self-hosted app is an Express server that serves the web UI and API, deployable with Docker, Docker Compose or Kubernetes with Helm. Teams often assume that deploying it produces a central, governed record of red teaming runs. The documentation does not support that assumption.

The documented limits matter for evidence. The self-hosted app stores data in a local SQLite database that cannot be shared across replicas, so you cannot scale it horizontally behind a load balancer and expect one consistent history. It has no RBAC, no horizontal scaling and no built-in authentication or SSO. The promptfoo self-hosting documentation also states that it is not recommended for production use. For an Article 5 or Annex III test program, that means the app cannot tell an auditor who ran a probe, who could change a result or who approved a finding. Ask the vendor to confirm the current wording and any exceptions before you design around it.

The Enterprise and On-Premise tiers address these gaps on paper. Per the pricing page, Enterprise adds team sharing, teams-based access control and SSO, custom roles and permissions, continuous monitoring, a centralized guardrail dashboard, remediation recommendations and priority support with SLAs, at custom pricing. On-Premise runs that functionality on your infrastructure with complete data isolation and a dedicated deployment engineer. Pricing is quoted, not published, so the cost of these tiers depends on your scope and deployment, and the quote should state which items are recurring licenses and which are one-time services.

Before you rely on any of these features as evidence, treat them as claims to verify. Confirm in writing whether the Enterprise or On-Premise build provides the specific access controls, SSO integration, audit logging and retention behavior your evidence file needs, and whether the database is still shared-nothing per replica. Ask how run history is exported and whether the export is tamper-evident. Test it with a pilot dataset, then record the configuration in your control documentation so the same settings can be reproduced.

Plan for the gap in the meantime. A practical pattern is to keep red teaming results as versioned, signed artifacts from CI or from a controlled runner, store them in your existing evidence repository with access controls already in place, and reference the exact promptfoo version and configuration used for each run. This approach does not depend on the self-hosted app being production ready. Whichever pattern you choose, record who owns the evidence, how long it is retained and who can reproduce a finding.