Production or staging penetration testing: which to choose first
Test staging first only if it truly mirrors production; otherwise, test production with controls that limit the risk. NIST SP 800-115 asks you to weigh three factors: the impact an exploit could have on availability, whether testing could expose sensitive personal data, and how closely the non-production environment matches the live one. It also notes that techniques likely to cause denial of service should probably run against a non-production system, but that differences between test and production can cause missed vulnerabilities. If your staging copy uses different identity providers, network paths, WAF rules or data volumes, a clean result there tells you little about live exposure.
Some programs leave you no choice. DORA threat-led penetration testing under Article 26 covers several or all critical or important functions and runs on live production systems, with scope validated by the competent authority. If your entity has been identified for TLPT, plan production windows, data handling and recovery procedures with the authority and your testers before work begins. The PCI SSC penetration testing guidance allows a separate environment identical to production, but only if the tester confirms the same application and network-layer controls exist there and records this in the rules of engagement. Confirm that statement in writing, and check it again after each release.
In practice, many teams combine both. Run aggressive or disruptive techniques on a verified replica, then validate the fixes on production, because PCI SSC guidance expects exploitable findings from a separate environment to be corrected on production and the testing repeated there. Ask any provider to name the environment differences it has documented, agree on off-hours windows and stop conditions for production work, and confirm how test data is masked. Those decisions, recorded before testing starts, matter more than the choice of location.
How production and staging penetration tests compare
This table compares production and staging penetration tests across the factors that change risk, fidelity and evidence. Read each row as a question to settle before scoping, and use the takeaway to decide which factors need a written decision from the rules of engagement.
| Factor | Production test | Staging or identical non-production test |
|---|---|---|
| Primary goal | Validate exploitability and detection against the live environment that customers and regulators depend on | Find and fix weaknesses earlier, with less risk to live services, provided the copy reflects production |
| Scope | Limited to systems and functions the owner has agreed to expose to exploitation, often narrowed by critical function | Can cover a wider set of hosts, data flows and failure cases, but only the scope that the copy actually mirrors |
| Environment fidelity | Exact by definition, so configuration drift, live integrations and real controls are all tested | Must match production. NIST SP 800-115 warns that differences between test and production can cause missed vulnerabilities |
| Availability risk | Real exploits can disrupt live services, so timing, monitoring and a stop procedure must be agreed in advance | Lower impact on customers, but the copy must still be isolated from production data and networks |
| Sensitive data exposure | Testers may reach real personal or card data, so access limits, masking and handling rules must be in the rules of engagement | Reduces exposure only if the copy uses masked or synthetic data. Copied live data keeps the same risk |
| Denial-of-service techniques | NIST SP 800-115 suggests these should probably run against a non-production system, so production use needs strong justification | NIST SP 800-115 says these probably belong on a non-production system where the impact can be contained |
| Rules of engagement | Must name production contacts, halt criteria, approved source addresses and escalation paths in writing | Must still state which environment is tested, how it matches production and who approves any difference |
| Off-hours windows | Often scheduled outside peak periods, with the owner on call. NIST SP 800-115 lists off-hours testing as a way to limit impact | Less necessary for impact control, but useful when shared staging shares resources with other teams |
| Identical-environment conditions | Not applicable, since the test runs on the live system | PCI SSC allows an identical separate environment only if the tester confirms the same application and network-layer controls exist and records this in the rules of engagement |
| Retest path | Fixes are verified on the same live systems, with changes scheduled through normal change control | PCI SSC says exploitable issues found in a separate environment must be fixed in production and the test repeated there |
| Regulatory naming | DORA Article 26(2) says threat-led penetration testing runs on live production systems, with scope validated by the competent authority | PCI SSC guidance describes when a separate environment may be used for testing. Confirm which regulator or assessor accepts each approach for your entity |
What production testing finds that staging misses, and the reverse
Production and staging each reveal different weaknesses. Production shows what attackers actually reach, while staging offers room to test disruptive techniques, but only when its configuration and data paths genuinely match live systems. The cards below set out what each environment finds, the condition that makes each finding matter, and the limits to check before choosing.
Configuration drift and live data paths
Staging often lags production in patches, firewall rules, feature flags and integration endpoints. Testing production shows the real routes sensitive records take between services, which matter most when a payment, identity or health data flow crosses several systems. Before relying on staging results, ask the team to compare configuration baselines and confirm which third-party connections are live only in production.
Controls that exist only in live infrastructure
Web application firewalls, load balancer rules, identity provider policies, managed cloud security services and monitoring agents are often configured only in production. A staging test can therefore report a weakness that production already blocks, or miss one that production's controls would catch. This matters when the test is meant to prove that detection and response work, so confirm which controls the rules of engagement must reproduce.
Missed vulnerabilities from environment differences
Staging environments can differ in software versions, database contents, network paths and scale, so a vulnerability that depends on one of those differences may never appear in the test. NIST SP 800-115 warns that such differences can lead to missed vulnerabilities. This is the main reason to run at least some tests against production or an identical copy, and the reason to document every known difference before the test starts.
Safe reach of denial-of-service checks
Techniques likely to cause denial of service are better aimed at a non-production system, according to NIST SP 800-115. This matters when availability is the main concern and a failed test could interrupt customer-facing services. The limit is that a non-production system only answers the question if its capacity, scaling behavior and dependencies resemble production closely enough to show the same failure.
Limits of production testing
Production testing carries the highest availability and data exposure risk, because real exploits run against live systems and sensitive records. It suits cases where the environment cannot be copied, or where the weakness depends on live data or live controls. Ask for a written rollback plan, agreed testing windows, data handling terms and a named contact who can stop the test.
Limits of staging testing
A staging test is only as informative as its similarity to production, and that similarity is hard to prove. Results can understate exposure from live controls, third-party integrations and real data, and can create false confidence if the report does not state the differences. Ask the tester to record every known difference in the rules of engagement, and plan to retest any exploitable finding on production once it is fixed.
What PCI SSC and NIST require before you test a copy
A separate test environment is acceptable only if it matches production. PCI SSC's penetration testing guidance allows a separate environment identical to production to avoid disruption and speed up testing, but the tester must confirm that the same application-layer and network-layer controls exist there. Those controls include the web application firewall rules, authentication and session handling, network segmentation, and the filtering and logging that sit in front of the application. The confirmation belongs in the rules of engagement, so the record shows which environment was tested and what was verified. Ask the tester to list each control they compared, not only to state that the environment is a copy. If a control differs, such as a WAF policy that was never promoted or a load balancer with different headers, the results describe a system you do not run.
Exploitable findings in a test environment do not close the risk on their own. The guidance requires that any vulnerability proven exploitable in the copy be corrected on the production systems, and that testing be repeated on production to verify the fixes. Plan for that second cycle in the engagement scope and budget, because it is a recurring cost of testing a copy, not a formality. Confirm with the provider who performs the retest, how fixes are verified (for example, whether the same exploit path is re-run against production), and how the findings are recorded so an assessor can trace each one from the copy to the production fix.
NIST SP 800-115 asks you to weigh several factors before choosing a non-production target or restricting work to off-hours. Consider the possible impact on production availability, whether testing could expose sensitive personal data, and how closely the test environment matches production. Denial-of-service techniques are generally better suited to a non-production system, yet the same guidance warns that differences between test and production can cause vulnerabilities to be missed. Record your answer to each factor in the rules of engagement. If sensitive personal data is present in the copy, confirm how it was masked or minimized and who can access the test results, since the report itself becomes a sensitive record.
How to limit risk in production and in staging
Testing live systems shows how production behaves under attack, but NIST SP 800-115 is clear that penetration testing with real exploits against production systems and data carries high risk. Limiting that risk starts before the first packet is sent, in the rules of engagement, the change calendar and the operations runbook. The aim is to make every action traceable, reversible and stoppable.
- Scheduling and windows: Agree test windows with the operations team. Where availability impact could be significant, restrict the most disruptive techniques to off-hours, as NIST SP 800-115 suggests.
- Rules of engagement: Write down permitted techniques, source addresses, target lists, named contacts and the exact stop conditions. NIST SP 800-115 includes a rules of engagement template in Appendix B that can serve as the starting point.
- Stop conditions: Define the signals that halt testing at once, such as service degradation, alerts from monitoring or unexpected access to data. Confirm who has authority to call the stop and how they reach the tester.
- Backups and rollback: Verify current backups and a tested rollback path for each target before exploitation begins, so a failed exploit or unintended change can be reversed.
- Monitoring and sensitive data: Have operations watch the targets during the window. Set rules for personal or regulated data the tester might reach, including limits on copying, retaining or exporting it.
Before approving the engagement, confirm that the rules of engagement name a person who can pause testing at any hour of the window.
A non-production copy lets testers run denial-of-service techniques and aggressive exploitation without placing customers at risk. NIST SP 800-115 also warns that differences between test and production environments can lead to missed vulnerabilities, so the copy has to be checked against production rather than assumed to match. The controls below address that gap.
- Environment matching: Compare operating systems, patch levels, configuration, network paths and the application stack with production. NIST SP 800-115 lists how closely the environments match as one of the factors to weigh before choosing a non-production target.
- Confirmed controls: PCI SSC guidance accepts a separate environment identical to production only if the tester confirms that the same application and network-layer controls exist there, and records this in the rules of engagement. Ask for a written comparison, such as a configuration diff, before testing starts.
- Data masking: Load masked or synthetic data so testers do not see sensitive personal data. NIST SP 800-115 names exposure of sensitive personal data as a factor in choosing the test environment.
- Denial-of-service techniques: NIST SP 800-115 says techniques likely to cause denial of service should probably be used against a non-production system. Keep those techniques on the copy and note that load-dependent findings may not reproduce at production scale.
- Known gaps: Document what staging cannot reproduce, such as production integrations, third-party endpoints and live traffic patterns, and plan separate checks for those areas.
Confirm the match in writing and keep the comparison with the test report, because auditors and regulators will ask how the copy relates to production.
Whichever environment the test used, findings only reduce risk once the fixes are verified on the system that matters. PCI SSC guidance says exploitable vulnerabilities found in a separate test environment must be corrected on production systems and the testing repeated to verify the fixes. This step is where a staging test and a production test converge.
- Production fixes: Apply each correction to the production systems, and link the change record to the finding it addresses, so the copy and production do not drift apart.
- Repeat the exploit path: Retest the exploited paths on production using the same rules of engagement and stop conditions, within a window agreed with operations.
- Detection checks: CIS Safeguard 18.4 asks that security measures be validated after each test so that detections cover the techniques the testers used. Confirm the monitoring team saw the activity and adjust alerts where they did not.
- Evidence trail: Keep timestamps, source addresses, test account names and cleanup records, so the team can separate test activity from a genuine incident during the engagement.
- Cleanup: Remove test accounts, planted files and temporary tooling, and confirm removal in writing once testing ends.
Ask the provider to state in the report which findings were verified on production and which were verified only on the copy.
How to combine staging tests with production validation
The sequence reaches a defensible risk position: the copy is tested hard enough to find exploitable weaknesses without touching live service, and the fixes are then proven on the production systems that carry real data and traffic. Each step produces a record that auditors and the client can check.
Agree rules of engagement covering both environments
The client and the testing provider agree a written rules of engagement that names the staging copy, the production targets, the permitted techniques, the testing windows and the people to call if a test causes disruption. NIST SP 800-115 provides a rules of engagement template in Appendix B that can serve as a starting point. The step closes when both parties have signed the document and the client's operations and security contacts have confirmed they can be reached during testing. Confirm any off-hours limits with your own change process, since the source material does not set them.
Document how the test copy differs from production
Before any exploitation starts, the provider and the platform owner list every difference between the staging copy and production, including versions, network paths, security controls, identity configuration, data sets and third-party integrations. Each difference is rated for how much it could hide or create a vulnerability. Where the copy does not match production on an application or network control, the gap is recorded as a known limitation in the rules of engagement. The step closes with a signed difference register that the later production retest will be measured against.
Confirm the copy matches production security controls
The tester checks that the same application-layer and network-layer controls exist in the copy as in production, such as web application firewalls, load balancers, authentication flows and segmentation rules. PCI SSC guidance allows an identical separate environment on this condition, and the confirmation must be recorded in the rules of engagement. If a control is missing from the copy, testers either stop that test path or record the result as unrepresentative. The step closes with a written confirmation from the tester and the platform owner that the copy is fit for the intended tests.
Run the agreed tests against the copy
Testers run the agreed network, application and authenticated tests against the staging copy using the techniques and time windows in the rules of engagement. Techniques that could cause denial of service are kept on the copy, as NIST SP 800-115 suggests, and any test with a possible impact on sensitive personal data is limited to synthetic or masked records. Findings are logged with the affected component, the evidence and whether the issue was exploited. The step closes with a draft findings list that marks which items are exploitable and which are informational.
Fix exploitable findings and retest in the copy
The platform team corrects every exploitable finding in the staging copy, then the tester reruns the specific test that produced each finding to confirm the weakness is closed. Changes are tracked through the normal change process so that the fix on production can later be matched to the same change record. Findings that cannot be fixed are documented with a compensating control and a named owner. The step closes when each exploitable finding is either verified closed in the copy or accepted in writing with its risk recorded.
Apply the fixes to production and repeat the tests
The same fixes are deployed to the production systems through the normal change process, and the tester repeats the tests that found the exploitable issues against production, within the production windows set in the rules of engagement. PCI SSC guidance requires that vulnerabilities found in a separate test environment are corrected on production and that testing is repeated to verify the fixes. Any test that would disrupt live service is limited to the minimum needed to prove the fix. The step closes with a retest result for each production target, marked as verified, not verified or changed.
Validate that detections cover the techniques used
The security operations team checks whether its monitoring generated alerts for the techniques the testers used in both environments, and records any technique that went undetected. CIS Safeguard 18.4 asks for security measures to be validated after each test so that detections cover the techniques testers used. Gaps are logged as detection engineering tasks with an owner and are retested in a later exercise. The step closes with a detection coverage record that maps each technique used to the alert that fired, or to the logged gap.
Record the combined assessment and risk position
The provider writes one report that covers the staging tests, the production retests, the difference register, the detection validation and every accepted risk. NIST SP 800-115 notes that organizations often combine techniques to reach an in-depth assessment at an acceptable level of risk, and this report states whether that level was reached for each in-scope system. The client signs off on residual risk and schedules the next test. The step closes when the report is accepted and its open items are assigned to owners in the client's own tracking system.
What drives cost and effort for each approach
Production and staging tests spend money in different places. Some costs fall on the client's own staff and change windows, while others go to the testing firm for planning, retesting and reporting. The cards below separate one-time setup from recurring work so buyers can compare written quotes on the same basis.
Staging environment build and upkeep
A non-production copy has to be built, loaded with representative data and kept running between tests. The build is usually one-time work for the client's platform team, while patching, data refreshes and access control recur. Ask the tester whether the quote includes setting up the environment or assumes one already exists, and who maintains it between engagements.
Matching effort and configuration drift
PCI SSC expects a separate environment to share the same application and network-layer controls as production, and the tester must confirm this and record it in the rules of engagement. Every production change that is not copied to staging adds verification work before results can be trusted. Ask who checks for drift, how often, and whether that check is billed separately.
Production change windows and coordination
Testing live systems requires agreed windows, change tickets, on-call cover and a way to halt the test quickly. These costs fall mainly on the client's operations and change teams, and they recur for each retest. Ask the quote to state how many coordination sessions and windows are assumed, and who pays if a window is missed.
Rules of engagement and legal review
Production testing needs a written rules of engagement that names permitted techniques, time windows, emergency contacts and data handling. NIST SP 800-115 provides a template in Appendix B that buyers can adapt. Much of this work is one-time per engagement but grows with scope, and legal or procurement review on the client side is often the longest delay. Ask the tester to specify whether drafting, review rounds and sign-off are included.
Retesting cycles after fixes
PCI SSC guidance says exploitable issues found in a separate environment must be fixed on production and the test repeated to verify the fix. Each retest is recurring cost, and it can be larger when fixes must be applied to live systems. Ask whether the quote includes one retest, how many rounds are allowed, and how retest time is priced after the initial report.
Reporting depth and internal staff time
Reports for production tests often need evidence of safe handling, exploitation details and change records for auditors, which raises writing effort. Internal staff also spend time reviewing findings, triaging with system owners and validating fixes. Ask the quote to list report formats, debrief sessions and the client roles expected to attend, since these hours are rarely shown in the headline fee.
Questions buyers ask about production and staging penetration tests
Can we run a penetration test against production safely?
Yes, but the risk is real and has to be managed in the rules of engagement, not assumed away. NIST SP 800-115 treats penetration testing as using real exploits against production systems and data, which it classifies as high risk. The factors that decide how much risk you accept are the likely impact on availability, whether the test could expose sensitive personal data, and how closely a non-production copy matches production. Ask the tester to name the techniques they plan to use, which systems are in scope, the testing windows, and the stop conditions and contacts if a system degrades. Techniques likely to cause denial of service are the ones NIST suggests moving to a non-production system, or restricting to off-hours.
Does DORA require penetration testing in production?
For a narrow group, yes. DORA Article 26 requires threat-led penetration testing at least every 3 years for entities their competent authority identifies, and Article 26(2) states that this testing covers several or all critical or important functions and is performed on live production systems, with the scope validated by the competent authority. Most financial entities are not in that group. For them, Article 25(1) lists a broader set of tests, including vulnerability assessments, scenario-based tests and penetration testing, without requiring that each one run in production. If you are unsure whether your entity is identified, confirm the status with your competent authority and your legal or compliance team before scoping.
Can a staging environment replace production for PCI DSS penetration testing?
Sometimes, under strict conditions. PCI SSC's Penetration Testing Guidance says a separate environment identical to production may be used to avoid disruption and speed up testing, provided the tester confirms that the same application and network-layer controls exist there, and that this is recorded in the rules of engagement. A staging copy that differs in firewall rules, authentication, WAF settings or data handling does not meet that condition. Also, exploitable vulnerabilities found in the test environment must be corrected on production systems, and testing must be repeated there to verify the fixes. Ask your assessor to confirm the approach before you rely on it.
What drives the cost and duration of a production or staging test?
No fixed figure applies, because the cost depends on the written scope. The main drivers are the number of systems and applications in scope, the number of user roles that need to be tested, whether testing runs in production or in a rebuilt copy, and how much work goes into building that copy and keeping it matched to production. Production work also adds coordination time for change windows, monitoring, and staff on call. Costs are one-time for the engagement itself and recur if you retest after fixes or repeat the test each year. Most engagements are fixed-scope, so ask for a quote that lists the systems, testing windows, environment preparation, retest rounds and reporting as separate line items. DeployOpen does not publish a price list, and a quote follows from the written scope.
What does our team need to provide for a production test?
Expect to supply access, decisions and people. Your team needs to confirm the target list and the authorized accounts, approve the rules of engagement, and name a contact who can stop testing and who has authority to take a system offline if needed. Operations or platform engineers should be available during agreed windows, and someone should own backups and rollback steps before testing starts. For a staging-based test, your team also needs to document how the copy differs from production, including controls and data, so the tester can record it. The guidance from NIST SP 800-115 on rules of engagement, including its Appendix B template, is a practical starting point for these agreements.
Can our own team run a production penetration test without outside help?
Your team can run the work, but the independence and evidence requirements of your framework decide whether you should. PCI SSC guidance describes penetration testing as essentially a manual process, and simply running an automated tool does not satisfy the requirement, so the people doing the testing need real exploitation skills. DORA and NIST SP 800-53 also set expectations around independence for testers in some cases. If your team does the work, check whether your assessor or regulator accepts internal testers, and document the rules of engagement and the stop conditions in the same way an external firm would. DeployOpen plans, implements and supports self-hosted platforms and performs security testing, and can be scoped for either a single engagement or retainer-based continuous testing.
How do we keep a production test from breaking things?
Start with a scope that excludes fragile systems and a schedule that avoids peak business activity. NIST SP 800-115 suggests weighing the impact on production before testing, and restricting techniques to off-hours when that impact is high. Agree in writing which techniques are allowed, which are excluded (such as denial-of-service attacks), and how the tester will pause if monitoring shows degradation. Confirm backups and rollback procedures before the first test, and keep the monitoring team informed so that alerts from the test are not mistaken for a live attack. After the engagement, fix any exploitable issues on production and retest the same paths to confirm the fixes hold.
Which sources set the rules for this choice
Three documents shape most decisions about where to test, and they do not all point the same way. *NIST SP 800-115* asks you to weigh the impact on production, whether testing could expose sensitive personal data, and how closely a non-production environment matches production. It suggests considering similarly configured non-production systems or restricting techniques to off-hours, and it warns that differences between test and production can cause missed vulnerabilities. Its rules of engagement template in Appendix B is the natural place to record which environment you chose and why. Read the full document in the NIST SP 800-115 PDF before you draft scope.
If you are in card payments, the PCI SSC Penetration Testing Guidance v1.1 (September 2017) sets a stricter condition. A separate environment can stand in for production only when it is identical in the application and network-layer controls that matter, and the tester must confirm this and record it in the rules of engagement. Any exploitable vulnerability found in the test copy must still be fixed on production and the test repeated there. Ask your assessor how they will verify the match, and whether they will accept a copy that differs in segmentation or monitoring. This guidance is available from the PCI SSC supplement.
For EU financial entities, *DORA* moves the default the other way. Article 25(1) lists penetration testing among the digital operational resilience tests, and Article 26 requires threat-led penetration testing at least every 3 years for entities their competent authority identifies. Under Article 26(2), that threat-led testing covers several or all critical or important functions and runs on live production systems, with scope validated by the competent authority. Operational technology sits in a different position. *NIST SP 800-82 Rev. 3* notes that production OT may need to be taken offline for testing, and lists a replicated, virtualized or simulated system as an example compensating control. Confirm with your regulator and your OT vendor which of these options is acceptable before you book a window. The DORA text on EUR-Lex is the primary source.
How DeployOpen scopes and runs production and staging security testing
DeployOpen plans and runs penetration testing inside your own infrastructure, whether the target is a production system, a staging copy or both. We start from a written scope that records which environment is tested, what safeguards apply and who approves each step, then document and hand over the results so your team can act on them.
How we scope
- Engagements are fixed-scope, with the target environment, techniques and off-hours windows agreed in writing before work starts.
- The scope records whether production or a non-production copy is tested, and why that choice fits your availability and data risk.
- Rules of engagement name the contacts who can halt testing, the data handling rules and the escalation path for critical findings.
How we build
- Before any production test, we confirm with you which denial-of-service or high-load techniques are excluded or moved to a copy.
- If a staging environment is used, we check with you how closely its application and network controls match production and record the gaps.
- Test accounts, backups and rollback steps are prepared with your team so that testing can stop cleanly at any point.
How we hand over
- Reports separate exploitable findings from observations and state the environment each finding was confirmed in.
- Where a fix is validated on staging, the report asks for the same fix on production and a repeat test to confirm it works there.
- We document the test evidence, scope decisions and retest results so your compliance or audit team can review them later.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers for teams that test production on a regular schedule.
- Retests after remediation are scoped in writing, so the production verification step has a defined owner and deliverable.
- Ask us to confirm in your quote which activities are one-time and which recur, so you can plan the budget for each cycle.
