Why continuous penetration testing is still a manual exercise
A continuous penetration testing offering is only as good as the manual testing inside it. A buyer who must show a regulator, customer or insurer that testing is ongoing should treat automated scanning as the cheap, frequent layer and manual testing as the layer that establishes whether an attacker could actually get past a control. A point-in-time annual test leaves the period between releases untested, and that gap grows with every significant change to an application, its configuration or its network path. PCI DSS requirements 11.4.2 and 11.4.3 ask for internal and external penetration tests at least every 12 months and after significant changes, so a change-driven trigger is a compliance expectation, not only good practice. CIS Safeguard 18.2 asks for external penetration tests at least annually and fails the CIS assessment metric if more than twelve months have passed since the last external test, which means a continuous model must still produce a dated, documented test at least that often.
Scanning establishes something useful: that known vulnerabilities, missing patches and some misconfigurations were absent from the scanned targets at the time of the scan. It does not establish that business logic can be abused, that an authorization check fails for one user role, that a chain of low-severity issues leads to access, or that a new feature exposes data in a way no signature describes. The PCI SSC guidance describes penetration testing as a mainly manual process that exploits vulnerabilities to defeat security features, which is the element a scanner cannot supply. The model addresses the risks that appear between releases, such as changes to authentication flows, new API endpoints and infrastructure drift, and the risks that automated tools routinely miss.
Before buying, ask the provider which findings in the last reporting period came from manual work, which test cases were run by a named tester against a changed component, and how a change request triggers a manual test. Ask for the scope of the manual hours per period in writing, and confirm that each periodic test follows a documented methodology that the provider will disclose in its report, as OWASP ASVS 5.0.0 expects of certifying organizations. A subscription that delivers mostly scan output with a manual test once a year does not close the gaps the model is meant to cover.
Four continuous testing models and what each one delivers
Most organizations that want testing between annual engagements choose one of four models. Each mixes automated scanning and manual testing differently, so the same label can mean very different coverage. The questions below help you compare them on what is tested, how often a tester looks by hand, and what you still need to confirm.
Annual test plus quarterly scans
This is the baseline most compliance programs start from. Vulnerability scans run on a schedule while penetration tests run once a year, with a retest after fixes. PCI DSS sets the floor for both: internal and external scans at least every three months, and internal and external penetration tests at least every 12 months and after significant changes. Confirm which systems the scans cover, who reviews scan results by hand, and what happens to a major release that lands between the annual tests.
Retainer with scheduled manual rounds
A fixed-fee or time-committed retainer sets aside tester hours for manual work on a planned cadence, such as named testing rounds across the year. The buyer can direct those hours toward the highest-risk applications or new features. Ask how many manual rounds are included, which targets each round covers, whether unused hours roll over, and how findings from one round are retested in the next. The retainer should name the testers, not only the firm.
Change-triggered testing tied to releases
Here, testing follows the delivery pipeline rather than the calendar. A test is triggered by significant changes such as a new authentication flow, a new external interface, a cloud account move or a major version upgrade. Automated checks run on every release, and manual testers review the changes that alter the attack surface. Agree in writing what counts as a significant change, who makes that call, and how quickly a manual test can start after the trigger fires.
Fully continuous platforms
These offerings combine automated scanning, attack surface monitoring and a dashboard with manual testing sold as an add-on or tier. The automated layer is often the bulk of what the buyer sees day to day. Ask how many hours a named tester spends on your environment each period, whether manual findings are reported separately from tool output, and whether a human validates exploitability before a finding is marked critical. OWASP ASVS notes that running an automated tool without thorough testing is not sufficient for assurance.
How a continuous penetration testing engagement runs, phase by phase
The sequence keeps testing tied to the systems as they change, so each round of findings is validated, fixed and checked again before the next cycle begins. Each phase has a defined output that the client can review.
Agree rules of engagement and approval
The client and DeployOpen agree the targets, accounts, testing windows, exclusions, escalation contacts and goals, then record them in written rules of engagement. NIST SP 800-115 treats this planning phase as the point where rules are identified, management approval is finalized and documented, and no testing takes place. The step closes when the client signs off on the rules of engagement and the baseline scope for the cycle.
Map the baseline and track changes
DeployOpen works with the client's team to record the current architecture, applications, APIs, identity configuration and hosting footprint as a baseline. A change register then captures releases, new integrations, infrastructure updates and new accounts as they happen. The output is an agreed inventory and a list of changes that will trigger testing, which the client confirms before any new work starts.
Run automated discovery and scanning
Automated scanning and discovery tools identify exposed services, known vulnerabilities and configuration weaknesses across the in-scope targets. The scans run on the cadence set in the agreement and are used to direct manual effort, not to stand in for it. The step closes with a triaged list of candidate issues, with false positives removed and each item tied to a scope target.
Perform manual attack testing
Testers examine the candidate issues and the logic of the application, identity and infrastructure paths by hand. They attempt to exploit weaknesses to see whether security controls can be defeated, and they chain lower-severity issues where that shows real impact. PCI SSC describes penetration testing as a mainly manual process of this kind, in contrast to scanning. The step closes when each tested objective has a recorded result, including the paths that were tested and did not succeed.
Report findings with a stated methodology
DeployOpen issues a report that states its scope, methodology, limitations, tools and environment clean-up, followed by each finding with evidence, a severity rating and the reasoning behind it. The PCI SSC report outline is a useful reference for structure. The step closes when the client has accepted the report and the findings are logged in its remediation tracker with owners and priorities.
Retest after remediation
Once the client reports that fixes are in place, testers repeat the relevant attacks to confirm that the new controls address the original risk. PCI SSC guidance says retesting should validate remediation, and PCI DSS requirement 11.4.4 requires exploitable findings to be corrected and the test repeated to verify the correction. The step closes with a retest record that marks each finding as closed, still open or partly addressed.
Re-enter planning for each change request
Each significant change from the change register goes back through planning, where the team confirms whether it alters scope, rules of engagement or the testing window. Small changes may be covered by the existing cycle, while major ones trigger a focused test of the affected components. The output is an updated scope note that the client approves before the next round of discovery begins.
Repeat the cycle and review coverage
The engagement returns to discovery on the agreed cadence, and each cycle builds on the baseline and the previous retest record. At set points, the client and DeployOpen review which targets have received manual testing, which findings recur and whether the cadence still matches the risk. Ask the provider to show this coverage record in each report, since it is the clearest evidence that the manual work is continuing.
What belongs in a continuous scope and the decisions to agree up front
An ongoing engagement stays useful only when its boundaries are fixed in writing and revisited as the estate changes. The cards below cover the decisions that keep recurring testing stable, from the target list to the rules that trigger extra manual work.
Target list and additions
Record every host, application, API, cloud account and identity domain in scope, with an owner for each. Agree how a new asset enters scope, for example by a named change request approved by the client, so that new internet-facing systems are not silently left untested. Ask the provider how the list is kept current between test cycles and who signs off each addition.
Accounts and roles per test
Decide which test accounts, privilege levels and roles the testers will use in each cycle, such as an unprivileged customer account, an administrator and a service identity. Each account needs a documented owner, an expiry date and a clear record of when it was created and removed. Agree whether testers will receive credentials, tokens or a supervised session, and how those secrets are stored and destroyed afterward.
Environments and data
Name the environments in scope, usually production, staging or a dedicated pre-production copy, and state which one a given test cycle uses. Testing against production carries more operational risk, so agree which actions are permitted there and which must be limited to lower environments. Confirm how test data is created, labeled and removed so that real customer records are never touched.
Exclusions and third-party systems
List what is out of bounds: third-party SaaS platforms, shared hosting, partner APIs, denial-of-service behavior and any system the client does not control. Systems owned by a cloud or SaaS provider may be governed by that provider's own testing rules, so confirm those terms before they are written into the scope. Exclusions should be specific enough that a tester can tell whether a finding falls inside the boundary.
Testing windows and change freezes
Set the hours when testing may run, the contacts available during a test, and the change-freeze periods when no deployments or infrastructure work should happen. Coordinate with the operations team so that scanning activity is not mistaken for a live attack, and agree how to halt testing if a service degrades. Recurring cycles should reuse these windows so that the calendar does not need renegotiating each time.
Triggers for extra manual testing
Define in advance which changes will prompt manual testing outside the regular cycle, such as a new authentication method, a payment flow change, a new externally exposed service or a major platform upgrade. PCI DSS requires internal and external penetration testing after significant changes, without fixing a count of tests per year, so the client must decide what counts as significant. Write the trigger list, the response time expected from the provider and the owner who decides whether a change qualifies.
Weakness categories that continuous testing should catch between releases
This table lists weakness categories that tend to appear as an estate changes between releases, and compares what each looks like, why automated scanning alone misses it, and what manual testing checks. Read it as a guide to scoping: the categories on the right-hand side are the reasons a continuous model needs a human tester, not only a scanner. The main takeaway is that a continuous offering should show manual coverage for each category it claims to address.
| Weakness category | What it looks like on a changing estate | Why scanning alone misses it | What manual testing checks |
|---|---|---|---|
| Broken access control | A new endpoint or feature ships without the same authorization checks as its neighbors, so one user can reach another user's records. | Scanners see a valid response and cannot judge whether the caller should have received it without understanding roles and ownership. | Tester compares access across roles and identities and checks object-level and function-level permissions after each change. |
| Authentication and session weaknesses | A new login path, SSO connector or token lifetime change weakens how sessions are issued, renewed or ended. | Scanners flag generic configuration issues but rarely see how a multi-step login or session lifecycle behaves together. | Tester walks each authentication flow end to end, including session handling, logout, re-authentication and recovery paths. |
| Misconfigured cloud or container settings | An infrastructure-as-code change exposes a storage bucket, widens a network rule or runs a workload with excess privileges. | Configuration checks can report known misconfigurations, but a combination of settings may create an exposure that no single rule flags. | Tester reviews how the settings combine in the live environment and confirms what is reachable from which position, within agreed rules. |
| Exposed services after a change | A new service, debug port or administrative interface appears on a public address after a deployment or DNS change. | Discovery scans run on a schedule and may miss a short-lived or newly added service between runs. | Tester validates the external footprint against the approved inventory and checks what each exposed service actually permits. |
| Insecure payment or identity flows | A checkout, refund or account-linking change alters the order of validation steps, so a state can be skipped or replayed. | Automated tools test individual requests and do not model the business sequence that the flow is meant to enforce. | Tester maps the intended flow, then checks whether steps can be reordered, repeated or bypassed, with findings tied to business impact. |
| Vulnerable third-party components | A new library, container base image or plug-in arrives with a known weakness, or an older version stays in use after an upgrade. | Dependency checks identify versions, but they cannot show whether the vulnerable code path is reachable in this application. | Tester judges whether the component is exposed in practice and whether the weakness can be used against this deployment, then records the reasoning. |
| Fixes that do not hold | A remediation for an earlier finding is reverted by a later release or applied only to one of several similar endpoints. | A scan may show a version or rule change but not whether the original risk is still present. | Tester repeats the original test on the fixed component and checks the related paths that share the same code or configuration. |
What drives cost and duration in a continuous engagement
This table compares the main cost drivers in a continuous testing engagement, why each one moves effort, how to keep it proportionate, and what to ask for in a quote. Read it as a checklist for turning a retainer conversation into a written scope, since recurring fees follow from the number of targets, the rate of change and the amount of manual testing agreed up front.
| Driver | Why it moves cost | How to scope it efficiently | What to request in a quote |
|---|---|---|---|
| Number and type of targets | Each external host, application, API or cloud account adds discovery, triage and manual test time. Internal and external targets need different access and tooling. | Group targets by risk and exposure. Test internet-facing and payment or identity flows more often than low-impact internal services. | A named target list with type, owner and test frequency, plus a rule for adding or removing targets during the term. |
| Change volume and significant change frequency | Each significant change triggers a manual retest of affected areas. Frequent releases raise the number of triggered tests in a term. | Define what counts as significant, such as new authentication flows, new external endpoints or architecture changes. Use release notes to trigger work. | The triggers in writing, how many triggered tests are included, and how out-of-cycle requests are priced. |
| Depth of manual work | Manual exploitation of business logic and chained weaknesses takes most of the tester time. Scanning alone is faster but produces a different deliverable. | Put manual hours where business risk is highest and let automated checks cover the remaining baseline. Ask which items are manual. | Hours or effort split between automated and manual work per target, and the named testers who perform the manual work. |
| Number of roles and test accounts | Each role, such as customer, administrator or partner, multiplies the authorization checks a tester must perform. | Limit roles to those that hold different privileges. Agree how accounts are created, rotated and removed. | The number of roles tested per application, who provides accounts, and how credentials are handled and deleted. |
| Testing knowledge level | PCI SSC notes that a black-box assessment may require more time, money and resources to meet PCI DSS deliverables. Grey-box and white-box testing usually use information the client provides. | Choose the knowledge level that matches the goal. Where the aim is coverage rather than realism, share architecture and credentials to reduce discovery effort. | The knowledge level for each test cycle and what documentation the client will provide before testing starts. |
| Retest scope | Retests verify that fixes work. Scope grows with the number of findings and with delays between the original test and the fix. | Set a retest window and retest only closed findings. Batch fixes into planned release cycles. | Whether retests are included, how many, the time window for each, and what the retest report contains. |
| Reporting needs | Reports tailored for an auditor, a regulator or a customer require more mapping work than a technical findings list. | Agree the report outline early and map findings to the controls or requirements that matter to the reader. | The report format, sections, severity method (for example, CVSS with environmental adjustments), and whether a summary for each cycle is included. |
| One-time setup versus recurring retainer | Setup covers scope definition, access, tooling and baseline testing. Recurring work covers scheduled cycles, triggered tests and reporting. | Keep the first baseline complete and then move recurring work to a defined cadence. Revisit scope at each renewal. | Separate line items for setup and recurring work, the term length, what is billed when a trigger fires, and how scope changes are handled. |
Which frameworks expect ongoing testing and what they ask for
Several frameworks set minimum testing rhythms, and they differ on what triggers a new test and how fixes get verified. Read the cards together to see where a continuous model can satisfy an expectation and where a dated, full engagement is still needed. DeployOpen prepares systems, controls and evidence for these assessments, but auditors and certification bodies issue the opinions.
PCI DSS testing cadence
PCI DSS requires internal and external penetration tests at least every 12 months and after significant changes, and internal and external vulnerability scans at least once every three months. Under requirement 11.4.4, exploitable issues must be corrected by risk ranking and the test repeated to verify the fixes. For continuous models, confirm that the manual cycle covers significant changes as well as the annual minimum, and that scan reports are kept separately from penetration test evidence.
CIS Controls penetration testing
CIS Controls v8.1 Safeguards 18.2 and 18.5 ask for external and internal penetration tests at least annually. Safeguard 18.2 fails the assessment metric if more than twelve months pass since the last external test, so a schedule that drifts past that point creates a gap even when scans run often. When you map a continuous program to these safeguards, record the date of each completed manual test and the scope it covered.
OWASP ASVS testing methods
ASVS 5.0.0 expects a verifying organization to disclose the testing methods it used in the report and states that running an automated tool without thorough testing is insufficient. For application-level continuous testing, ask the provider to name which ASVS requirements were tested manually, which were tested with tooling, and which were not tested at all. A dashboard of automated alerts does not by itself satisfy this expectation.
NIST SP 800-115 methodology
NIST SP 800-115 describes planning, discovery, attack and reporting as the phases of a penetration test, and it does not endorse one methodology over another. Use it as the reference for how a continuous program defines rules of engagement, documents management approval and records findings. Ask providers to show how each recurring cycle moves through these phases, including the planning step that precedes any testing.
Who issues the assurance
DeployOpen prepares systems, controls and evidence for compliance assessments, but it does not issue audit reports, attestations or certifications. Independent auditors and certification bodies form the opinion on whether a framework's testing expectation is met. Before you start, ask your assessor which artifacts they accept, such as dated test reports, scope statements and retest records, and agree how a continuous cycle will be presented to them.
Questions buyers ask before adding continuous penetration testing
Does continuous vulnerability scanning satisfy a penetration testing requirement?
No. Scanning and penetration testing answer different questions, so a scanning program alone usually does not meet a requirement written for penetration testing. PCI SSC's guidance describes scans as mostly automated, identifying, ranking and reporting vulnerabilities, while a penetration test is a mainly manual process that tries to exploit weaknesses to defeat security features. PCI DSS asks for internal and external vulnerability scans at least every three months and for penetration tests at least every 12 months and after significant changes, so both activities belong in the same program. Continuous penetration testing sits between them: scanning provides the frequent signal, and periodic manual testing provides evidence that exploitation paths are closed. Confirm with your assessor which evidence they expect before you buy a subscription.
How much effort does our team need to put into continuous testing?
Expect a continuing commitment from your team, although the amount depends on the scope and the test model. Each cycle needs named contacts, current asset inventories, network and account details, agreed testing windows and someone who answers questions during the test. Your team also reviews change notices so the provider knows when a release or infrastructure change should trigger a manual test. Engineers then need time to triage findings, fix issues and tell the tester when a fix is ready for retest. Effort rises with the number of targets, how often they change, how many accounts and roles the tester uses and how quickly your ticketing and change processes move. Ask the provider to list, in the written scope, every input it needs from you and how often it needs it.
How long does each continuous testing cycle take?
The duration depends mainly on how much manual work the scope contains, so plan from the drivers below. PCI SSC's guidance says automated scans identify and rank vulnerabilities in seconds to minutes per host, while penetration tests are mainly manual, exploit weaknesses to defeat security features and may last days or weeks depending on scope. In a continuous program, the scanning component runs on the schedule you set, and the manual component runs on a cycle agreed in advance. The main drivers are the size and complexity of the application or network, the test type, the number of authenticated roles, the amount of retesting and whether a change is small or touches authentication, payment or data flows. Ask for the planned days of manual effort per cycle in the written quote, and ask how that effort changes after a major release.
Can our own security team run continuous penetration testing in-house?
Partly. Internal teams can usefully run scanning, track findings, verify fixes and maintain asset inventories. The manual testing component, however, needs skilled testers who can demonstrate impact, and many organizations find that difficult to staff on a continuous basis. Independent manual testing also carries weight because the tester is separate from the people who built the system, and assessors often look for that separation. PCI SSC's guidance lists indicators of tester skill, including OSCP, GIAC certifications such as GPEN and GWAPT, CREST certifications and CHECK, while stating that certifications alone do not establish experience. Decide which tasks stay internal and which go to an external provider, write that split into your program charter, and confirm with your auditor whether an internal tester meets their independence expectations.
How should change triggers for extra penetration tests be agreed?
Agree them in writing before the first cycle, using a definition of significant change that both your team and the provider accept. PCI DSS requires penetration tests after significant changes, so the definition should reflect your own risk assessment and environment. Typical triggers include new internet-facing services, changes to authentication or session handling, changes to payment or personal data flows, new cloud accounts or network segments, and major platform or framework upgrades. Write down who can raise a trigger, how quickly the provider can schedule a test, which open findings carry into the next cycle and how a trigger that proves minor is closed without dispute. A change calendar linked to your release process makes triggers visible to both sides before they become urgent.
What happens when remediation runs late after a continuous test?
The findings stay open, the retest is delayed and the risk identified in the test remains until someone validates the fix. PCI SSC's guidance says the tester should retest once remediation is complete, and that remediation extending long after the original test may require a new engagement. PCI DSS requirement 11.4.4 requires exploitable vulnerabilities to be corrected according to the entity's risk ranking and penetration testing to be repeated to verify the corrections. In a continuous model, set remediation deadlines by severity in the program charter, track them in the same system as other engineering work and escalate overdue items to a named owner. Ask the provider how it schedules retests, whether retests are included in the fixed scope or billed separately, and at what point it would recommend a fresh test. DeployOpen offers continuous testing as a retainer, so confirm retest terms in the written agreement.
How can I confirm how much manual work a continuous testing provider does?
Ask for the split of hours or effort between automated scanning and manual testing, and require that split in the written quote instead of relying on marketing alone. Strong evidence includes a sample report that shows manual findings with the reasoning behind each one, beyond tool output. OWASP ASVS says certifying organizations should disclose their testing methods in the report and that running an automated tool without thorough testing is insufficient, so ask the provider to describe its methods against a recognized framework. Check tester qualifications, but treat certifications as indicators only, because PCI SSC notes that they do not establish experience. Ask how many findings in the last cycle were found manually, how authorization and business logic were tested and whether the report explains how separate weaknesses combine into an attack path. Compare the answers against the scope you have written.
How to check that a continuous offering still includes real manual work
A continuous offering is only as good as the human testing inside it. The cards below set out the checks to run before signing, from sample cycle reports to written retest terms, so the scanning cadence does not stand in for manual penetration testing.
Count manual findings in sample cycles
Ask for two or three anonymized reports from past cycles and count the findings that came from manual testing, such as broken authorization, chained logic flaws or business-rule abuse. Scanner output usually reads as a list of matched signatures with little tester narrative. If most findings cite a tool rule and none describe a tested attack path, the manual share is too thin. Ask who wrote each finding and when the testing took place.
Name the testers and their experience
Ask which named testers will work on your environment, how long each has done penetration testing, and which platforms or application types they have tested. Continuous programs often rotate staff, so confirm that the people in the proposal are the people who will test your system. Agree in writing how any change in tester is communicated before the next cycle starts.
Check accreditation and certifications
Look for recognized accreditation or individual certifications, such as CREST membership or certified testers, and ask how they map to the work being bought. PCI SSC lists OSCP, GIAC certifications such as GPEN and GWAPT, CREST certifications and CHECK as indications of skill, but it does not validate or endorse them. Certifications alone do not establish experience, so treat them as one input alongside sample work.
Confirm the written methodology
Request the written methodology and read how it separates scanning from manual exploitation and logic testing. NIST does not endorse one methodology, so a provider should name the one it uses and explain how it is adapted. Ask for the statement of limitations, which should say what the test did not cover, such as business logic in areas excluded from the cycle or time-boxed coverage of each release.
Review safe testing and data terms
A continuous model runs against live or near-live systems, so the contract must set safe testing practices, testing windows, rules for denial-of-service avoidance and how evidence containing personal or payment data is stored and deleted. Ask where test data is held, who can access it, how long it is kept and how the provider reports an accidental outage. These terms should be in the agreement, not in a slide deck.
Agree retest terms in writing
Confirm in writing whether retesting of fixed findings is included in each cycle, how many retest rounds apply, and how long after a release a retest must happen. PCI DSS requirement 11.4.4 expects exploitable findings to be corrected and the testing repeated to verify the fix. Flag any offering that describes manual work only as scan review, because a scan rerun cannot confirm that a logic flaw has been closed.
How DeployOpen scopes, tests, reports and retests continuous engagements
DeployOpen runs security testing as fixed-scope engagements agreed in writing before work starts, and ongoing support is available as a retainer. For continuous penetration testing, we define what is covered, combine automated checks with manual testing after material changes, and hand over documented findings and retest results your team can act on.
How we scope
- Written scope names the platforms, environments, accounts and roles covered by the continuous engagement.
- Scope states which changes trigger manual testing, such as major releases, new integrations or infrastructure changes.
- Rules of engagement, testing windows and exclusions are agreed and documented before any testing starts.
How we build
- Automated scanning runs on the agreed cadence and results are triaged by a tester, not forwarded unreviewed.
- Manual testing focuses on business logic, authentication, authorization and configuration of each in-scope target.
- Each testing cycle is planned against the change log so that manual effort follows what actually changed.
How we hand over
- Each cycle produces a report with scope, methodology, stated limitations and tools used.
- Findings include severity reasoning, evidence and remediation advice, with any environment-based score adjustments explained.
- Retests confirm whether fixes address the original risk, and closed findings are documented for your records.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer after the initial written scope.
- Your team can ask for a scope review when platforms, integrations or compliance requirements change.
- Confirm with us during scoping how report formats, evidence needs and auditor requests will be met.
