Vulnerability assessment vs penetration testing

Vulnerability assessment vs penetration testing: which one your review needs

If a framework, customer questionnaire, launch review, insurer or due diligence request asks for "testing," you need to know which kind it means before you buy anything. A vulnerability assessment is an automated scan and ranking of known weaknesses across a set of hosts or applications. A penetration test is a mainly manual exercise in which a tester tries to exploit weaknesses to defeat security controls and show what an attacker could reach. Both are useful, but they answer different questions, so a request for one rarely satisfies a request for the other.

The PCI SSC guidance draws the line clearly. Vulnerability scans identify, rank and report weaknesses using mostly automated tools, and a scan of a single host can finish in seconds to minutes. A penetration test is a mainly manual process that may last days or weeks depending on scope, because a tester has to chain conditions, validate exploitability and judge whether a finding matters in your environment. A scan tells you where the known weaknesses are. A penetration test tells you whether they can be combined into a path to sensitive data or critical systems. For that reason, the two are not interchangeable, and a scan report does not prove that a system resists attack.

Before you commission either test, check the wording of the requirement or customer question and confirm with the assessor or customer which one they mean. Scans are often needed on a regular cadence, while penetration tests are usually scheduled after significant change and at a set interval. Many programs need both: scans to catch new exposure between tests, and a penetration test to validate that the controls work once changes have been made. Confirm these expectations in writing so that your scope, budget and report format match what the reviewer will accept.

How scanning and penetration testing differ in purpose, method, frequency, duration and report

Vulnerability scanning and penetration testing answer different questions, so they differ in every part of the engagement. The cards below compare them on purpose, method, frequency, duration and report, using PCI SSC's guidance as the reference point.

Purpose of each test

A vulnerability scan identifies, ranks and reports known weaknesses so they can be fixed. A penetration test asks a narrower and harder question: can an attacker defeat the security features protecting a system? Scans answer whether a weakness exists. Penetration tests answer whether it can be used to get past controls. Buyers should confirm which question their framework or customer actually asks before choosing.

Method and tester judgment

Scanning is mostly automated: tools probe hosts and match results against vulnerability data. Penetration testing is a mainly manual process in which testers chain findings, adapt to what they observe and decide where to go next. Ask a provider how much of the work is manual, who performs it and how tester judgment shows up in the report. A tool run without thorough manual testing does not meet the bar OWASP ASVS sets for testing.

Frequency and triggers

Scans run on a schedule and can repeat often. PCI DSS asks for internal and external vulnerability scans at least once every three months. Penetration tests run less often: at least every 12 months and after significant changes. Because the two serve different purposes, a program that relies on scans alone will miss the attacker-driven view, and one that tests only annually will leave long gaps between changes.

Duration of each engagement

PCI SSC describes scans as taking seconds to minutes per host, which makes them practical across large inventories. Penetration tests may last days or weeks depending on scope, number of targets and depth of manual work. Duration is therefore a planning input, not a sign of quality. Confirm with the provider how much time is allocated to testing versus reporting and retesting in the written scope.

Report contents and use

A scan report is usually a list of ranked findings with affected hosts and references. A penetration test report is a narrative: executive summary, statement of scope, methodology, limitations, testing narrative, findings with severity reasoning, tools used and retest guidance. Auditors and customers generally expect the narrative form. Check whether a provider's sample report explains how risk ratings were derived and states what was not tested.

How a penetration test runs from planning to retest

The sequence turns a stated goal into a documented, approved test, then into findings the client can fix and prove fixed. Each phase produces an artifact that the next phase depends on, so the report and the retest stay traceable to the original rules of engagement.

Agree rules of engagement and approval
The client and the tester identify the rules, finalize management approval and set testing goals in writing before any test traffic is sent. NIST SP 800-115 treats this planning phase as the one in which no testing occurs. The step closes when a signed rules of engagement document covers targets, testing windows, permitted techniques, contacts for emergencies and the named approver. Appendix B of NIST SP 800-115 offers a template the parties can adapt.
Confirm scope and information provided
The tester and the client fix the targets, accounts and environments, and agree the knowledge level, whether black box, grey box or white box, because that choice shapes how much time the work needs. Exclusions, such as production systems that cannot tolerate load, are recorded here. The step closes with a scope statement that the final report can repeat word for word.
Run automated scanning for discovery
Discovery begins with information gathering and enumeration of hosts, services and versions. Automated vulnerability scanning fits here as one input: scanners identify, rank and report known weaknesses, usually within seconds to minutes per host according to PCI SSC's guidance. The tester uses these results to decide where manual effort should go. The step closes with a candidate list of weaknesses, each marked as confirmed or unconfirmed.
Validate and exploit weaknesses manually
The attack phase is mainly manual. The tester checks which candidate weaknesses are real in this environment, then attempts to defeat security features such as authentication, authorization and segmentation to measure what an attacker could reach. PCI SSC describes this as the step that separates a penetration test from a scan, and it may take days or weeks depending on scope. The step closes with confirmed findings and evidence of impact, recorded in a way that lets the client reproduce the result.
Stop and clean up test artifacts
Once the agreed goals are met or the window closes, the tester stops active testing and removes accounts, files, tools and test data created during the engagement. Anything that could not be removed is listed for the client. The step closes with a clean-up record that the client can check against the rules of engagement.
Write the report with method and limits
The report presents an executive summary, scope, methodology, limitations, the testing narrative and findings with references such as CVE and CWE identifiers. The methodology must be disclosed, because NIST does not endorse one methodology over another and OWASP ASVS requires certifying organizations to state the methods they used. Severity rankings should show how they were derived, typically from CVSS, with reasoning for any environmental adjustment. The step closes when the client has a report that an auditor or customer could follow back to the agreed scope.
Retest fixes and close findings
After the client remediates, the tester repeats the relevant tests to confirm that the new controls address the original risk. PCI DSS requirement 11.4.4 requires exploitable findings to be corrected according to risk ranking and the test to be repeated to verify the corrections. Remediation that drags on well after the original test may call for a new engagement instead. The step closes with a retest report that marks each finding as closed, open or partly mitigated.

What drives the cost and duration of each test

This table compares the main drivers of cost and duration for vulnerability assessments and penetration tests, and shows who is paid for each effort. Read each row for why it moves the price, whether the cost is one-time or recurring, and what to request in a written quote. The main takeaway is that manual effort and scope size, not the tool used, set most of the cost.

Driver	Why it changes cost or duration	One-time or recurring	What to ask for in a quote
Scope size and number of targets	Scans run in seconds to minutes per host, so host count mainly adds tool time and triage. Penetration tests scale with the number of applications, hosts and trust boundaries the tester must work through manually.	Usually one-time per engagement, recurring if repeated on a schedule	A target list with counts by type (hosts, applications, APIs, cloud accounts) and what is excluded
Depth of manual work	PCI SSC describes penetration testing as a mainly manual process that may last days or weeks. Scanning is mostly automated, so manual hours are the largest variable in a penetration test.	One-time per engagement	Estimated tester days split between manual testing, analysis and reporting, with the named roles doing the work
Test type (automated scan, penetration test or both)	A scan is a lower-effort activity that identifies and ranks weaknesses. A penetration test adds validation of exploitability and business impact, which is why PCI DSS treats the two as separate requirements.	Scans often recur (PCI DSS requires internal and external scans at least every three months). Penetration tests are periodic.	Whether the quote covers scanning, penetration testing or both, and how scan results are triaged before delivery
Knowledge level (black, grey or white box)	PCI SSC notes that a black-box assessment may require more time, money and resources to meet PCI DSS deliverables. Grey and white box testing usually give more findings per day because the tester starts with more context.	One-time per engagement	Which knowledge level is priced, what information the client will supply, and the cost difference if the level changes
Number of environments and roles	Each environment (production, staging, cloud tenants) and each user role (standard user, administrator, partner) may need separate setup, accounts and testing time. Multi-tenant or segmented networks add analysis work.	One-time per engagement, with added cost for each additional environment	Environment list, number of test accounts per role, and whether segmentation testing is included
Reporting needs	A report that follows a standard outline, includes CVSS-based severity reasoning, covers segmentation results or is shaped for an auditor or customer takes more writing and review time.	One-time per engagement	Sample report, required sections, audience (technical, executive, auditor), and any evidence format the assessor needs
Retest	PCI SSC advises retesting after remediation to confirm the new controls address the original risk. Each retest re-checks prior findings and may add time if fixes are partial or delayed.	Recurring for each retest round, or a fixed block of retest hours	Number of retest rounds included, retest window, and what happens if remediation takes longer than planned
Timing and testing windows	Out-of-hours windows, change freezes, vendor notice requirements and cloud provider testing rules can extend the calendar time and add coordination effort. Rush delivery usually increases cost.	One-time per engagement	Permitted testing windows, lead time needed, notice to hosting or cloud providers, and any surcharge for compressed timelines

What PCI DSS, CIS Controls and ASVS expect from each test

PCI DSS treats scanning and penetration testing as separate controls with separate rhythms. Internal and external vulnerability scans must run at least once every three months under requirements 11.3.1 and 11.3.2. Internal and external penetration tests must run at least every 12 months and after significant changes under requirements 11.4.2 and 11.4.3. Requirement 11.4.4 then requires exploitable vulnerabilities found in a penetration test to be corrected according to the entity's risk ranking and the test to be repeated to verify the fixes. For most payment environments, scans are the recurring hygiene layer and the penetration test is the periodic proof that exploitable paths were closed.

Scans on a quarterly cycle: Requirements 11.3.1 and 11.3.2 set a minimum of once every three months, so scanning alone cannot satisfy the penetration testing requirement.
Penetration tests on a 12-month ceiling: Requirements 11.4.2 and 11.4.3 require testing at least every 12 months and after significant changes, so a major release or network change can trigger an extra test.
Repeat testing after fixes: Requirement 11.4.4 expects verification testing once corrections are made, which is a retest, not a new scan.
Scope matters to the assessor: Ask the provider to document scope and segmentation testing so the penetration test maps to your cardholder data environment.

Before you commission work, confirm with your qualified security assessor which tests and dates they expect to see as evidence.

What you receive and whether it satisfies an auditor or customer

A vulnerability assessment and a penetration test produce different documents, and each carries different weight with an assessor or customer. The cards below cover what arrives at the end of each type of work and the checks to run before you rely on it. A report is evidence for an assessor; it does not certify compliance.

Scan output and ranked list

A vulnerability scan returns a ranked list of identified weaknesses, host by host, usually with the tool's own severity field and a record of which hosts were reached. Check host coverage first: a list that omits hosts, segments or authenticated checks is incomplete, whatever its length. Ask how false positives were handled, whether each one was verified by a person, and whether suppressed findings are recorded with a reason so an assessor can trace them.

Penetration test report structure

A penetration test report should follow a recognized outline. The PCI SSC guidance suggests an executive summary, statement of scope, statement of methodology, statement of limitations, testing narrative, segmentation results, findings with CVE and CWE references, tools used and environment clean-up. Confirm that every section is present, because a missing limitations statement is a common reason an assessor asks for rework.

Traceable severity reasoning

Each finding needs a severity or risk ranking, and the report should say how it was derived. Many teams use CVSS base scores, then adjust them for the environment, such as exposure, compensating controls or data sensitivity. Adjustments are only defensible when the report records the reason for each one. Ask whether the scoring method is written down and whether a reader can reproduce the rating from the evidence provided.

Retest results showing closed findings

A retest report shows, for each original finding, whether the fix was validated, whether it failed, or whether the risk was accepted. This is the document that answers the question an assessor asks after remediation. Confirm that the retest covers the same targets and that closed findings reference the original report identifiers. Remediation that finishes long after the original test may require a new engagement, so agree the retest window before testing starts.

Summary letter for an auditor or customer

Some customers and assessors want a short attestation-style letter or summary in place of the full report, especially when the report contains sensitive detail. DeployOpen prepares testing evidence and documentation for compliance assessments but does not issue audit reports, attestations or certifications. Ask the auditor or customer which format they accept before the test, and agree in writing who may receive the full report.

Scope of the evidence

Whatever the format, the document is only as strong as its scope and limitations statements. Check that the report names the targets tested, the knowledge level used, the accounts and roles tested, the testing window and any exclusions. A scan that covered only externally reachable hosts cannot support a claim about internal segmentation, and a penetration test without stated limitations overstates its coverage.

How to prepare so the scan or test produces usable results

Before either a scan or a penetration test starts, the client should name the people who make decisions and the people who answer questions during the work. Each engagement needs a technical contact who can confirm asset ownership, a business owner who can approve scope changes, and an escalation path agreed in writing for the moment a critical issue appears mid-test. That path should state who is called, how fast, and whether testing pauses while the client decides. The client also supplies the target inventory: hostnames, IP ranges, cloud accounts, application URLs and the environments they sit in. A scan run without an up-to-date asset list produces results that are hard to act on, because nobody can say who owns a flagged host, whether it is still in service, or whether it is a test system that should be excluded. Reconcile the inventory against your CMDB or cloud tagging before the scan window opens.

For a penetration test, the scope should be agreed in writing before any work begins, covering targets, exclusions, testing windows, change freezes and notice to hosting or cloud providers where their testing rules apply. Check each provider's published testing terms yourself, since they differ, and confirm which testing your cloud or hosting contracts permit. If the test is grey or white box, the client prepares the access: accounts and roles for each user type, such as a standard user and an administrator, plus any documentation the tester will need. Record how credentials are issued, rotated and revoked after the test. Schedule testing windows around patch cycles, releases and business peaks, and place a change freeze on the target systems for the duration so that findings reflect a stable environment. Then confirm with your monitoring team that expected test traffic will be recognized and not treated as a live incident, while still being logged for review.

Clear preparation also shapes what you receive back. Ask the provider to confirm, in the statement of work, how it will identify the systems it tested, which accounts it used, and which findings it could not verify because of access limits. Those details become the statement of limitations and scope that an assessor or customer will read later. Agree up front what happens to the test data and any captured credentials at the end of the engagement, and who in your organization receives draft findings. A scan or test that is set up this way gives the remediation team a list they can act on, and gives the assessor evidence that scope was controlled.

What to check before you buy a vulnerability assessment or penetration test

Buyers compare providers on who does the work, what the report looks like, who performs the testing and how the provider handles your data and retests. Each check below connects to the difference between automated scanning and manual testing, so it helps to confirm the method before comparing prices.

Manual testing or scanner resale

Ask who performs the work and what share is manual. PCI SSC describes vulnerability scans as mostly automated tools that run in seconds to minutes per host, while penetration tests are a mainly manual process that tries to defeat security features and can last days or weeks. A provider that delivers a scan export and calls it a penetration test is selling a different service. Ask for the tools, the manual steps and the hours planned for each.

Sample report quality

Request a redacted sample report before signing and check whether it follows a recognizable outline: executive summary, scope, methodology, limitations, testing narrative and findings. Each finding should show evidence, the severity method and remediation advice. Severity should explain how the rating was derived, typically from CVSS, and any environmental adjustments should be traceable. A report that lists scanner output with no narrative is a warning sign.

Tester experience and certifications

Ask who will test your environment and what they have done in similar systems. PCI SSC lists OSCP, GIAC certifications such as GPEN and GWAPT, CREST certifications and CHECK as indications of tester skill, but it states that it does not validate or endorse them and that certifications alone do not establish experience. Ask for named testers, their relevant work and how the provider assigns staff to your scope.

Accreditation and CREST membership

Accreditation shows that a company has passed an external assessment, but it does not replace a review of the work itself. CREST is a not-for-profit company registered in the UK that accredits member companies and certifies individuals in disciplines including security testing and red teaming. Confirm the accreditation is current, check which disciplines it covers and ask whether the specific service you are buying falls inside that scope.

Safe testing and data handling

Confirm how the provider protects your systems and the data it sees. Ask how testers avoid disruptive actions, how they handle credentials and sensitive records found during testing, where evidence is stored, how long it is kept and how it is deleted. Ask for the provider's rules of engagement and confirm that a testing window and emergency contacts are agreed in writing before work starts.

Retest terms in writing

Ask whether retesting is included, how many retests are allowed and within what period, and whether retests are documented in a separate report. PCI DSS requires exploitable vulnerabilities to be corrected and the penetration test to be repeated to verify the corrections, and PCI SSC notes that remediation completed long after the original test may require a new engagement. Get these terms in the written scope so they are not negotiated after fixes are made.

Questions buyers ask before commissioning a scan or penetration test

Can a vulnerability scan replace a penetration test?

No, not where a framework requires a penetration test. PCI SSC's Penetration Testing Guidance describes vulnerability scans as mostly automated identification and ranking of weaknesses, and penetration tests as a mainly manual process that exploits vulnerabilities to defeat security features. PCI DSS requires both, with internal and external vulnerability scans at least once every three months and penetration tests at least every 12 months and after significant changes. A scan tells you where weaknesses may exist. A penetration test tells you which ones an attacker could actually chain into a meaningful impact.

How long does a vulnerability scan take compared with a penetration test?

PCI SSC's guidance puts a vulnerability scan at seconds to minutes per host, while a penetration test may last days or weeks depending on scope. The gap comes from manual work: testers pursue leads, validate exploitability and check whether controls hold across connected systems. Total calendar time also depends on scheduling, testing windows, client review of findings and retesting, so ask your provider for a timeline built from your written scope.

How often does each one need to run?

PCI DSS sets the cadence most buyers follow: internal and external vulnerability scans at least once every three months, and internal and external penetration tests at least every 12 months and after significant changes. CIS Controls v8.1 Safeguards 18.2 and 18.5 also ask for external and internal penetration tests at least annually, and Safeguard 18.2 fails the assessment metric if more than twelve months pass since the last external test. Scanning can run more often than testing, and major releases, new internet-facing services or architecture changes should trigger an extra test regardless of the calendar.

What does the client's team need to provide?

Both engagements need a named technical contact, agreed targets and accounts, and a testing window that operations can support. A scan needs asset lists, credentials if authenticated scanning is wanted, and rules for avoiding production disruption. A penetration test needs more: accounts and roles for each user type, architecture and data-flow information, agreed exclusions, rules of engagement and someone available to answer questions during testing. The more the client shares, the more of the effort goes to finding issues rather than rediscovering the environment.

Can our own team run the scans in-house?

Yes, scanners can be run in-house, and many teams do this on the quarterly cadence PCI DSS sets. The limit is interpretation. Manual testing, chaining weaknesses, judging business impact and deciding how severity should be adjusted for your environment need experienced testers. OWASP ASVS states that running an automated tool without thorough testing is insufficient for assessment purposes. A common model is in-house scanning between formal tests, with an independent manual test for the annual or change-triggered requirement.

What drives the price of each test?

The cost of a scan mostly follows the number of hosts, the scanning frequency and whether the scanner is licensed or open source. A penetration test is priced on the scope, the depth of manual work, the knowledge level (black, grey or white box), the number of retests and the reporting required. Ask for a quote that separates the one-time work (scoping, testing, report) from recurring items (scans, retests, retainers). See the penetration testing cost guide on this site for the full cost table, and note that DeployOpen does not publish a price list. A quote follows from the written scope.

What does the report look like for an auditor?

A scan report lists identified vulnerabilities, their ranking and affected hosts, which is useful evidence of a recurring scanning process. A penetration test report is written for a wider audience and, following PCI SSC's suggested outline, typically includes an executive summary, statement of scope, statement of methodology, statement of limitations, testing narrative, segmentation results, findings referencing CVE and CWE where relevant, tools used and environment clean-up. It should explain how severity was derived, usually from CVSS, and show reasoning for any environmental adjustments. Confirm with your auditor which format they accept before the test starts. DeployOpen prepares evidence for assessments but does not issue audit reports or attestations.

How DeployOpen works on this

DeployOpen runs security testing as part of its work on self-hosted open-source platforms inside the client's own infrastructure. Each engagement starts with a written scope that states whether the work is a vulnerability assessment, a manual penetration test, or both, and that scope governs what is built, tested, documented and handed over.

How we scope

Most engagements are fixed-scope, with targets, environments, exclusions and deliverables agreed in writing before testing starts.
We confirm whether the client needs automated scanning, manual penetration testing or both, and which purpose each serves.
Ongoing support, upgrades and continuous testing can be added as a retainer once the initial written scope is in place.

How we build

Scanning covers the agreed hosts, applications and platform components, with results reviewed by a person before they are reported.
Manual testing targets exploitable weaknesses in the agreed scope, with the testing method documented alongside the findings.
Testing rules, testing windows and any limits on destructive activity are recorded in the rules of engagement before work begins.

How we hand over

The report states scope, methodology, limitations and tools, so an auditor or customer can see what was and was not tested.
Each finding carries its severity reasoning, evidence and remediation advice, with any environment-specific score adjustments explained.
Retesting of corrected findings is set out in the written scope, so the client knows what verification is included.

How we support

Remediation questions from the client's engineering team are answered against the findings in the delivered report.
Retests confirm whether fixes address the original risk, and the retest report records which findings are closed.
Where the client needs repeat testing after releases or changes, a retainer can set the cadence the client's own risk requires.