Safeguard 18.4 Validate security measures after testing

What Safeguard 18.4 asks you to validate after a penetration test

Safeguard 18.4 requires you to validate your security measures after each penetration test and to modify rulesets and capabilities where needed so that the techniques the testers used are detected. It is an Implementation Group 3 safeguard only, assigned to the Network asset class and the Protect security function in CIS Controls v8.1. Because it is IG3, it applies to enterprises with security specialists and regulated, sensitive assets that face sophisticated, targeted attacks. It also depends on a penetration testing program under Safeguard 18.1, which sets the scope, frequency and remediation routing that make each test produce something to validate. The CIS Controls Assessment Specification lists 18.1 as a prerequisite for measuring 18.4, and it gives an enterprise a failing score for this safeguard if it has no process for validating security measures after a penetration test.

The risk is easy to state. A penetration test finding that is patched or closed without changing detections leaves the same attack path open to the next attacker. The tester may have moved laterally, staged a payload or exfiltrated test data without triggering any alert, and if the monitoring stack still does not see that behavior, the organization has learned about a gap but not closed its ability to notice it. Validation asks a narrower question than remediation: after the fix, can the defenders see the technique being used? That check belongs to the security operations team and the owners of the detection tooling, not only to the testers.

To act on this, confirm three things with your testing provider and your detection team before the next test. First, ask the provider to log the techniques, timestamps and source hosts used during testing so you can match them against alerts. Second, agree who reviews those alerts and who approves a change to a detection rule. Third, decide what counts as validated for each finding, such as a rule that fires on the technique in a replay or a documented reason why no rule is needed. The CIS Controls Assessment Specification measures the share of security measures requiring modification that have been properly addressed, so keep a record that ties each finding to the detection change or the justification for not making one.

Which detection layers, teams and tests 18.4 reaches

Safeguard 18.4 changes the detection stack only where a penetration test actually touched it, so the scope of each test decides which sensors, log sources and teams do the validation work. The cards below map the layers that usually fall inside that scope and the decisions that widen or narrow it.

Network sensors and firewalls

Perimeter and internal firewalls, and intrusion detection sensors such as Suricata running on OPNsense, record the traffic a tester generates from reconnaissance through exploitation. After each test, check whether those sensors raised alerts, whether any test traffic passed unlogged, and whether rules need changes for the techniques that got through. Confirm which sensor sits on each segment the test crossed, since a sensor that never saw the traffic cannot validate anything.

Host and cloud log sources

Endpoint, server and cloud control plane logs show what a tester did after gaining a foothold, such as new processes, credential use or changed permissions. Map each test objective to the log sources that should have recorded it, and confirm those sources were forwarded to the central pipeline during the test window. Gaps here are often the most useful finding, because they show the team could not have seen the technique at all.

SIEM and alerting pipeline

The correlation and alerting layer decides whether a matching event becomes an alert someone sees. Tools such as Wazuh, which maps alerts to MITRE ATT&CK, or OpenSearch Security Analytics, which runs Sigma-based detectors, can show which rules fired for each tester action. Check the alert delivery path as well, including notification channels and dashboards, because an alert that fires into an unmonitored queue does not count as detection.

Security operations team and tuning

Analysts who triage alerts are responsible for turning test results into rule changes. Record which team members saw the test activity, how long they took to escalate, and which rules they adjusted afterward. Confirm the team has a named owner for ruleset changes and that each change is reviewed before it reaches production, so tuning does not silence the alerts the test was meant to validate.

External and internal penetration tests

Safeguard 18.2 covers external tests and 18.5 covers internal tests, and each produces the technique list that 18.4 validates against. Ask the testing party for a timeline of actions with source addresses, accounts and times so the detection team can match each step to logs. Without that record, the validation cannot show which techniques were caught and which were missed.

Scoping decisions that change the work

Scope determines how much validation is needed. Decide which networks, cloud accounts and log types the tests exercise, whether tests run as clear box or opaque box, and whether internal tests reach segments with sensors. A narrow scope produces a smaller and easier review, but leaves detection gaps outside it unvalidated. Document these choices in the 18.1 program so the 18.4 review knows exactly what it covers.

How an assessor tests 18.4 and what fails

This table walks through the steps an assessor is likely to take when testing Safeguard 18.4, what each step checks, the evidence usually requested and the findings that most often appear. Read it as a preparation checklist: each row shows where an enterprise can show proof before the assessor asks for it.

Test step	What the assessor checks	Evidence requested	Common findings
Confirm the program under 18.1	A penetration testing program exists, since the CIS assessment specification lists 18.1 as a prerequisite for measuring 18.4	Approved program document, defined scope and frequency, named contacts, remediation routing	Testing happens ad hoc with no written program, so 18.4 cannot be measured
Confirm a validation process exists	A documented process for validating security measures after each penetration test	Written procedure, assigned owners, step that links test close-out to detection review	No validation process at all, which the CIS assessment specification scores as a failure
Sample completed tests	A selection of recent tests, checking each one reached the validation step	Test reports, dates of close-out, tickets created from findings, sign-off records	Reports closed without any detection or blocking review recorded
Trace techniques to changes	Each technique the testers used maps to a detection rule, block rule or documented decision not to change	Technique list from the report, matching rule or alert identifiers, change tickets, decision records	Techniques listed in the report with no corresponding rule, alert or justification
Verify changes were retested	Modified rules or controls were checked against the same technique after the change	Retest results, replay logs, alert output showing the technique now fires or is blocked	Changes made to rulesets but never retested, so effectiveness is unproven
Score the Compliance metric	Share of security measures requiring modification that have been properly addressed	Count of measures flagged for change, count closed with evidence, open items with owners and dates	Open modifications with no owner, or closed items with no verification
Check ownership and records	Responsibilities for detection engineering are assigned and decisions are recorded	Role assignments, change history, approvals from the security owner	Testers and detection engineers work in separate queues with no shared record

Sequence to validate security measures after each test

This sequence turns each penetration test into a measured check on detection and blocking. It shows which techniques were caught, which were missed, what changed, and whether the retest closed the gap, so the Compliance metric for Safeguard 18.4 has evidence behind it.

Agree techniques and log sources before testing
Before the test starts, the penetration testing lead, the detection engineering owner and the tester agree which techniques the test may use and which log sources should show them. Those sources usually include host telemetry, network sensor output such as Suricata EVE JSON, and cloud audit logs. The output is a technique-to-detection matrix listing each planned technique, the expected log source, and the alert or block that should fire. This matrix becomes the baseline the rest of the sequence measures against, and it should be approved by the owner of the penetration testing program under Safeguard 18.1.
Collect the tester's timestamped technique log
The tester delivers a log of every technique executed, with start and end times, source and target assets, and the account or tool used. The detection team uses these timestamps to define the review window so alerts from unrelated activity are not counted as detections. The output is a signed-off technique log stored with the test evidence. Confirm with the tester that the log uses the same asset names as your inventory so each entry can be matched to a host or service.
Check each technique against alerts and blocks
For every logged technique, the analyst searches the defensive platforms for the review window and records whether an alert, a block or nothing was produced. Wazuh alerts and their MITRE ATT&CK mapping, OpenSearch Security Analytics findings, and OPNsense Suricata alerts are the usual places to look. Record the rule or detector that fired, not only the fact of a hit. The output is the completed matrix with a result per technique: detected, blocked, partially detected or missed. Ask whether the result was checked by a second person, because a single analyst's judgment is hard to show to an assessor.
Record gaps as tracked changes
Each missed or partially detected technique becomes a tracked change with an owner, a description of the gap, the log source involved and the planned fix. Gaps should be logged in the same system used for vulnerability remediation so they are not treated as informal notes. The output is a change ticket per gap, linked to the matrix row it came from. Where a technique was detected but the alert was too noisy to act on, record that as a tuning item rather than a pass.
Modify rulesets or capabilities
The detection engineer changes the rules, decoders, Sigma detectors or sensor signatures needed to catch each gap, and tests the change in a non-production context where the environment allows. Changes may include a new Wazuh custom rule, a Sigma detector in OpenSearch, a Suricata rule update in OPNsense, or enabling a log source that was off. Each change is peer reviewed and recorded with its version or commit reference. The output is a merged change with its ticket number, and a note on any change that was not made, with the reason and the compensating control. Confirm whether your change process requires approval before rulesets go live.
Retest the same techniques
The tester, or the internal team if the test was internal, reruns the same techniques against the changed environment in the same order, and the analyst repeats the check in step three. The purpose is to confirm the modified detection fires for the intended technique and that it does not depend on the earlier test's conditions. The output is a retest record per technique, linked to the original matrix row and the change ticket. A technique that is still missed goes back to step four rather than being closed.
Close the item with the Compliance metric result
The program owner calculates the Compliance metric defined in the CIS Controls Assessment Specification for Safeguard 18.4: the share of security measures requiring modification that were properly addressed. The numerator and denominator are taken from the matrix and change tickets, so every figure can be traced back to a test record. The output is a closed test record that states the metric result, the open items with reasons, and the date of closure, filed with the penetration testing evidence. Ask your assessor early which evidence format they expect, because the metric is only as credible as the traceability behind it.

Evidence that shows each test was validated

Safeguard 18.4 is only as credible as the paper trail behind it. The cards below cover the artifacts an assessor will ask for, from the procedure that governs validation to the accepted risks that were signed off rather than fixed. Each artifact should be dated, traceable to a specific penetration test, and kept current between audits.

Validation procedure and owner

This is the written process that says when security measures are validated after a penetration test, who runs the validation, and who approves changes to detection rules and capabilities. Name one accountable owner and a backup, and tie the procedure to the penetration testing program under Safeguard 18.1. Check that the document states the trigger for validation, which means the close of each test rather than an annual review. Review it whenever the testing scope or tooling changes.

Penetration test report excerpt

The report excerpt lists each technique the testers used, mapped to the affected asset and the time window of the exercise. Ask the testing provider to structure findings so techniques can be listed one by one, since a narrative summary alone will not let you trace a detection back to an action. Keep the excerpt with the test's scope and rules of engagement so the assessor can confirm the techniques were in scope. Store the signed report according to your retention policy.

Technique-to-detection mapping

This matrix pairs each technique from the test with the detection rule, alert or block that should have fired, and records a pass or fail for each row. Use MITRE ATT&CK technique IDs as the common key so Wazuh alerts, Suricata signatures and OpenSearch Security Analytics findings can be matched against the same list. A fail is a valid entry as long as it leads to a logged remediation. Refresh the matrix after every test and after every rule change.

Change records for rule updates

Each ruleset or capability change needs a ticket that names the technique it addresses, the rule or signature changed, the reviewer, and the deployment date. Keep Sigma rule changes, Wazuh custom rules and Suricata ruleset updates in version control so the diff is visible. Assessors look for the link between a failed row in the mapping and a dated change that closes it. Confirm that rule changes went through your normal change approval process.

Retest results and accepted risks

Retest records show that the detection fired or the block held after the change, with the test case, timestamp and alert or log reference as proof. Where a measure cannot be changed, the accepted risk log records the technique, the business reason, compensating controls, the approver and the review date. Expired acceptances should trigger a fresh decision. Keep both in one register so the Compliance metric for the share of required changes properly addressed can be read directly from it.

Open-source tools that produce detection and validation evidence

OpenSearch Security Analytics, released under the Apache-2.0 license, runs detectors over ingested logs from hosts, network devices and cloud services. Each detector matches events against prepackaged or custom Sigma rules, so a rule written for a technique a tester used can be loaded and checked against the same log types. Matches are stored as findings, which lets a reviewer see which rule fired, on which log type, and at what severity.

Sigma-rule detectors: Detectors use prepackaged or custom Sigma rules, so new rules written after a test can be added without changing the log pipeline.
Findings by rule, log type and severity: Each match is recorded as a finding, giving a dated record that a detection was present after the test.
Alerts to Slack or email: Alerts can be routed to channels such as Slack or email, which provides evidence that the alert reached a responder.
Correlation graph: The correlation engine links findings across log types, which helps show that a multi-step technique the testers used is detectable as a sequence.

Confirm the Sigma rule sources you use and their licenses, and check that your deployed version supports each alert channel you intend to cite as evidence.

What validating security measures costs in money and effort

This table breaks down the cost components of validating security measures after penetration tests, who is paid for each one, whether it is one-time or recurring, and what drives it up or down. Use it to compare quotes on equal terms. The main takeaway is that open-source detection tools remove license spend but not the recurring engineering and review effort that Safeguard 18.4 depends on.

Cost component	One-time or recurring	What drives it	How to reduce it
Detection tooling and licenses	Recurring infrastructure, one-time setup	Log volume, number of sensors and whether components are self-hosted; OpenSearch, Wazuh, Suricata and OPNsense carry no license fee but still need hosting and storage	Confirm the platform is self-hosted in your own environment, size storage to actual log sources, and ask the partner to separate license, infrastructure and support lines
Engineering time to map techniques and write or tune rules	One-time for the first pass, recurring for each test cycle	Number of techniques the testers used, the number of log sources, and how many existing Sigma or Wazuh rules already cover them	Agree the technique list before testing starts, reuse prepackaged rules where they fit, and limit custom rules to gaps the test actually exposed
Retest effort after each penetration test	Recurring, per test	Scope of the retest, number of findings that required a measure change, and whether testers repeat the same techniques	Define retest scope in writing, retest only the techniques that triggered changes, and schedule retests with the next test cycle
Security operations time to review alerts	Recurring	Alert volume from new or broadened rules, analyst staffing, and tuning cycles needed to remove false positives	Tune rules against the test's own telemetry, set severity thresholds, and track alert volume after each change
Evidence upkeep	Recurring	Number of measures changed, how many records must show before and after results, and the retention period set by your own policy	Standardize a change record template, link each rule change to its test finding, and store exports automatically
External review of the validation work	Recurring, per test cycle	Whether an independent party checks that changed measures detect the test techniques, and how many tests are in scope each year	Ask for a defined review scope and sample rather than a full re-review, and align it with the annual testing schedule
Auditor sampling	Recurring, per audit	Number of test cycles in the audit period and the number of measure changes the auditor selects for sampling	Keep one traceable record per change so sampled items can be produced quickly, and confirm the sample expectations with the auditor early

Who owns the validation work, the partner and the auditor

Your organization owns Safeguard 18.4. The obligation to validate security measures after each penetration test sits with you, and so does the decision about what changes to the detection stack are approved. Your security leadership accepts any residual risk that remains after a technique is detected or consciously left open. Your team also owns the penetration testing program under Safeguard 18.1, so the rule changes that follow a test feed back into the same scope and retrospective process.

Control ownership: Name an accountable owner for the validation process and record it in the documented process.
Rule change approval: Review and approve each detection rule, tuning change or new capability before it reaches production.
Residual risk acceptance: Document which tested techniques remain undetected, why, and who accepted that risk in writing.
Tester access and scope: Keep the test scope, timing and contacts under your control so validation can be matched to each test.
Operational follow-up: Confirm that alerts from the new rules reach a team that is staffed to act on them.

Before relying on this tab, check that your process names an owner, a approval path and a written record of accepted residual risk.

Where organizations fall short on validating after testing

Safeguard 18.4 is measured on whether validation actually happens after each test and whether detection changes are addressed. The gaps below tend to appear together: findings get closed, detections stay untested, and nobody owns the follow-up once the tester leaves.

Findings closed without detection checks

Teams often mark a penetration test finding as remediated once the vulnerable service is patched, without confirming that the techniques the testers used would now raise an alert. Under the assessment specification, a process that skips this step fails the Safeguard. The fix is a closure checklist for each finding that asks whether a detector, rule or log source fired during the test, with the matching alert or finding ID recorded in the ticket. Illustrative: a patched web application with no Sigma match for the injected payload is not closed.

Rule changes never retested

A new Sigma rule or Wazuh custom rule is often deployed straight to production and assumed to work. Without a replay of the original test activity, nobody knows whether the rule matches the real log format or fires on the right host. The fix is to keep the tester's commands or scripts, replay them in a staging environment after every ruleset change, and store the before and after results. This gives the Compliance metric in the assessment specification something concrete to measure.

No mapping from techniques to alerts

Testers report the techniques they used, but the detection team rarely writes down which rule or detector should have caught each one. Without that table, gaps stay invisible until the next test. The fix is a living register that lists each technique, its MITRE ATT&CK identifier where the report gives one, the expected alert, the detector or rule name, and the owner. Wazuh alerts already carry ATT&CK mappings, so the register can be checked against them rather than rebuilt by hand.

Tested segment logs not collected

The network segment or host group under test sometimes sends no logs to the analytics platform, or sends only a subset. The tester then succeeds with no record of what was seen. The fix is to confirm, before each test, that hosts, network devices and cloud services in scope forward events to the central platform, and that Suricata EVE output or OPNsense syslog reaches it. Log source coverage should be recorded as evidence alongside the test report.

No owner after the tester leaves

Validation is usually handed to a single engineer or to the testing firm, and it lapses once the report is delivered. Safeguard 18.1 expects a program with defined contacts and remediation routing, and Safeguard 18.4 depends on that program. The fix is a named owner for detection validation, a deadline set in the program document, and a standing agenda item in the retrospective that reviews which findings still lack a validated detection. Confirm the owner's role in the written program, not only in the project plan.

Accepted risks without approval

When a finding cannot be fixed quickly, teams sometimes record it as an accepted risk without a documented decision, and the detection gap stays open. Under the assessment specification, measures that need modification must be properly addressed, so an informal acceptance does not satisfy the metric. The fix is a signed risk acceptance that names the residual detection gap, the compensating control, the approver and a review date, stored with the test evidence. Confirm your organization's approval thresholds before the first test.

Questions buyers ask about validating security measures after testing

How much extra effort does safeguard 18.4 add to each penetration test cycle?

Safeguard 18.4 adds a validation step after each test rather than a separate project, so the added effort is the work of checking each finding against detection and prevention controls and changing what needs changing. The effort rises with the number of techniques the testers used, how many security measures need modification, and how much of the detection stack is managed by people outside the security team. Ask the provider to show, in the written scope, which techniques will be replayed, which systems will be checked, and who signs off each change. The CIS Controls Assessment Specification measures this safeguard through the share of modified measures that have been properly addressed, so a quote should name the measures expected to change.

Can our team validate detections in-house with open-source tools?

Often yes, provided the team already runs a log pipeline and has people who can write and tune detection content. Open-source options such as OpenSearch Security Analytics, Wazuh and Suricata each let you match ingested events against rules, record findings and review alerts, so the validation step is largely a matter of confirming that the techniques used in the test produced an alert. The gaps usually sit in log coverage, rule authorship and the discipline of re-running each technique after a change. A partner can build that workflow inside your environment, but your organization still owns the decision to accept each change and the evidence it produces.

What does the security operations team need to provide for this work?

The security operations team needs access to the detection platform, the logs that the test should have produced, and the authority to change rulesets and alert routing. It also needs to know which alerts are expected to fire during a test window so that testers and analysts do not confuse exercise activity with a live incident. In practice, the team should provide a list of the detection points in scope, a named owner for each ruleset, and a way to record the before and after state of every change. Confirm these needs with your analysts before the test starts, because a missing log source discovered after the report is issued delays the whole cycle.

How long does a retest cycle take after a penetration test?

The length of a retest cycle depends on the scope of the test, the number of techniques and findings that need checking, and how many detection and prevention changes are required. A narrow external test with few findings will move through validation faster than a broad internal test that touches many segments and log sources. Rather than relying on a general estimate, read the timeline and deliverables in the written scope, which should state what is retested, when the retest happens and what counts as complete. Ask your provider to list the dependencies, especially change windows and approvals, that could extend the cycle.

Does safeguard 18.4 apply if we are not in Implementation Group 3?

Safeguard 18.4 is assigned to Implementation Group 3 only, so it is not one of the Safeguards that CIS sets as a baseline for Implementation Group 1 or 2. That does not mean validation after testing has no value for smaller organizations. Any enterprise that runs penetration tests can benefit from checking whether its detections caught the techniques used, and this can be a sensible step even when it is not a requirement of your chosen Implementation Group. If a customer, contract or regulator asks for the CIS Controls at a particular Implementation Group, confirm the required level in writing and check the Safeguard list for that group rather than assuming.

Will the auditor test our detection rules directly?

An independent auditor or assessor decides what to test and how, so you should expect them to ask for evidence rather than to write rules for you. Typically, they will look for a documented process for validating security measures after a penetration test, records of the techniques used, the changes made in response and proof that the changed detections now catch those techniques. The CIS Controls Assessment Specification links this safeguard to the penetration testing program in Safeguard 18.1, so the auditor will also check that program exists. Ask your auditor which evidence they expect before the assessment begins, and keep the records in a form they can sample.

What should a partner include in a fixed-scope agreement for this work?

A fixed-scope agreement should state the systems and detection points in scope, the techniques that will be replayed, the rulesets and capabilities expected to change, and the evidence delivered at the end. It should also name who owns each decision, including approval of changes, and what happens if a detection gap needs a new log source. DeployOpen works on fixed-scope engagements where scope, deliverables and timelines are agreed in writing before work starts, and a quote follows from that written scope. Be clear that the partner prepares systems and evidence, while only an independent auditor or certification body issues reports or certificates.

Which open-source tools produce evidence for validating detections?

Three open-source platforms commonly produce the records this safeguard needs. OpenSearch Security Analytics runs detectors against ingested logs using Sigma rules and records matches as findings with severity, and its correlation engine links findings across log types. Wazuh compares events against a built-in ruleset plus custom rules, writes alerts to files and maps them to MITRE ATT&CK. Suricata's EVE output writes JSON records for alerts and protocol transactions, which can feed a log management tool. Confirm that each platform captures the specific techniques your testers used, because coverage differs by log source and rule set.

How DeployOpen validates detections after penetration tests

DeployOpen helps organizations build the post-test validation loop that Safeguard 18.4 asks for: checking that security measures caught the techniques testers used and tuning detections where they did not. The organization owns its controls and its detection decisions, while DeployOpen plans, implements and documents the self-hosted tooling and evidence inside the client's own infrastructure.

How we scope

Agree in writing which penetration test findings feed the validation cycle and which detection platforms are in scope.
Define which techniques from each test must be checked against logs, alerts and blocking rules before closure.
Confirm the owner of each measure, the approval path for ruleset changes and the evidence the client needs for review.

How we build

Configure log sources for hosts, network devices and cloud services in OpenSearch Security Analytics, Wazuh or both, as scoped.
Load and customize Sigma rules, Wazuh rules and Suricata rulesets so they map to the techniques observed during testing.
Set up Suricata EVE JSON output on OPNsense so alerts and flows are retained and searchable for validation runs.

How we hand over

Document each technique, the rule or detector that should fire, the test result before and after tuning, and the change record.
Provide a validation log that shows the Compliance measure for the share of modified measures that were properly addressed.
Deliver runbooks for re-running validation after future penetration tests, with the client retaining administrative control.

How we support

Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
Ongoing support, upgrades and continuous testing are available as a retainer for teams that retest on a recurring cycle.
Independent auditors and certification bodies, not DeployOpen, assess the evidence and issue any reports or certificates.