Phishing simulation vs security awareness training

Phishing simulation vs security awareness training: which to start with

A phishing simulation tells you how staff behave today, and awareness training changes that behavior, so most programs need both. If you must fund one before the next assessment, fund the training program first when you have no baseline and no way to act on results, and fund the simulation first when training already exists and you need evidence of whether it worked. The deciding question is whether your organization can act on click and report data. If nobody will review results, coach the people who fall for a lure, or change the mail controls that let lures through, a simulation produces numbers without consequences and training produces completion certificates without evidence.

A simulation alone is enough only for a narrow purpose, such as checking whether a newly deployed reporting button reaches the security team or whether a specific business unit reacts differently to a lure type. It is not enough to meet the expectations behind control AT-2 in NIST SP 800-53 Rev. 5, which asks for literacy training that covers the forms of social engineering listed in enhancement AT-2(3), including pretexting, impersonation, baiting and tailgating. A campaign that only uses email lures leaves most of that list untested and untaught. Training alone is enough for a small team that handles little sensitive data, but you then lose the ability to show that behavior changed. Combine the two when you can answer three questions: who reviews the results, what coaching follows a click, and which controls such as reporting workflows or filtering rules will be adjusted from the findings.

Before you commit budget, confirm with your legal and HR teams how exercises will be reviewed, communicated and recorded. NIST SP 800-50 Rev. 1 recommends that legal review planned exercises, that staff know exercises happen at random, and that results guide learning rather than punish anyone. Ask your simulation vendor or internal team to show how click data is attributed to individuals, retained and excluded from performance reviews, and how the NIST Phish Scale can rate campaign difficulty so that results from different lures are comparable. Those answers, more than the choice of tool, determine whether the program is one you can run.

How the two approaches differ on goals, scope, method and evidence

This table compares phishing simulation and security awareness training across the factors an assessor or program owner will ask about. Read each row as a question about what the program is for, and use the NIST references to check what your own program can evidence. The main takeaway is that simulation measures behavior, while training builds the knowledge that behavior depends on.

Factor	Phishing simulation	Security awareness training
Primary goal	Measure whether staff spot and report a deceptive message under realistic conditions (NIST SP 800-50 Rev. 1 describes these as organization-wide exercises)	Build the knowledge and habits staff need to recognize and report social engineering (NIST SP 800-50 Rev. 1 treats this as the learning program)
What staff experience	A single unannounced deceptive email, with no instruction at the moment of the test; results guide later learning	Scheduled sessions, modules and reminders that explain threat types, reporting paths and decision points
Scope and audience	Targeted groups or whole populations, chosen with legal and HR review so exercises are not punitive or singling out individuals	Broad audiences by role, with extra depth for high-risk roles, as the literacy training control expects
Method	Sends a deceptive message and records clicks, data entry and reports; difficulty can be rated with the NIST Phish Scale	Covers forms of social engineering named in AT-2(3) such as phishing, pretexting, impersonation, baiting and tailgating; AT-2(1) adds practical exercises
Metrics	Click-through and report rates, as NIST SP 800-50 Rev. 1 describes; confirm how your tool defines and trends them	Completion, assessment results and knowledge checks; confirm which measures your learning platform actually records
What it proves to auditors	That a exercise ran, how staff responded, and how the program acted on results; it does not prove understanding on its own	That required literacy training was delivered and recorded for the control, including the content covered and who attended
Typical cadence	Recurring, with timing set by the program; confirm with your assessor how often they expect exercises	Ongoing and at onboarding, with refreshers; confirm the frequency your control mapping and policy state
Main risk to the program	Punitive handling or surprise tests that erode trust, reporting, or legal acceptability if HR and legal review is skipped	Training that is delivered and logged but never tested, so the program cannot show staff behavior changed

What each approach finds and misses

Phishing simulations and awareness training answer different questions about staff behavior. Each produces numbers that look like evidence, and each has blind spots that the other can partly cover.

Click and report behavior by group

A simulation shows which staff clicked, which reported, and how those rates shift across departments, roles and difficulty levels. NIST SP 800-50 Rev. 1 describes measuring click-through and report rates, and the NIST Phish Scale can help rate how hard each lure was. Check that the difficulty rating is set before the campaign runs, so that results from easy and hard lures are not compared as if they were equal. Ask how groups are formed so that a low rate in one team is not simply an artifact of a softer lure sent to it.

Reasoning behind a suspicious message

A click or report tells you what someone did, not why. A simulation cannot tell whether a reporter spotted a mismatched sender domain or simply distrusted an unexpected invoice. Training that covers the forms of social engineering listed in AT-2(3), such as pretexting, impersonation and thread-jacking, is designed to build that reasoning. Ask program owners whether they debrief reporters and clickers to find out which cues they used, because that answer separates luck from recognition.

Quiz scores and reporting volume

Training platforms usually report completion rates, quiz scores and the number of messages staff forward to the security team. High completion and high scores show that content was delivered and understood in a test setting. Reporting volume rises when staff are encouraged to report, so it can reflect enthusiasm as much as accuracy. Ask for the share of reports that were real threats, since a flood of harmless reports can overload the team and hide the one that matters.

Behavior under pressure and at speed

Quizzes and reporting metrics measure calm, deliberate choices. They do not show what happens when someone is mid-task, under a deadline, or handling an urgent request from an executive. Simulations capture some of this pressure, but only when the lure resembles real work. Be cautious about any claim that a training score predicts behavior during an actual incident, and ask how the program checks behavior in conditions closer to the job.

Scope gaps: technical controls and live pretexts

Neither a phishing simulation nor standard awareness training tests email filters, link scanning, MFA configuration or endpoint controls. Phone pretexts, in-person tailgating and vishing are also outside typical campaigns unless they are scoped separately. A strong click rate does not show that technical defenses would stop a credential harvest that gets past a user. Confirm which controls are tested by other work, such as a penetration test or a social engineering assessment, before treating these metrics as a full picture.

What NIST SP 800-50 and SP 800-53 AT-2 ask for

NIST SP 800-50 Rev. 1, *Building a Cybersecurity and Privacy Learning Program* (September 2024), describes organization-wide phishing exercises as a way to see whether users spot a deceptive email, measured by click-through and report rates. It is guidance on program design rather than a control with a pass or fail test, so the question for your program is whether your design follows its safeguards. The publication sets out several conditions that shape how exercises should be run.

Legal review: The legal team should review planned phishing exercises before they run, so that monitoring, employee data handling and any deception are approved in advance.
Random timing: Staff should be told that exercises happen at random, which keeps the exercise realistic without giving individuals advance warning.
Non-punitive design: Exercises should not be punitive or single out employees, so results feed learning rather than discipline.
Results guide learning: Click and report data should direct follow-up training and content, not serve as a personnel ranking.
Difficulty rating: The NIST Phish Scale can help rate how hard each simulated message is, so click rates from different campaigns can be compared fairly.

Confirm with your HR and legal teams which of these conditions your works council, employment contracts or local labor rules already require.

Which legal and HR safeguards a phishing program needs

Before the first simulated phish goes out, NIST SP 800-50 Rev. 1 expects the legal team to review the planned exercises. The questions to put to counsel are specific: does the campaign collect personal data beyond whether someone clicked or reported, which jurisdictions do the recipients sit in, and does the lure impersonate a real brand, a regulator, or a colleague in a way that creates exposure. Ask counsel to confirm the retention period for click and report logs, who can access them, and whether pseudonymized identifiers are enough for the reporting you need. Ask HR to confirm whether a works council or employee representative must be consulted before the program starts, and whether the change needs to be recorded in existing monitoring or employee notices. These are questions to confirm with your counsel and HR, not legal advice.

The same guidance says staff should be told that exercises happen at random and that results guide learning, not discipline. Put that commitment in writing, for example in the security awareness policy or the acceptable use standard, and make sure the people who will read the results have agreed to it. Do not name employees in campaign reports, do not use click results in performance reviews, and do not let a manager see which named staff clicked. Under NIST SP 800-53 AT-2(3), literacy training should cover the forms of social engineering staff will meet, so the program needs a documented route from a failed simulation to targeted coaching that HR can defend if someone asks why they were contacted.

Reporting thresholds protect the program from becoming a list of individuals. Report results by department, site, or role only when a group is large enough that no single person can be identified from the figures, and agree that minimum group size with HR and counsel before the first campaign. Confirm who receives identifiable data at all, usually only the program owner for follow-up training, and how long that data is kept. Record these decisions in the program charter so an auditor or works council can see that the safeguards were set before the exercise, not adjusted after an unwelcome result.

How to run simulations and training as one program

The sequence turns simulated phishing from a scorecard into a loop: a non-punitive baseline shows where staff struggle, targeted training addresses those gaps, and repeat exercises and reporting data show whether behavior and response actually improved.

Set the rules and get legal review
Before any simulation, the security lead drafts the exercise rules and sends them to the legal team, as NIST SP 800-50 Rev. 1 recommends for planned phishing exercises. The rules cover who is in scope, which lures are allowed, how results will be stored, and a commitment that results will not be used to discipline individuals. HR confirms that the approach fits existing employment policies and that staff representatives are informed where required. The step is done when the signed-off rules exist in writing and the program owner can explain them to a manager in two sentences.
Run a non-punitive baseline simulation
The security team sends a deceptive email to a defined population at a time staff cannot predict, and records click-through and report rates without linking outcomes to named employees in reporting. Staff are told in general terms that exercises happen at random, so the baseline reflects normal behavior rather than a warned audience. The NIST Phish Scale can help rate each lure's difficulty so later results are comparable. The step closes when the baseline is recorded with lure difficulty noted, and the team has a short list of which lure types most often succeeded.
Map results to social engineering forms
The team sorts each lure by the form of social engineering it used, such as phishing, pretexting, impersonation, baiting, quid pro quo, thread-jacking, social media exploitation or tailgating, using the forms listed in NIST SP 800-53 AT-2(3). Because simulations mostly test email, lures that cover other forms are either omitted or recorded as untested. Illustrative example: if finance staff reported a fake invoice thread well but a caller posing as IT support succeeded, the gap is in voice pretexting, not in email. The step is done when every training topic has a named result or is marked as not measured.
Deliver targeted training to the gaps
Training owners build or select modules for the topics the mapping flagged, and they include the forms that simulations cannot test, such as phone calls, in-person tailgating and social media contact. The content also covers how to report a suspicious message, because NIST SP 800-50 Rev. 1 treats results as guidance for learning, not as a grade. Practical exercises, tabletop walkthroughs or cyber range scenarios can reinforce topics where staff need to act, not only recognize a lure. The step is done when each targeted module has a completion record and the training owner can name the staff groups it reached.
Cover all forms in AT-2 training content
The training catalog is checked against the full AT-2 expectation for literacy training on social engineering and recognition of the forms listed in AT-2(3), since email simulations alone do not evidence coverage. Where the organization is subject to a baseline that selects AT-2(3), the compliance lead keeps a map from each form to a module, a delivery method and a completion record. Practice tests for phone and in-person forms may be scheduled separately under AT-2(1), which the NIST catalog describes as practical exercises but which is not selected in any SP 800-53B baseline, so the organization should confirm whether it applies. The step is done when a reviewer can trace each required form to content and evidence.
Rerun simulations at random intervals
The team repeats simulations on a schedule it does not publish, using lures of comparable difficulty so that changes in click and report rates reflect behavior rather than an easier test. Results are compared against the baseline by lure type and staff group, and small or noisy groups are not reported individually. Illustrative example: if report rates rise for thread-jacking lures after targeted training while click rates stay flat, the program has moved staff toward reporting, which is the behavior incident response needs. The step is done when a comparison against the baseline exists and the owner has recorded which lure types still show weakness.
Feed report volume into incident response
The security operations team receives counts of reported suspicious messages, along with the simulated messages that were reported, so it can tune triage, check whether real lures are being reported, and confirm that the reporting channel works. Incident responders use the data to decide whether a surge of reports reflects a real campaign or a planned exercise, which is why the simulation calendar is shared in a restricted form. Any real-world report that matches a simulation template is handled as a live incident. The step is done when the incident response playbook references the simulation calendar and reporting volumes appear in the monthly security review.

What you receive from each approach

This table compares what a buyer receives from a phishing simulation program and from a security awareness training program, row by row, and marks which items to require in writing in a fixed-scope agreement. Read the middle column for phishing simulation and the last column for awareness training, then check the required column before signing anything.

Deliverable	Phishing simulation	Security awareness training
Campaign plan (required in writing)	Scenarios, send schedule, difficulty level, target groups and stop conditions, agreed before launch	Curriculum outline, modules mapped to the social engineering forms covered, delivery schedule and audience
Legal review record (required in writing)	Written sign-off that the planned exercise has been reviewed by legal before any email is sent, as NIST SP 800-50 Rev. 1 recommends	Usually limited to content licensing and data handling terms, so confirm what the provider will and will not reuse
Targeting list rules (required in writing)	Who may be targeted, exclusions for executives or sensitive roles, how the list is sourced and deleted afterward	Enrollment rules, such as whether all staff or only selected roles complete modules
Click and report reports	Click-through and report rates by campaign, group and template, which NIST SP 800-50 Rev. 1 names as core measures	Usually not produced; confirm whether the platform can show reporting trends alongside phishing data
Difficulty ratings (depends on platform)	Template difficulty scores, which the NIST Phish Scale can help define; ask whether the vendor uses it	Not applicable in most cases
Training content (required in writing)	Short just-in-time lessons shown after a click or report, if the program includes them	Full modules covering the forms of social engineering that NIST SP 800-53 AT-2(3) lists, with ownership and update terms
Completion records (depends on platform)	Rarely a deliverable; confirm whether the platform exports participation data for your records	Per-person completion and assessment results, retained for your audit trail; confirm export format and retention
Remediation plan (required in writing)	Written actions for repeat clickers that are non-punitive and do not single out employees, as NIST SP 800-50 Rev. 1 advises	Follow-up content plan for teams with low report rates, plus a schedule for refreshing modules

What drives cost and effort for each approach

Phishing simulation and security awareness training cost money mainly through people's time, repeated over the life of the program. No price list exists for either approach, and a quote follows from a written scope that names the audiences, rounds, languages and reporting you need.

People paid and their time

Campaign designers, trainers, legal and HR reviewers and internal coordinators each consume hours, whether they are your staff or a provider's. Ask the quote to show who does each task, since a campaign run in-house shifts cost to salaried time while a provider bills for the same work. Confirm who signs off each campaign before work starts, because legal and HR review is easy to omit from an estimate and hard to add later.

One-time build versus recurring work

Setting up the program, including policy, legal review, baseline measurement and the first content library, is largely one-time effort. Running campaigns, refreshing lures, retraining and reporting recur for as long as the program runs. Ask the quote to split these lines, because a low first-year figure with high recurring work often costs more over time than a larger setup with lighter operations.

Headcount and number of audiences

Cost rises with the number of distinct audiences, such as executives, finance staff, help desk, developers and contractors, because each group needs its own scenarios, timing and follow-up. Staff count drives the volume of messages and training completions. Ask how the quote scales when a new business unit or a contractor population is added to the scope.

Content development and languages

Realistic lures and training modules take effort to write, test and retire, and each language multiplies that work because phrasing, local brands and regulatory references differ. Ask whether the quote includes translation and cultural review, and who owns the content library after the engagement ends.

Simulation rounds and difficulty

The number of rounds per year and their difficulty drive design effort and the volume of follow-up. Difficulty ratings from the NIST Phish Scale can help set a consistent level across rounds, but each harder round usually needs more design work and more support for staff who fail it. Ask what is included per round and what counts as a new scenario versus a re-run.

Reporting depth and metrics

Click-through and report rates are cheap to collect, while trend analysis by team, role and difficulty, plus integration with the reporting channel your staff use, takes more build and analyst time. Ask whether the quote includes dashboards, periodic written analysis and evidence packs for an assessor, and how results are kept so they are not used to single out individuals.

Questions buyers ask before choosing phishing simulation or awareness training

Does running phishing simulations alone satisfy NIST SP 800-53 AT-2?

No. A simulation on its own does not cover what AT-2 asks for. AT-2 is a literacy training control, and its enhancements describe the forms of social engineering that training should cover, including phishing, pretexting, impersonation, baiting, quid pro quo, thread-jacking, social media exploitation and tailgating. Simulated phishing measures one of those forms against staff behavior, but it does not teach staff how to recognize the others or how to report them. Confirm with your assessor which AT-2 enhancements your baseline selects, since AT-2(3) is selected in the moderate and high baselines and AT-2(1) is not selected in any SP 800-53B baseline. Ask them how they expect you to evidence the training content, not only the exercise results.

How much internal staff time does a phishing program need?

The effort is mostly internal and recurring, so it is easy to underestimate. Someone must approve exercise plans, coordinate with legal and HR, write or tailor lures, schedule sends, handle help desk questions from staff who clicked, and report results to leadership. Training content needs its own owner to keep it current. Effort rises with the number of audiences you segment, the frequency of exercises and the number of reporting channels you support. Ask any provider or internal team to write down the roles, hours per month and handoffs before you commit, and ask who maintains the content after the first year.

What drives the setup time for a phishing simulation or training program?

Setup time is driven by approvals and integration, not by the sending tool. The main factors are the legal review NIST SP 800-50 Rev. 1 recommends before planned exercises, agreement with HR on how results will be used, integration with your mail system and reporting button, and the choice of who receives which lures. Content localization for multiple languages and accessibility review also add time. Ask for a written plan that separates one-time setup tasks from recurring tasks, and confirm which approvals your organization requires before the first send.

Can our own team run phishing simulations in-house?

Often yes, if you have an owner, approved procedures and a clear way to handle results. Teams that run exercises in-house usually need someone who can design realistic but appropriate lures, a mail configuration that does not trigger external filters in unexpected ways, and a process for managing exceptions. Outsourcing makes more sense when you lack capacity, need independent measurement, or want a provider to supply content and analytics. If you use a service, ask who holds the target list, where the data is stored and how long results are kept. Confirm these points against your own data protection and records policies.

How do we avoid punitive outcomes that damage trust?

Design the program so that a click is a learning event, not a disciplinary record. NIST SP 800-50 Rev. 1 says exercises should not be punitive or single out employees, that staff should be told exercises happen at random, and that results should guide learning. Practical safeguards include agreeing with HR in writing that simulation results will not be used in performance reviews, reporting results by group rather than by name, and giving immediate, short feedback after a click. Ask your provider or internal team how results are stored, who can see individual data and how long it is kept.

How should we choose the difficulty of phishing exercises?

Start with the threats your staff actually face, then increase difficulty in steps. NIST SP 800-50 Rev. 1 notes that the NIST Phish Scale can help rate the difficulty of an exercise, so you can compare campaigns over time. Easier lures measure basic awareness, while spear phishing that uses role-specific context tests whether staff apply their training under pressure. Pick difficulty by audience, since finance, executive assistants and IT administrators face different lures. Ask how the provider or internal team rates difficulty and whether results can be compared across campaigns of different levels.

Should we buy a phishing platform or a managed service?

Buying a platform gives you control and recurring internal effort. A managed service shifts content creation, scheduling and analysis to a provider, and it usually adds ongoing fees. Platform licenses, managed service fees and internal labor are separate cost lines, and each recurs. The deciding factors are your internal capacity, how often you want exercises, whether you need independent reporting, and where data may be processed. Ask each option for a written breakdown of setup fees, recurring fees, included content and what is excluded. Confirm data residency and retention in writing before you share staff names.

When is a social engineering assessment the better next step?

A social engineering assessment is the better next step when you need to know whether an attacker could reach a specific objective, not just whether staff click. A scoped assessment may use phishing, phone pretexts or physical access attempts against a defined goal, and it requires written authorization and legal review before testing begins. Choose it after your awareness program has a baseline, when leadership asks about real access paths, or when you need evidence for controls such as reporting workflows and identity checks. Ask the provider how they define the objective, how they handle out-of-scope staff and how findings feed training or control fixes.

When a social engineering assessment is the next step

Move to a scoped social engineering assessment when the question changes from "do staff click?" to "can an attacker reach a specific objective through people?" A recurring awareness simulation measures click and report rates across a population, using the deceptive-email exercises that NIST SP 800-50 Rev. 1 describes. A social engineering assessment is different in kind: testers combine phishing, phone pretexts or physical access attempts to reach a defined target, such as a payment approval, a privileged account or a secure area. If your team needs evidence that a control failed under pressure rather than a trend in user behavior, the assessment is the right tool. The sibling topic on phishing simulation vs social engineering assessment explains the purpose and metrics of each in more depth.

Before a provider starts, the buyer must supply four things. First, a named owner who can make decisions during the test and stop it if a live incident begins. Second, an approved scope written as a rules of engagement document, listing the people, channels, locations and systems in bounds and those that are off limits, following the structure of the rules of engagement template in Appendix B of NIST SP 800-115. Third, legal authorization: a signed letter that covers the testers, the systems and the third parties involved, such as an outsourced phone line or a hosted email domain. NIST SP 800-50 Rev. 1 also expects the legal team to review planned exercises, and the same review applies with greater force when testers impersonate staff or enter premises. Confirm with counsel and your HR team how staff who are tested will be protected from punitive follow-up.

Fourth, a clear success measure agreed before testing begins. A useful measure names the objective, such as whether a tester can obtain a password reset or move past reception without escort, and defines what a detection looks like. Ask the provider how each outcome will feed a control fix or a training change, and confirm that findings will be reported as process and control gaps rather than as individual failures. Scope, cost and effort will vary with the number of channels, the locations involved and the depth of the objective, so request a quote built from a written scope rather than a headline figure.

How DeployOpen scopes and delivers phishing and awareness programs

DeployOpen plans, implements and documents phishing simulation and security awareness work inside your own infrastructure, so the campaign tooling, reporting data and staff records stay under your control. Engagements are fixed-scope, with the written scope fixing what is measured, what is taught and who approves each exercise before any message is sent.

How we scope

The written scope names which exercise types run, such as click and report measurement, no-notice attempts or scheduled teaching.
Legal and HR review of planned exercises is recorded in the scope before any simulated message reaches staff.
Scope sets the metrics to report, including click-through and report rates, and who receives individual versus aggregate results.

How we build

We configure the phishing platform in your environment, with templates reviewed for difficulty and aligned to the agreed scenarios.
Awareness content maps to the social engineering forms covered by control AT-2(3), such as pretexting, baiting and tailgating.
Exercise tests run against a non-production group first, confirming delivery, tracking and reporting work before wider rollout.

How we hand over

Deliverables include the campaign runbook, template library, metric definitions and the procedure for staff who report suspicious messages.
Reports present results at team or department level and avoid naming individuals, consistent with non-punitive program design.
Documentation shows how exercise evidence supports your AT-2 readiness, with independent assessors deciding what the evidence proves.

How we support

Ongoing support, template refreshes and campaign operations are available as a retainer after the initial handover.
Retainer scope should be confirmed in writing, including how often campaigns run and how results feed refresher training.
Ask any provider, DeployOpen included, to confirm how exercise data is retained, deleted and access-controlled in your environment.

Phishing simulation vs security awareness training