CIP-009 backup and recovery testing | what it takes

What does CIP-009 backup and recovery testing cost and require?

When a Regional Entity audit or a customer contract asks for CIP-009 evidence, the registered entity owns the backup and recovery test program, and the cost comes from the scope of systems covered and the effort of running the tests, not from a fixed fee. The entity pays for its own staff time, the storage and infrastructure used for backups and test recoveries, and any outside help it chooses to buy. Part 1.3 requires documented processes for backing up and storing the information needed to recover BES Cyber System functionality, and Part 1.4 requires processes to verify that backups completed successfully and to address failures. Those two parts recur every day, so their cost is mostly ongoing storage, monitoring and staff effort.

Parts 2.1 and 2.2 set the test cadence. Under Part 2.1, each recovery plan must be tested at least once every 15 calendar months through an actual recovery, a paper drill or tabletop exercise, or an operational exercise. Under Part 2.2, a representative sample of the recovery information must be tested at least once every 15 calendar months to confirm it is usable and compatible with current configurations. Part 2.3 adds, for high impact BES Cyber Systems, an operational exercise of recovery in a production-representative environment at least once every 36 calendar months, and an actual recovery can substitute. An entity with many high impact systems will spend far more on that exercise than one with a few low impact assets, so the impact rating and the number of BES Cyber Systems in scope are the main reasons no single price applies. Ask any provider for a written scope that lists the systems, the test types and the environments before accepting a quote, and confirm current requirement language directly in the NERC CIP-009-6 standard. This section gives no price or duration beyond the supplied facts, so the cycle lengths above are the only timing figures to plan around.

Who is paid for what in a CIP-009 backup and test program

This table compares the main cost components of a CIP-009 backup and recovery testing program by who is paid, whether the cost is one-time or recurring, and what drives it up. Use it to separate internal effort from outside work before you request a written scope or quote. No fixed figures are given because none are supplied for this topic.

Cost component	Who is paid	One-time or recurring, and what drives it
Internal staff time for recovery plan tests	Registered entity's own staff (salary and time allocation)	Recurring. Each 15-month plan test and data sample test consumes operations, OT and compliance time; driven by the number of BES Cyber Systems and recovery plans in scope.
Backup tooling and storage	Software licensing or support provider, cloud or data center storage provider	Recurring. Storage volume, retention periods, licensing model and whether backups are stored off-site; driven by how long backup and catalog records must be kept.
Implementation partner work to configure backups and write test procedures	Implementation partner under a fixed-scope written agreement	One-time for setup and procedure writing; recurring only if a retainer is added. Driven by the number of systems, backup jobs, recovery plans and environments in the written scope.
Operational exercise staging in a production-representative environment	Registered entity for its own test or lab hardware, cloud accounts or vendor services; partner if included in scope	Recurring at each 36-month high impact exercise and when environments change. Driven by how closely the staging environment must match production and whether it must be built or rented.
Independent audit fees for CIP-009 monitoring	Compliance Enforcement Authority or its contracted auditors (not paid through an implementation partner)	Recurring on the audit or monitoring cycle set by the Regional Entity. Driven by the entity's registration and the scope the auditor selects; ask the Regional Entity for its schedule and evidence request.

How backup, verification and recovery tests fit together

CIP-009 splits the work into two rhythms, and an auditor will expect you to keep them separate. Backup under Part 1.3 and verification under Part 1.4 run continuously, every time a job executes. Plan tests under Part 2.1, data sample tests under Part 2.2 and high impact operational exercises under Part 2.3 are scheduled events with their own records. A backup job that completes every night does not, by itself, show that a recovery plan works. The scheduled tests are where you prove that the stored information can actually bring a BES Cyber System back.

The continuous side is about trust in the copies. Part 1.4 asks for processes that confirm backups completed successfully and that address failures, so the record that matters is the job outcome, the failure alert and the follow-up action. Tools such as Bareos keep job status and catalog entries, and retention settings determine how long those records and volumes survive. Confirm with your own tooling how job logs are exported and kept for the audit period. The scheduled side is about usability. Part 2.2 requires a representative sample of recovery information to be usable and compatible with current configurations, which means a backup taken before a firmware or software change may pass verification yet fail the test. Restic's check command with --read-data-subset can read pack files back to confirm they are unmodified, but that confirms integrity, not compatibility, so the sample test still needs a human-verified restore.

For high impact BES Cyber Systems, the 36-month operational exercise under Part 2.3 is the heaviest requirement, and the standard allows an actual recovery to substitute for it. Decide early whether you will plan for scheduled exercises or wait for a real recovery, because the second option depends on an event you do not control. Whatever the choice, the evidence should name the plan version, the systems recovered, the restore points used and the gaps found, and it should link to the corrective actions that updated the plan. Ask your backup vendor or integrator to show how the tool records each restore so that a single report can show the whole chain from backup to recovered system.

How a CIP-009 test cycle runs from first backup to audit evidence

This sequence turns CIP-009 from a set of backup jobs into a traceable record that shows what was backed up, that the backups worked, that recovery plans were tested on the required cycles, and that failures were fixed. Each phase produces a document or record that the next phase depends on.

Scope the BES Cyber Systems and their impact ratings
Start from the CIP-002 categorization. The compliance lead confirms which BES Cyber Systems are in scope for CIP-009 and records each system's impact rating, because high impact systems carry the additional 36-month operational exercise in Part 2.3. The output is a scoped inventory that lists each system, its rating and the recovery plan that covers it. The step is done when the CIP-002 owner signs off that the inventory matches the current categorization.
Define backup and verification processes
The asset owner and operations team write the processes required by Parts 1.3 and 1.4: what information is backed up to recover each system, where copies are stored, how backup completion is verified, and how failures are detected and handled. Include the backup tool's configuration, such as job schedules, retention settings and the storage location. The output is a written backup and verification procedure approved by the CIP Senior Manager or delegate. It is complete when every item needed for recovery appears in the procedure with a named owner.
Run backups and verify completion
Operators run scheduled backups and check each job's result against the verification step in the procedure. Backup tools such as Bareos keep job records in a catalog, and restic can run a check that confirms repository structure and, when configured, reads back pack files to confirm stored data is unmodified. Any job that fails is logged and assigned for follow-up. The output is a dated backup log showing successful jobs, verification results and failure tickets. The step is done when the log covers the full review period without unexplained gaps.
Schedule the 15-month plan and sample tests
Plan the Part 2.1 recovery plan tests and the Part 2.2 sample tests so that each recovery plan and each representative sample of recovery information is tested at least once every 15 calendar months. The test method can be an actual recovery, a paper drill, tabletop exercise or operational exercise. The sample test confirms that backed-up information is usable and compatible with current configurations. The output is a test calendar with a named test lead and a defined scope for each test. It is done when every in-scope plan and sample has a scheduled date within the cycle.
Run the 15-month tests and record outcomes
Execute each scheduled test and capture what was done, who took part, which backup sets were restored, the time the recovery took and whether the restored system worked with current configurations. Record the test as pass or fail against the plan's success criteria. The output is a signed test record for each plan and sample, with screenshots, tool logs or restore reports attached. A test is complete only when its record includes the result and any deviations from the plan.
Run the 36-month operational exercise for high impact systems
For high impact BES Cyber Systems, schedule an operational exercise of recovery at least once every 36 calendar months, carried out in an environment representative of production. An actual recovery can substitute. The exercise uses the same recovery plan and backup set that operations would use in a real event. The output is an exercise report that states the environment used, the systems recovered, the personnel involved and the gaps found. Confirm with the Regional Entity how it expects the representative environment to be documented.
Remediate failures and update the recovery plan
Each failed backup, failed verification, failed sample or unsuccessful exercise produces an action item with an owner and a closure date. Fixes can include changing backup scope, correcting configuration drift in the recovery procedure or updating the recovery plan itself. Where a lesson changes the plan, the plan revision and approval date are recorded. The output is a closed action register and a revised recovery plan where needed. The step is done when each action is closed or has an approved plan with a documented reason.
Assemble the record set for the audit file
Bring the outputs of every earlier phase into one audit file organized by requirement part: the scoped inventory, the approved procedures, the backup logs, the test calendar, the test and exercise records, and the action register. Check that each record shows a date, an owner and the system it covers, and that retention settings in the backup tool keep the needed records available for the audit period. The output is an indexed evidence package that a Regional Entity auditor can trace from requirement to record. Ask your auditor which retention and format it expects before you start.

How Bareos and restic record verifiable backup results

Bareos uses a central Director to schedule and supervise backup, restore, verify and archive jobs across the systems it protects. The Director stores what it backed up in an SQL catalog, which records the files in each job and the volumes that hold them. That catalog is the main place to look when an assessor asks which backup covers a given BES Cyber System, when it ran and whether it completed. Restores can be selected from the current backup or from the backup in place before a given date and time, which helps when a recovery test needs a known earlier configuration.

Scheduled job records: The Director logs each backup, restore, verify and archive job, so you can show the job name, run time and outcome for the backup cycle under Part 1.4.
Catalog of backed-up files: The SQL catalog ties files to jobs and volumes, which supports showing what the backup of a recovery-relevant system contains.
Point-in-time restores: Restores from the current backup or from one before a date and time support representative sample tests under Part 2.2 and operational exercises for high impact systems.
Retention settings: Job, file and volume retention periods control how long records stay in the catalog and how long a volume is guaranteed not to be reused. Set these to cover your audit period and the 15 and 36 month test intervals.
Verify jobs: Verify jobs compare stored data with the source, which you can cite as a backup verification record.

Confirm with the Bareos documentation and your own environment which features and editions your deployment includes, and check the license terms, since Bareos is licensed under AGPL-3.0-only.

What makes CIP-009 testing more expensive or slower

Most of the effort in CIP-009 testing comes from how many systems need exercises, how much recovery information must be kept current, and how many retests failed backups create. The cards below cover the drivers that move cost and schedule, and the controls that keep them in check.

High impact systems in scope

Each high impact BES Cyber System needs an operational exercise of its recovery plan at least once every 36 calendar months, in an environment representative of production. Every additional high impact system adds its own exercise, its own test environment and its own evidence package. Before scheduling, confirm with your CIP-002 categorization which systems are high impact, and group systems that share a recovery plan so one exercise can cover them where the plan allows. An actual recovery can substitute for the exercise, so note any real recovery events in your records.

Size of recovery information

Part 1.3 requires backups of the information needed to recover BES Cyber System functionality, and every item in that set must be restorable and current. Large sets of configurations, images, keys and vendor media multiply the time needed to verify each recovery plan. Keep an inventory of recovery information per system, remove what the plan does not need, and record where each item is stored. A smaller, well-indexed set makes the 15-month exercises faster and the evidence easier to produce.

Production-representative test environment

The 36-month operational exercise must run in an environment representative of production, which usually means duplicated hardware, network segments, or virtualized equivalents that cost money to build and maintain. Ask early whether a lab, a virtualized replica or a scheduled window on production hardware is acceptable for your systems, and document how representativeness was judged. Keeping the test environment configuration under the same change control as production avoids a second baseline to maintain.

Retention periods extending storage

Bareos keeps job, file and volume records in its catalog for configured retention periods, and a volume is guaranteed not to be reused until its retention has passed. Longer retention means more volumes, more catalog growth and more storage to pay for. Set retention to cover the audit period and the evidence you must show for the 15-month cycle, not longer by default. Confirm retention values with your compliance lead and your backup vendor, because they determine both cost and what the auditor can see.

Sample size for Part 2.2

Part 2.2 requires testing a representative sample of the recovery information every 15 calendar months to confirm it is usable and compatible with current configurations. The effort depends on how the sample is chosen, how many items it includes and how many systems it spans. Define the sampling method in the recovery plan, choose items that change most often, such as configurations and credentials, and rotate the selection across cycles. Document why the sample is representative so the auditor does not need to ask.

Failed backups that force retests

Part 1.4 requires processes to verify that backups complete and to address failures. A failed backup can leave a gap that must be found, fixed and retested before the next scheduled exercise can give clean evidence. Schedule automated verification after each job, alert on failures, and keep a log of each failure with its cause and fix. Restic's check command with read-data options can confirm that stored pack files are unmodified, which helps separate storage faults from configuration faults during the retest.

Who owns CIP-009 controls, who prepares the evidence, and who audits it

The registered entity owns CIP-009 in full. Its compliance and operations team writes and maintains the recovery plans, decides which backups are needed to restore BES Cyber System functionality, sets the scope of each 15-month test and each 36-month operational exercise for high impact systems, and approves the results. Only the entity can accept a failed test, record a backup failure as resolved, or decide that a recovery plan needs revision after a lesson learned.

Plan ownership: Named owners keep recovery plans current and approve changes after tests, incidents or system changes.
Test scope decisions: The team chooses whether each 15-month test is a recovery from an actual incident, a paper drill, a tabletop exercise or an operational exercise, and documents why.
Result approval: Someone accountable reviews test outcomes, signs the results and tracks corrective actions to closure.
Backup failure handling: Operations staff act on verification failures and keep the record of how each was addressed.
Evidence retention: The entity decides where signed results live and how long they are kept.

Confirm with your compliance lead that every recovery plan has a named owner and that test approvals are signed by a person, not only logged by a tool.

Where CIP-009 programs usually stall

CIP-009 programs rarely fail on the existence of a recovery plan. They stall where recovery information drifts from the live systems, where backup success is assumed rather than tested, where exercises leave no record, and where retention quietly removes the proof an auditor will ask for. Each card below names one of these points and a step to avoid it.

Recovery information drifts from configurations

The sample of recovery information required by Part 2.2 is only useful if it still matches what runs today. Configuration files, images, license keys and vendor documentation change after a patch, a firmware update or a replaced device, and a sample pulled from last year's baseline can restore a system that no longer exists. Check each tested sample against the current baseline configuration, and record the date and the configuration version it was compared with. Then the test shows usability and compatibility, not just that a file opened.

Backups reported as successful but never verified

A green job status in the backup scheduler shows that a job finished, not that the data can be restored. Part 1.4 expects processes that verify successful completion and address failures, so the check has to read the data back. For Bareos, schedule verify jobs against the catalog and review their results. For restic repositories, run restic check with --read-data-subset on a rotating schedule so pack files are read and confirmed unmodified. Keep the output of each run.

Operational exercises run without records

High impact BES Cyber Systems need an operational exercise of recovery in a representative environment at least every 36 calendar months, unless an actual recovery substitutes. Teams often perform the work but keep no scope statement, participant list, timestamps, or recorded deviations from the plan. Before the exercise, write the objectives and the environment used. During it, log each recovery step with its time. Afterward, record every gap found and the plan update it triggered. Without that trail, the exercise cannot be shown to have happened.

Retention settings that delete evidence early

Bareos job, file and volume retention periods decide how long catalog records and backup volumes survive. A short file retention or an aggressive volume recycling schedule can remove the records of earlier tests before the audit period is covered, and the tests are then impossible to prove. Set retention for catalog and job records to cover at least the testing cycle plus the audit period, and confirm with your Compliance Enforcement Authority how far back evidence will be requested. Document the chosen values and review them after each retention change.

CIP-009 backup and recovery testing questions

How much effort does a 15 month recovery plan test take?

The effort depends on how many recovery plans the entity has, how many BES Cyber Systems each plan covers, and whether the test is a paper drill, a tabletop exercise, an operational exercise or an actual recovery. CIP-009-6 Part 2.1 requires each recovery plan to be tested at least once every 15 calendar months, and Part 2.2 adds a separate test of a representative sample of recovery information at the same interval. A tabletop exercise mostly costs staff time for planning, the session and the write-up. An operational exercise also consumes engineering time in a non-production environment, plus the cost of that environment. Ask any partner for a written estimate that lists each recovery plan, the test method chosen, the environment needed and who performs each step.

Can an actual recovery replace the 36 month operational exercise?

Yes, for high impact BES Cyber Systems. CIP-009-6 Part 2.3 requires an operational exercise of recovery in an environment representative of production at least once every 36 calendar months, and it states that an actual recovery can substitute for that exercise. The recovery must still be documented well enough to show which plan was used, what was restored and whether it worked, so an unplanned outage is not a substitute unless the record supports it. Entities that rely on an actual recovery should confirm with their Compliance Enforcement Authority how they will evidence it before an incident occurs.

What parts of CIP-009 can an entity do in-house?

Most of the work can stay in-house if the entity has staff who know its BES Cyber Systems and backup tooling. Typical in-house tasks include writing and maintaining recovery plans, scheduling backups, reviewing verification results, running tabletop exercises and keeping test records. Outside help is usually most useful for designing a repeatable backup and verification pipeline, building a non-production recovery environment that reflects production, documenting evidence in a form an auditor can follow, and running a first operational exercise. DeployOpen plans, implements and documents self-hosted platforms inside the client's own infrastructure and prepares evidence, but the entity keeps ownership of its controls and any compliance determination is made by the auditor.

What evidence does a Regional Entity ask for under CIP-009?

Auditors generally ask for evidence that each requirement is in place and was performed on schedule. For CIP-009 that usually means the current recovery plans with revision history, backup process documents, backup job logs or reports showing completed backups and how failures were handled, verification records, dated test records for each 15 month test and the sample data test, and for high impact systems the operational exercise or actual recovery record. Records should show the scope tested, the people involved, the results and any corrective actions. Confirm the exact request format and the audit period with the Regional Entity before the engagement, because evidence expectations are set during audit preparation.

Do open-source backup tools satisfy CIP-009 on their own?

No. A backup tool helps with the technical parts of CIP-009, but it does not make an entity compliant by itself. Bareos, for example, schedules and supervises backup, restore, verify and archive jobs, keeps a SQL catalog of backed-up files, and supports job, file and volume retention periods that control how long records stay in the catalog. restic's check command verifies the structural consistency of snapshots, trees and pack files, and with --read-data or --read-data-subset it reads pack files back to confirm the stored data is unmodified. Those outputs can feed the verification evidence required by Part 1.4. Recovery plans, the sample data test, the 15 month and 36 month tests, and the people and process controls still have to be written, performed and recorded by the entity. Confirm the license terms (Bareos is AGPL-3.0-only, restic is BSD-2-Clause) with legal counsel before deployment.

What does a partner need from the entity to scope a CIP-009 engagement?

A partner needs enough information to define written scope before any work starts. The entity should provide its CIP-002 categorization, which identifies the high, medium and low impact BES Cyber Systems that CIP-009 applies to, plus the current recovery plans, the backup tools and storage locations in use, the recovery environment available (if any), the list of recovery information held for each system, and the dates of its last 15 month and 36 month tests. It should also name the internal owner who approves the plan and who accepts the evidence. With those inputs a partner can propose deliverables and a timeline in a quote. Ask any partner to confirm in writing which tasks the entity keeps and which the partner performs.

When to bring in outside help for CIP-009

An in-house team can run CIP-009 testing once the recovery program is stable, and a partner adds the most value during the first build. If your staff already maintain backup jobs, know which BES Cyber Systems are high impact, and can document a recovery plan that a stranger could follow, the recurring work is mostly scheduling 15-month plan tests and data sample tests, recording results, and updating plans after each exercise. The 36-month operational exercise for high impact systems is the harder piece, because it needs a recovery environment representative of production. Those are the areas where a partner's experience with multi-site backup tooling is worth paying for, particularly when several high impact sites each need their own backup targets, retention settings and restore procedures built to the same standard.

Outside help is most useful in three situations: building Bareos or restic configurations across several Control Centers for the first time, standing up a test recovery environment that matches production closely enough to count as an operational exercise, and writing verification evidence so that each run produces a record an auditor can follow without staff explaining it. A partner can prepare the systems, scripts, runbooks and evidence templates. It cannot decide your recovery priorities, sign off on your plan, or certify that you comply. Ask any partner to confirm in writing which sites, Applicable Systems, tools and deliverables are in scope, who performs each test, who approves the results, and what happens to work that falls outside the agreed scope before any work begins.

Whatever you choose, an independent auditor still needs to assess compliance. The Regional Entity or NERC, acting as the Compliance Enforcement Authority, reviews the evidence your organization produces, so the entity owns the CIP-009 program and every record in it. Confirm with your Regional Entity how it expects backup verification results and exercise reports to be presented, and confirm the audit period you need to cover. A partner that helped build the program should not be the only reviewer of its evidence. Plan an internal review step before any audit, and keep the people who ran the tests available to explain their results.

How DeployOpen prepares CIP-009 backup and recovery evidence

DeployOpen plans, implements and documents the backup, restore and recovery testing systems that an entity uses to meet CIP-009 within its own infrastructure. The entity owns its recovery plans and its compliance program, and DeployOpen prepares the systems and evidence that the entity's independent auditor or the Compliance Enforcement Authority reviews.

How we scope

We agree in writing which BES Cyber Systems, recovery plans and backup repositories fall inside the engagement before work starts.
We confirm with you whether any system is high impact, since operational exercise obligations apply to that category.
We map each backup job, verification step and test to the CIP-009 part it supports, and record the gaps we agree to close.

How we build

We configure backup jobs and retention periods in your environment, using open-source tools such as Bareos or restic where they fit your requirements.
We set up verification steps that check backup completion and record failures, along with the alerting and follow-up process for them.
We build restore test procedures for representative data samples and document how each 15-month and 36-month test is run and logged.

How we hand over

You receive documented backup and recovery procedures, configuration records and test runbooks written for your own staff to operate.
Test results, job logs and catalog records are delivered in a format you can retain for the audit period, with retention settings documented.
We walk your team through running the next test cycle so the evidence trail continues without us. Confirm the handover format in the written scope.

How we support

Ongoing support, upgrades and continuous testing are available as a retainer after the initial engagement ends.
Retainer support can cover backup tool upgrades, restore test reruns and updates to recovery plans after system changes.
Ask us to confirm what a retainer includes, response expectations and how changes to recovery scope are priced before you sign.

Backup and recovery testing for CIP-009