How automated controls and benchmarking work in a SOX audit
If your company has an automated application control, such as a calculation or a three-way match that runs without human intervention, an auditor can test it once and then benchmark it in later years, provided the general controls over program changes, access and computer operations are effective and keep being tested. PCAOB AS 2201 supports this in paragraphs B28 and B29: entirely automated application controls are generally not subject to breakdowns from human failure, and the auditor may rely on a baseline of an unchanged control only when those ITGCs are working. The PCAOB Auditing Standard No. 5 (now AS 2201) sets out the full logic.
Whether a given control qualifies depends on the control, the system and the IT control environment around it. Paragraph B31 points to factors that favor benchmarking: the control can be matched to a defined program, the application changes rarely, and a reliable report of program compilation dates exists. Paragraph B30 adds that the control may depend on tables and parameters, such as a rate table feeding an interest calculation, and a change to those inputs can break the baseline even when the code is untouched. Management owns the controls and must document why a benchmark still holds. The external auditor decides whether to rely on it.
Before you ask for a benchmarking strategy, confirm three things with your audit team: the control's baseline and its supporting parameters are identified, the change, access and operations ITGCs over that system have operated effectively, and the baseline will be reestablished after a period set by your IT control environment, as paragraph B33 requires. Ask the auditor which evidence of compile dates and vendor version control they expect to see. Where the software is purchased and the vendor does not allow source code changes, paragraph B32 notes that benchmarking is often especially effective.
Which automated controls qualify for benchmarking
The first factor is whether the automated control can be tied to a specific, identifiable program or configuration. Auditors need to point to the object they tested, such as a named report program, a calculation module or a version of a configured routine, and confirm that the same object is what runs in production in later years. If the control cannot be matched to one program, the auditor cannot establish a baseline that means anything in the next year.
- Named object: The program, module or configuration item has a unique name or identifier that appears in the change and production records.
- Traceable mapping: Management documents which production objects implement the control, so the auditor can follow the link from the control description to the code.
- Weak case: Programs that are copied, renamed or duplicated per business unit without a clear master record.
- Weak case: Logic split across several objects where no one can say which one drives the result.
Confirm with the control owner and IT that the mapping document is current and that the object tested in the prior year is the one still in production today.
Benchmarking assumes the automated logic has not moved. AS 2201 treats a stable application with few changes as a factor that favors benchmarking. The auditor reviews the change history for the application and looks at how often the logic was modified, who approved the modifications and whether any modification touched the control logic itself. A high volume of changes, even if each one was approved, makes it harder to argue the baseline still holds.
- Change history: Program change logs show the in-scope object was not modified, or only modified for changes that did not touch the control logic.
- Change classification: Management separates cosmetic, performance and logic changes so the auditor can see what was touched.
- Weak case: Frequent emergency changes or hotfixes applied directly to production.
- Weak case: Business rules maintained in code that is edited regularly for new products or regions.
Check that the change log covers the full period since the baseline and that every change to the control's object has an approval and a test record.
A baseline is only useful if the auditor can prove that the program in production is the same program tested earlier. AS 2201 lists a reliable report of program compilation dates as a factor favoring benchmarking. The compile date shows when the executable was built, and matching it against the date of the baseline test is the evidence that nothing changed underneath the control. Without a trustworthy report, the auditor has no independent way to confirm the object is unchanged.
- Source of the report: The report should come from the system that controls the production environment, not from a document prepared by the application team.
- Completeness: It should list every in-scope object and its compile date, so no program falls outside the comparison.
- Access to the report: The people who can edit or regenerate the report should be limited and logged under access controls.
- Weak case: Compile dates that are overwritten on each build or that cannot be reproduced for an earlier period.
Ask IT to show how the report is generated, who can run it, and whether it can be produced for the exact date of the prior-year baseline.
AS 2201 notes that benchmarking is especially effective for purchased software when program changes are remote, such as when the vendor does not allow the customer to access or modify the source code. In this situation the organization cannot alter the logic internally, which reduces the chance of an undocumented change. The argument still depends on the organization controlling its installed version, its upgrade path and any vendor patches applied to production.
- Vendor restriction: The license or support terms confirm the organization cannot modify the source code.
- Version control: The installed version and each patch or upgrade are recorded with dates and approvals.
- Vendor evidence: Release notes or vendor certifications describe what the version does for the control.
- Weak case: Customizations layered on top of purchased software, such as custom scripts or exits, that change the output of the vendor logic.
Confirm whether any customization, configuration or script touches the control, because those items must be tested and controlled separately.
AS 2201 (paragraph B30) observes that automated application controls often depend on related files, tables, data and parameters. A common example is a rate table used by an automated interest calculation. The program can be unchanged and still produce wrong results if the rate table is edited without control. For benchmarking, the auditor needs assurance that these inputs are controlled with the same discipline as the program itself, or the baseline does not reflect what the control actually does.
- Parameter inventory: List every table, parameter file and master data set the control reads, such as rate tables, tolerance thresholds and account mappings.
- Change control on data: Edits to these inputs go through the same authorization, testing and approval steps as program changes.
- Access restrictions: Only authorized users can modify parameters, and their activity is logged.
- Weak case: Rates or thresholds maintained in spreadsheets, or changed directly in the database by application support staff.
Confirm with the control owner which parameters feed the control and whether program change and access controls cover them.
What evidence supports a benchmark
A benchmark holds only when the evidence shows that the tested automated control ran on the same program, under the same governance, in every year it is relied on. The cards below cover the artifacts the organization owns and retains, and the auditor verifies their accuracy before any baseline is carried forward.
Program compile date reports
A compile date report lists when each program object was built, so the auditor can match the production module to the version that was tested. Check that the report comes from the production environment, that it covers every program in the control's logic, and that the dates line up with the baseline test. A report pulled from a development copy does not prove what ran in production, so confirm its source before accepting it.
Change tickets showing no changes
Program change records for the application over the period should show that no code affecting the tested control was modified. Each ticket needs an approver, a test record and a production deployment date. If a change does touch the control, the baseline must be reestablished, so the change log is the first check on whether benchmarking still applies.
Vendor release and version records
For purchased software, the vendor's release notes, patch history and installed version records show whether the application changed during the year. These records matter most when the source code is not available to the customer, because the organization cannot inspect modifications directly. Confirm that the version in production matches the version in the vendor documentation for each period.
Parameter and rate table logs
Automated controls often depend on tables, rates and parameters rather than code alone, such as a rate table that drives an interest calculation. Change logs for these files should show who edited each value, when, and who approved it. Changes to these inputs can alter results without touching the program, so they need the same scrutiny as code changes.
Access and job-scheduling evidence
Benchmarking relies on the general controls over the application staying effective. Gather user access reviews for production, privileged access logs, and the job schedule showing when the control runs and whether it completed. If these records show unauthorized access or failed jobs, the auditor may not be able to carry the baseline forward.
Baseline test script and walkthrough record
The original test script, its inputs and outputs, and the walkthrough notes form the baseline the auditor compares against in later years. Keep the script, the test data and the sign-off together so the baseline can be reperformed. Record the date the baseline was established, because reestablishment intervals depend on how long the baseline has been in use.
How the baseline is set, tested and reestablished
This sequence establishes a benchmark for an automated control in a base year, confirms the evidence that makes the control repeatable, and keeps reliance valid in later years by testing the supporting ITGCs and checking for unchanged code. It also defines when the baseline must be reset.
Scope the control and its ITGC dependencies
Management and the external auditor first identify the automated application control, the program it runs in, and the related files, tables, data and parameters it depends on, such as a rate table used by an interest calculation. They then map the access, program change and computer operations ITGCs that protect those elements. The output is a documented control description with a list of dependencies and the ITGCs that must remain effective for reliance to hold.
Establish the baseline through a base year test
In the base year, the auditor performs a full test of the automated control to confirm it operates as designed, including testing the logic against defined criteria and reviewing exceptions. This test is the reference point for every later year. The output is a signed-off base year test with the program identified, the version tested and the conclusion on design and operating effectiveness.
Confirm program identity and compile date evidence
The team obtains a reliable report of program compilation dates and matches the tested program to a defined version, so the auditor can show that the code in production is the code that was tested. Where the application is purchased software, the team also confirms with the vendor which version is installed and whether source code access or modification is possible. The output is a compile date report linked to the baseline program, with the vendor version evidence retained in the workpapers.
Test the ITGCs that support reliance
Each year, before relying on the baseline, the auditor tests the program change, access and computer operations ITGCs that protect the control and its dependencies, and confirms they operated effectively throughout the period. If any of these ITGCs fail, reliance on the unchanged baseline is not supportable without further testing. The output is a documented conclusion for each ITGC area that either supports reliance or identifies the failure and its effect on the benchmarked control.
Roll forward with an unchanged control check
For the current year, the team confirms that the program and its dependencies have not changed since the baseline, using the compile date report, change tickets and access to the production environment. They also confirm that the related files, tables and parameters did not change in ways that alter results. The output is a roll-forward memo that compares current compile dates and change records to the baseline and documents any differences that require additional testing.
Reestablish the baseline when circumstances require
The baseline does not last indefinitely. Under PCAOB AS 2201 paragraph B33, the baseline should be reestablished after a period that depends on circumstances, including the strength of the IT control environment. Management and the auditor set that interval during planning and reset the baseline sooner after a program change, a failed ITGC or a version upgrade. The output is a new base year test that replaces the prior benchmark and an updated record of when the next reset is due.
When a benchmark must be reestablished
A benchmark is not valid indefinitely. PCAOB AS 2201 (paragraph B33) says the baseline of an automated control should be reestablished after a period that depends on the circumstances, including the IT control environment. There is no fixed interval in the standard, so the first decision is to agree the refresh cycle with the external auditor for your own environment before the baseline is tested. Ask the auditor which changes they would treat as resetting the clock and whether they expect a fixed maximum period in addition to event-driven refreshes.
In practice, the events that most often justify a new baseline are a program change to the application during the period, a failed or emergency change that reached production without the expected approval, an access or computer operations deficiency affecting the application or its database, a vendor upgrade or patch that alters the program, and a new parameter or rate table that changes how the control calculates. Each of these can break the link between the tested version and the version running in later years. Build a trigger log that records each event, its date, the owner who assessed it and whether it required retesting. Illustratively, a change to a tax rate table with no code change still belongs on that log, because AS 2201 treats related tables and parameters as part of the control.
The benchmark also depends on the general controls staying effective. If program change, access or operations testing shows a deficiency in the year, the reliance on the prior baseline is at risk even when the application itself did not change, so the auditor may require the control to be retested from a new baseline. Plan for this before year end by confirming, with your auditor, how a deficiency in a related ITGC affects the baseline, and by keeping compile dates, change tickets and access reviews for the application together so the evidence for any refresh can be produced quickly.
Where benchmarking usually breaks down
Benchmarking an automated control is only as sound as the IT general controls around it. The pitfalls below share one root cause: the application can change without the change being visible to the people and records that support the baseline. Each fix is an action the organization must own, not something an auditor or implementation partner can perform on its behalf.
Compile dates that do not match the baseline
A baseline is only defensible when the compile date of the program the auditor tested matches the compile date of the program in production. Auditors look for a reliable report of compilation dates that ties each production module to its tested version. Check whether the organization can produce that report for every in-scope module, and whether anyone reconciles it against the change tickets for the period. Owners of the application and the change management process must run this reconciliation, not the auditor.
Undocumented emergency changes
Emergency fixes often reach production through a fast path that skips testing and approval, and the code then differs from the benchmarked version. Even one undocumented emergency change can invalidate a baseline for the affected control. Management should define what qualifies as an emergency, require after-the-fact approval within a set window, and review emergency access and deployment logs each period. Ownership sits with IT change management and the control owners who depend on the application.
Shared administrative access to production
When several people share an administrator account on the production application or its database, no one can be shown to have made or been unable to make a change. That undermines the access controls that keep the program unchanged between tests. Management should assign named administrator accounts, restrict privileged access to a defined group, and review that group's activity on a schedule. Access ownership belongs to the organization, and the review must be performed and retained by its staff.
Spreadsheets and report writers outside ITGCs
An automated control often depends on a spreadsheet, a report writer or a custom query that supplies a rate, a list or a threshold. If those tools are not under access and change controls, the automated control can be altered without any trace in the application. Inventory every file, table and query the control depends on, then assign each an owner and bring it under the same access and change rules as the program. The organization must maintain this inventory, since it reflects its own reporting chain.
Vendor patches applied outside change management
Purchased software is often the best case for benchmarking because the source code is not changed locally, but vendor patches, hotfixes and version upgrades still alter the program. If these arrive through a side channel, the compile date changes without a matching approval record. Management should log every vendor release installed, confirm each one against the baseline, and decide whether a new baseline is needed. Patch intake and approval are organization-owned steps, even when the vendor delivers the code.
Parameter changes made outside the program
Automated controls often read rate tables, tolerances and approval limits that live in configuration tables rather than code. Changing a value in a table can change the control's output while the program's compile date stays the same, so a benchmark based only on the program misses the change. Management should treat configuration data as part of the baseline, restrict who can edit it, and log each change with a reviewer. Setting these owner responsibilities and review cadences is an action the organization must take itself.
Benchmark or test the control every year
This table compares benchmarking an automated control, which relies on effective ITGCs, with testing the control directly each year. Read each row as a decision point: the conditions that favor each approach, what the auditor needs to see, the main risk, and the effort it creates. The main takeaway is that benchmarking saves repeat work only when change, access and operations controls are effective and tested every year.
| Factor | Benchmark with ITGC reliance | Test the control each year |
|---|---|---|
| When it fits | Application is stable, maps to a defined program, and changes are few or remote, such as purchased software the vendor does not let you modify | Application changes often, is custom-built, or ITGCs over changes, access or operations are weak or not yet tested |
| What the auditor needs | Baseline test of the unchanged control, a reliable compile date report, and evidence that program change, access and computer operations ITGCs are effective and tested each year | Each year's sample of control operations with supporting evidence, without reliance on a prior baseline |
| Main risk | An undetected change or unauthorized access after the baseline, which invalidates reliance if ITGCs fail | Higher annual sample and evidence effort, and more exposure to deficiencies in manual follow-up |
| Dependence on ITGCs | High: reliance rests on change, access and operations controls working every year | Lower: the control is tested on its own operation each year |
| Related files and parameters | Rate tables, thresholds and configuration must also be covered by ITGCs and reviewed as part of the baseline | Parameters are checked as part of each year's operating test |
| Baseline refresh | Reestablish the baseline after a period set by circumstances and the IT control environment, confirm this with your auditor | Not applicable |
| Effort profile | Heavier first-year baseline work, then lighter annual testing if ITGCs hold; confirm the scope and effort with your auditor and internal audit | Consistent effort every year, scaled by sample size and control frequency |
Who owns the controls, the evidence and the auditor's conclusion
Management owns the internal control over financial reporting, including every automated application control that management relies on and the IT general controls that keep those automated controls reliable. Management also owns the evidence. Benchmarking only works if management can show that the program in production matches the approved baseline, that changes to it were authorized, and that access to it stayed limited. The auditor tests what management produces, so the records must exist and be retained in the company's own systems.
- Control ownership: Named owners for each automated control, the baseline it is matched to, and the decision to rely on it in each year.
- Change records: Approved change tickets, test sign-offs and promotion records showing that the application was not modified outside the process.
- Access reviews: Periodic reviews of who can change the program, its parameters and the rate or reference tables it uses.
- Compile evidence: Reports of program compilation dates that tie the production object to the version tested.
- Baseline reestablishment: A documented decision on when the baseline must be retested because the IT control environment changed.
Before relying on a benchmark, confirm in writing which person signs off each of these records and where they are stored.
An implementation partner such as DeployOpen can help prepare the systems and evidence that management uses. For self-hosted open-source platforms in the client's own infrastructure, that work can include configuring logging and evidence collection, setting up version control and promotion pipelines, documenting configuration baselines and building repeatable extraction of compile and access data. The partner does not decide whether a control is effective, does not select the reliance strategy for the auditor, and does not sign anything on the company's behalf. Those decisions stay with management and the auditor.
- System preparation: Hardening, separation of development and production, and locking down who can deploy changes.
- Evidence configuration: Audit logging and exports for change, access and job activity, set up so management can retrieve them.
- Baseline documentation: Written records of the approved program version, its configuration and the related tables and parameters.
- Repeatable retrieval: Scripts or procedures that produce the same evidence set each cycle, with clear ownership.
- Scope boundaries: A written statement of what the partner did and did not touch, so the company can explain it to the auditor.
The engagement should be fixed in writing with defined deliverables, and the company should confirm that the partner's role is limited to preparation.
The external auditor tests internal control over financial reporting under PCAOB AS 2201 and assesses IT risks under AS 2110. The auditor decides whether a baseline can be relied on in later years, based on whether general controls over program changes, access and computer operations are effective and continue to be tested. Only the auditor issues the attestation on ICFR. A consultant, a partner or management's own staff may prepare materials for the audit, but none of them can issue the auditor's report or the attestation.
- Testing: The auditor tests the automated control and the related ITGCs, including whether the benchmark matches the production program.
- Reliance decision: The auditor determines whether benchmarking is appropriate given the stability of the application and the reliability of compilation reports.
- Remote change factors: For purchased software where the vendor restricts source code access, the auditor evaluates whether program changes are indeed remote.
- Opinion: The auditor issues the ICFR opinion as part of the Section 404 attestation where it applies.
- Independence: Management and any advisers must keep the auditor's independence boundaries in mind when preparing materials.
Confirm early with your audit firm which evidence it will accept and how it will test reliance for each automated control.
Questions buyers ask about benchmarking automated controls
Can we benchmark a purchased ERP module instead of retesting it every year?
Often yes, and purchased software is where benchmarking is most effective. PCAOB AS 2201 (paragraph B32) notes that benchmarking works especially well for purchased software when program changes are remote, for example when the vendor does not allow access to or modification of the source code. Your own configuration still matters, though. Auditors will want to see that the module is the vendor's unmodified release, that your configuration settings are controlled, and that changes to them go through change management. If your team customizes the code or writes its own calculations, those pieces need a different approach.
What does the auditor need from us to rely on a benchmark?
The auditor needs evidence that the automated control is the same one tested before and that the general controls around it still work. AS 2201 (paragraph B31) points to three factors: the control can be matched to a defined program, the application is stable with few changes, and a reliable report of program compilation dates exists. In practice that means a compile date report, change tickets showing no unapproved modifications, access reviews for the people who could alter the program, and operations evidence for the processing that feeds it. Your team should also keep the related tables and parameters, such as a rate table used by an interest calculation, under the same control, because AS 2201 (paragraph B30) says automated controls can depend on them.
How long does a benchmark baseline stay valid?
There is no fixed period in AS 2201. Paragraph B33 says the baseline should be reestablished after a period that depends on circumstances, including the IT control environment. A stable application with strong change, access and operations controls can usually go longer between re-baselines than one with frequent releases or weak general controls. Ask your auditor early what re-baselining cadence they expect for your environment, and record any upgrade, patch or configuration change that touches the control, since those events normally trigger a fresh test.
Does benchmarking reduce our SOX audit fees?
It can reduce the testing effort for those controls in years after the baseline, because the auditor tests the unchanged control once rather than sample by sample each year. Whether fees fall depends on how many controls you benchmark, how stable they are and how much of the general control testing still has to be repeated. Benchmarking does not remove general control testing, and it does not remove the work of confirming that no change occurred. Ask any firm you engage to show, in writing, which controls they plan to benchmark, which ITGCs they will test each year, and how their estimate changes if a control fails.
What happens if one ITGC failed last year?
A failed ITGC weakens the case for benchmarking, because AS 2201 (paragraph B29) only lets the auditor rely on a baseline of an unchanged automated control if the general controls over program changes, access and computer operations are effective. If a change control failed, for example an emergency change that skipped testing, the auditor may need to retest the automated controls that depend on that area, and may not be able to benchmark them until the failure is remediated and tested. Your team should assess the impact on each affected control, document any compensating controls, and confirm with the auditor whether the baseline can be kept or must be rebuilt.
Can our own team set up and evidence benchmarking without outside help?
Yes, many organizations do this in-house, provided they have the people and records to support it. Management owns the controls and the evidence, and the external auditor makes the independent judgment on whether the benchmark is reliable. Your team will need staff who understand the application, change and access records that can be produced on request, and a clear list of the automated controls and their dependencies. Outside help is commonly used to prepare systems, document the control design, collect compile and access evidence, or run a readiness review before the auditor arrives. Any engagement should have a written scope, and only the independent auditor issues the opinion.
How DeployOpen prepares systems and evidence for benchmarking
DeployOpen's role in a benchmarking strategy is to prepare the systems and evidence that the organization and its external auditor will examine, not to decide whether a control passes. We start by writing down, for each automated application control the company intends to benchmark, the program it runs, the related files, tables, rate tables and parameters it depends on, and the change, access and computer operations controls that the auditor would need to rely on. That written scope is agreed before work begins, so the organization can see which ITGCs must stay effective and continue to be tested in every year the baseline is used.
Next, we implement and document the self-hosted open-source platform inside the client's own infrastructure, with the evidence trail the benchmark depends on. That means a documented version-controlled change process with approvals and test records, access reviews for the application, its database and its administrative accounts, and operations records such as job schedules and backup restores. For the compile date evidence that supports a benchmark, we configure build and deployment logging so each production build can be matched to its source revision and timestamp. Where the platform is customized, we record which code is stable and which parameters are owned by the business, so the baseline covers the right items.
At handover, the client receives a baseline package: the control scope document, the list of dependent files and parameters, the compile date and build records, the ITGC evidence index, and a reestablishment schedule that the organization sets with its auditor. Decisions remain with the company: whether a control is benchmarked, how often the baseline is reestablished, and which deficiencies are escalated. DeployOpen does not issue reports, attestations or opinions on ICFR or on any control. Those are issued only by the independent external auditor, and the scope of that auditor's work should be confirmed with them before any package is prepared.
