Design vs operating effectiveness testing for SOX ITGCs

Design vs operating effectiveness testing for SOX ITGCs: what each one proves

Design testing asks whether an IT general control, if performed by competent people with the necessary authority, would meet its objective and prevent or detect errors or fraud that could cause a material misstatement. Operating testing asks whether the control was actually performed that way throughout the period. In a 404 year, both answers feed the management assessment, and the auditor's request list usually separates them into distinct items. PCAOB AS 2201 paragraphs 42 and 43 describe design effectiveness and the inquiry, observation and inspection procedures that ordinarily support it, while paragraphs 44 and 45 describe operating effectiveness and add re-performance to that mix.

A design conclusion can be reached from a walkthrough. If a user access review is designed to compare granted roles against approved job requests every quarter, a walkthrough that traces one request through approval, provisioning and review shows whether the review step would catch an inappropriate grant. An operating test then samples the reviews that actually happened during the period, checks that the reviewer had authority and competence, and re-performs the comparison on a sample. A control can pass design and still fail operation, for example when the quarterly review is signed off without anyone comparing the lists. Operating evidence therefore needs to show the work, not only the signature.

Small finance and IT teams often lack the segregation of duties that the design of many ITGCs assumes. AS 2201 accepts alternative controls in that situation, and paragraph 44 lets the auditor weigh the combined competence of company staff and outside parties who help with reporting. For a team of a few people, the practical question is whether a compensating review, such as a monthly report of administrator activity reviewed by someone outside the administrator group, is designed to catch what segregation would have caught and whether it ran every period. Confirm with your auditor early which alternative controls they will accept, because the organization owns the conclusion, the auditor tests it independently, and the answer depends on the control, its risk and the size of the team.

Which procedures show a control operated as designed

Inquiry means asking the people who perform a control how it works, how often it runs, who approves it and what they do when something looks wrong. The auditor under PCAOB AS 2201 treats inquiry as part of the operating effectiveness procedures, but never as the only one. Answers show understanding and authority, not that the control actually ran in the period.

What it proves: that the performer knows the steps, the frequency and the escalation path, and has the authority and competence the standard expects.
What it does not prove: that the control operated on any specific date. A confident answer can describe the intended process rather than what happened.
Illustrative example: asking the access administrator how the quarterly user access review is performed, who signs off and what happens to accounts flagged for removal.
Typical gap: an answer that differs from the written procedure, which signals a need for further procedures before any conclusion.
Pairing: inquiry is combined with observation, inspection or re-performance so the answer can be checked against records.

Check that every inquiry is documented with the person's name, role and date, and that each answer is corroborated by at least one of the other procedures.

How sample size, period and frequency change operating testing

Operating effectiveness testing does not use one fixed amount of evidence for every control. The factors below shape how much an auditor or management tester needs to collect, and each one should be confirmed against the engagement plan before fieldwork starts.

Control frequency drives the population

A control that runs once a year, such as an annual access recertification, produces one instance to test, while a daily batch reconciliation produces hundreds across the period. Frequency sets the size of the population the tester samples from, and it is the first input to any sample size. Ask the engagement team how each in-scope ITGC was classified by frequency and how the population was counted from system logs or tickets.

Automated versus manual controls

An automated control that is tested once and shown to be unchanged, with its change and access controls working, may need far less operating evidence than a manual review performed each cycle. Automated application controls rely on ITGCs that keep the logic stable, so testing often shifts toward those supporting controls. Confirm with the auditor which ITGCs support each automated control and whether they are tested directly.

Period covered by the test

The period over which a control is tested determines how many instances fall within it. A test covering a partial year needs more care to reach the fiscal year-end assessment date, and any gap between the tested period and year end must be bridged by additional evidence. Ask whether the testing window matches the fiscal year, and how interim results will be rolled forward to the assessment date.

Risk of the underlying account

Controls over accounts with higher risk of material misstatement, or with greater susceptibility to error or fraud, generally receive more extensive testing than controls over low-risk balances. The risk assessment that places an application or process in scope therefore feeds directly into how much evidence is pulled. Ask for the documented risk rationale behind the chosen testing extent for each key ITGC.

Sample sizes set by the auditor

Exact sample sizes are not published in a single SEC or PCAOB table. They are set by the external auditor's methodology and by the firm's own testing policies, and they can change with the factors above and with prior-year results. Do not assume a number in planning. Request the sample size and selection method for each control in the written engagement plan, and confirm whether management's testers will use the same approach.

How a failed design or operating test becomes a deficiency

A failed test is not automatically a material weakness. Under PCAOB AS 2201 (paragraph A3), a control deficiency exists whenever a control's design or its operation does not let management or employees prevent or detect misstatements on a timely basis in the normal course of their work. That definition covers both branches of this page. A design gap, such as an access review that never compares user rights against job duties, is a deficiency even if nobody has yet exploited it. An operating failure, such as a change approval that was skipped for two releases, is also a deficiency even if the control was well designed. Both go on the same log, and both need an owner and a remediation plan.

Severity is a separate judgment. The terms *significant deficiency* and *material weakness* are defined in 17 CFR 210.1-02, and the decision turns on whether there is a reasonable possibility that a misstatement would not be prevented or detected in time, and on how large that misstatement could be. For ITGCs, the reviewer asks which automated controls, reports and interfaces depend on the failed control, and whether a compensating control, such as a management review of the affected report, would catch the error. Compensating controls only count if they are precise enough to detect a misstatement of the relevant size and are themselves tested. Confirm the threshold and documentation expectations with your external auditor before the test results are final, because the same failure can rank differently depending on the reports it touches.

How smaller teams with limited segregation of duties are evaluated

Smaller companies often cannot separate every incompatible duty, so auditors look for compensating controls that catch the same errors or fraud. The cards below cover the AS 2201 notes that allow this, the kinds of alternative controls that get tested, and the evidence that lets an assessor re-perform them.

The note on alternative controls

A note to AS 2201 paragraph 42 says a smaller company with fewer accounting staff may implement alternative controls where duties cannot be segregated. The auditor then evaluates whether those alternatives are effective. The alternative does not excuse the gap. It has to do the job that segregation would have done, so check first which conflicting duty it replaces, such as a developer who can also promote code to production.

Independent review of privileged access

*Illustrative example:* a finance manager with no system administration role reviews a quarterly export of administrator accounts for the general ledger and ERP, confirms each user still needs the role, and signs the review. This works only if the reviewer cannot grant or change access, and if the export comes from the system rather than a spreadsheet the administrator prepared. Test it by selecting a review period and tracing a flagged account to its removal.

Reviewed change reports

*Illustrative example:* when one person both builds and deploys a configuration change, a second person reviews a system-generated report of every production change in the period, compares it to approved tickets, and investigates any change without a matching approval. The report must be complete, which means the reviewer or auditor should confirm it pulls from the production change log and not a filtered list. Keep the reviewer's sign-off dated and tied to the report version.

Competence of personnel and outside parties

A note to AS 2201 paragraph 44 lets the auditor consider the combined competence of company personnel and outside parties who assist with financial reporting functions, which matters most in smaller companies. An outside accountant or IT provider can strengthen a review, but the auditor will ask whether that party has the authority and skill to spot errors and whether the company retains responsibility for the review. Document who performs each review, their role, and their knowledge of the system.

Evidence that makes an alternative testable

An alternative control can only be tested if it leaves a trail. Auditors look for a defined review frequency, a named reviewer, a population the reviewer actually saw, a documented follow-up on exceptions, and retained evidence of each review period. Without a complete population, an assessor cannot confirm the review covered everything. Ask control owners to show the report, the sign-off and the follow-up for one sampled period before fieldwork begins.

Deficiencies in compensating controls

If the alternative control is not precise enough, or the reviewer lacks the competence to detect misstatements on a timely basis, the auditor may classify the gap as a control deficiency under AS 2201 paragraph A3. Its severity depends on whether a misstatement could go undetected, which is a judgment for management and the auditor. Confirm early whether the alternative operates at the precision needed, rather than discovering the shortfall during year-end testing.

How management sequences design and operating tests in one cycle

The sequence confirms that each IT general control is designed to meet its objective before anyone tests whether it ran as designed, so that operating results mean something. Each step ends with a named output that the next step depends on.

Confirm the control objective and risk
The control owner and the evaluator agree on the financial reporting risk the control addresses, such as unauthorized changes reaching a financially significant application or a batch job failing without notice. The output is a short control description that states the objective, the system it covers, and the people who perform and review it. If the objective cannot be stated against a reporting risk, the control is not yet ready for testing and the scope should be revisited.
Run the design walkthrough
The evaluator follows one representative transaction or change through the system with the control owner, using inquiry, observation and inspection of documents. The question is whether the control, if performed as described by someone with the right authority and competence, would prevent or detect an error or fraud that could cause a material misstatement. The output is a walkthrough memo that records the steps observed and a clear conclusion on design.
Fix design gaps before testing operation
Where the walkthrough shows that the control is missing a precision threshold, a review of the right population, a reviewer with access the preparer should not have, or an unmonitored exception path, management corrects the design or documents an alternative control. Testing operation against a design that does not work wastes effort and can hide the real gap. The output is either a confirmed design with the updated control description, or a documented deficiency that goes into evaluation.
Select the operating period and samples
The evaluator sets the period the control must have operated across and selects items from its population using a method that can be explained, such as random selection from a complete system report. For automated controls, the evaluator often tests a small number of instances because the control should work the same way each time, provided that related general controls remain effective. The output is a documented population, sample selection and the reason each item was chosen.
Gather operating evidence
The evaluator obtains the evidence that the control operated for each selected item, using inquiry of the people involved, observation where practical, inspection of approvals, logs and tickets, and re-performance where the control can be recalculated independently. Evidence should show who performed the control, when, and what was checked. The output is an evidence file for each sample item, cross-referenced to the control description so a second reviewer can re-perform the test.
Evaluate deficiencies and classify them
Each exception is assessed against the definition of a control deficiency, which covers both a design problem and a failure in operation to prevent or detect misstatements in the normal course of work. The evaluator considers whether compensating controls exist, how likely and how large a misstatement could be, and whether the exception is isolated or systemic. The output is a deficiency log that records severity, root cause and whether the item is a significant deficiency or material weakness under the applicable definitions.
Document the conclusion for each control
The evaluator writes a conclusion that states whether the control is designed effectively, whether it operated effectively throughout the period, and what evidence supports each view. Management reviews the conclusion, and the external auditor, if the company is subject to an attestation, can then decide how far it will rely on the work. The output is a signed-off test conclusion per control that stands on its own and can be reperformed by someone who did not do the original test.

What to check before the auditor tests design and operation

Auditors can only test a control if it is described, evidenced and owned by someone qualified to run it. The four checks below cover the artifacts that let an assessor move from design to operating effectiveness without stopping to rebuild the record.

Written control descriptions with owners

Each ITGC needs a written description that states what the control does, who performs it, how often it runs, and what it produces. A description that says only "access is reviewed" gives an auditor nothing to test against. To verify this, pick three controls at random and check whether a stranger could identify the named owner, the frequency and the output from the description alone. If the frequency is missing or reads "as needed," the control is not yet testable.

Evidence retained for the full period

Operating effectiveness is judged across the whole fiscal period, so the auditor needs evidence from the first day of the period through the last, not a sample from the quarter when the control was set up. Check that access review sign-offs, change approvals and job logs exist for every period the control ran, and that they can be retrieved without asking the system owner to recreate them. Ask your retention policy owner to confirm how long logs and tickets survive in each system.

Reviewers with authority and competence

A review is only evidence if the reviewer has the authority to act on what they find and the knowledge to recognize a problem. An auditor will ask whether the person signing an access review actually understands the application's roles and whether they can remove a user. Confirm that each reviewer is named in the control description, has the system access or business knowledge to judge the list, and is not the same person who requests the access under review.

Alternative controls for small teams

When one person must hold conflicting duties, segregation of duties cannot be achieved and the design has to compensate. Document each conflict, the alternative control that covers it, such as an independent review of logs or of transactions the individual approves, and who performs that review. Then test the alternative itself: ask whether the reviewer would detect an inappropriate change, and keep proof that the review occurred each period. An alternative that exists only on paper will fail operating testing.

Who performs each test and who owns the conclusion

This table compares management's self-testing, internal audit or a co-sourced team, and the external auditor across design testing, operating testing and deficiency evaluation. Read each row as a question of who does the work and who signs the conclusion. The main takeaway is that the organization owns its controls and its ICFR conclusion, while only the independent auditor issues the attestation report.

Activity	Management self-testing	Internal audit or co-sourced team	External auditor
Design testing	Management documents each ITGC, walks through a transaction and concludes whether the control could meet its objective	Tests design independently when management relies on it, using inquiry, observation and document inspection	Tests design through walkthroughs and inquiry, observation and inspection under AS 2201
Operating testing	Gathers direct tests, monitoring and self-assessment evidence for the 404(a) assessment	Re-performs or samples control operation, reporting results to management and the audit committee	Selects samples, re-performs controls and decides how much of the internal work it can use
Competence and objectivity	Weighted less when the tester performs the control or reports into its owner	Must be competent and objective to be relied on, with reporting lines separate from the control owner	Evaluates whether the tester's work is adequate before relying on it
Alternative controls in small teams	Documents compensating controls where duties cannot be segregated and shows they operate	Can test whether those alternative controls are designed and operating as described	Evaluates whether alternative controls are effective, per the AS 2201 note on smaller companies
Deficiency evaluation	Assesses severity and whether a deficiency is a control deficiency, significant deficiency or material weakness	Reports deficiencies and suggests severity, with final classification owned by management	Independently evaluates deficiencies and communicates significant ones to management and the audit committee
Conclusion and report	Management issues the ICFR assessment as of fiscal year end	Provides supporting evidence and testing results, without issuing an opinion	Issues the attestation report on ICFR where Section 404(b) applies
Role of implementation partners	Owns the controls and decides what evidence to retain	Owns its testing scope and results	Not applicable: an implementation partner prepares systems and evidence but does not test for the auditor or sign conclusions

Design vs operating effectiveness questions buyers ask

Is a walkthrough alone enough to test design of a SOX ITGC?

For design, a walkthrough that combines inquiry, observation and inspection of documentation ordinarily is sufficient, according to PCAOB AS 2201. The auditor asks whether the control, if operated as prescribed by people with the right authority and competence, would meet its objective and prevent or detect errors or fraud that could cause a material misstatement. A walkthrough does not prove the control ran throughout the year. It shows that the control is built correctly. Operating effectiveness needs separate evidence, such as re-performance and inspection of records across the period. Ask your auditor which walkthrough steps they expect to see written down for each access, change, development and operations control.

How much operating evidence does an ITGC need to pass?

The amount depends on the control's nature, its frequency and the risk it addresses, and the auditor sets the sample. AS 2201 describes operating tests that combine inquiry of personnel, observation of operations, inspection of documentation and re-performance of the control. A monthly user access review needs twelve instances with evidence of review, while a daily job check needs many more. Automated controls that do not change often may need fewer tests once the change history supports reliance. Confirm sample sizes and the population period with your auditor before testing starts, because a population that is incomplete or unsupported by system reports will usually lead to extra work.

Can a small team pass ITGC testing without full segregation of duties?

Yes, a smaller company can pass when it implements alternative controls, provided the auditor finds those controls effective. AS 2201 includes a note that a smaller company with fewer accounting staff may use alternative controls where duties cannot be segregated, and the auditor evaluates whether they work. Typical alternatives include an independent review of administrator activity logs, approval of changes by someone other than the developer, and periodic review of who holds privileged access. The review must be documented, performed by someone with the authority and competence to catch problems, and retained. Ask your auditor to agree the alternative controls in writing before the year begins.

Who pays for ITGC testing: auditors, staff, partners or tools?

Cost falls into four buckets. The external auditor charges for its testing under the attestation, which is a recurring annual fee. Internal staff time for control owners, evidence pulls and walkthrough support is an ongoing internal cost. An implementation partner charges for preparing systems, configuring logging, building evidence collections and documenting controls, usually as a one-time project with some recurring support. Tooling may add licenses or hosting. The SEC estimated in its 2020 release on accelerated filer definitions (Release 34-88365) that an issuer newly exempt from the auditor attestation would save about $210,000 per year on average, roughly $110,000 in audit fees and $100,000 in other costs. That estimate covers the attestation burden, not ITGC testing for companies that remain subject to it. Ask each provider for a quote that separates one-time and recurring items.

Can an implementation partner run the ITGC tests for us?

A partner can prepare the systems and evidence, but it cannot issue the conclusion. DeployOpen plans, implements, configures, hardens, documents and supports self-hosted open-source platforms in your own infrastructure, and it can prepare controls and evidence for assessment. It does not issue audit reports, attestations or certifications. Management owns the controls and must perform or oversee its own evaluation under the Section 404(a) assessment, and the independent auditor performs the attestation under AS 2201. Any testing a partner performs should be scoped in writing, and the results should be reviewed by an evaluator independent of the control owner. Confirm with your auditor how much of the testing they will rely on before agreeing scope.

What must our team supply for design and operating tests?

Your team must supply the people, system access and records that the tests depend on. For design, that means control narratives, a list of the applications and infrastructure in scope, the people who perform each control and their authority, and access for walkthroughs. For operating effectiveness, it means complete populations from the systems, such as access lists, change tickets, job logs and backup reports, along with proof that each review or approval happened. Each control owner should be available during the period to answer inquiry and explain exceptions. Ask your auditor for a document request list early, and confirm that system-generated reports are complete and accurate before they are given to testers.

What happens if a control passes design but fails operation?

The control is designed correctly but did not operate as designed, which is an operating deficiency. AS 2201 defines a control deficiency as a design or operation of a control that does not let management or employees prevent or detect misstatements on a timely basis in the normal course of work. Failed operation does not automatically mean a material weakness. The auditor and management evaluate its severity by considering the reasons for the failure, whether compensating controls exist and whether other controls cover the same risk. Keep exception logs, root cause notes and retest evidence, because these determine how the deficiency is classified. For definitions of material weakness and significant deficiency, see the SEC rules on definitions in 17 CFR 210.1-02.

How DeployOpen prepares systems and evidence for design and operating tests

DeployOpen plans, implements and documents the self-hosted open-source platforms that carry financially significant processes, inside the client's own infrastructure. The work is set up so that management can show how each IT general control is designed and how it operated across the period, while the independent auditor keeps its own role in testing and reporting.

How we scope

Engagements are fixed-scope, with the systems, controls and periods agreed in writing before work starts.
The written scope names which controls are designed for review and which operating samples the client will pull.
Ongoing support and continuous testing are available as a retainer after the initial scope is delivered.

How we build

Access, change and operations controls are configured so each one leaves a record a reviewer can inspect.
Approval workflows, role assignments and logging are set up to match the control descriptions in the client's matrix.
Test scripts for design walkthroughs and operating re-performance are prepared alongside the configuration work.

How we hand over

Documentation covers control narratives, system configurations and the evidence location for each control.
Evidence packs are organized by control and period so the client's testers and the external auditor can re-perform checks.
Your team receives written handover notes that name the control owners, the review cadence and open items to confirm.

How we support

Support after handover can cover configuration changes, upgrades and fixes that affect in-scope controls.
Retainers can include periodic re-testing so operating evidence stays current between assessment dates.
Findings from testing are documented so management decides on remediation and reports them to its auditor.

Design vs operating effectiveness testing for SOX ITGCs