Hardening review vs penetration testing for self-hosted platforms

A hardening review checks your Keycloak, GitLab, Nextcloud or Odoo settings against vendor guidance, while a penetration test tries to break in. Here is when each one fits and how to sequence them.

Book a callTalk to us
On this page

Hardening review vs penetration testing: which one to commission first for a self-hosted platform

For most new or inherited self-hosted Keycloak, GitLab, Nextcloud or Odoo deployments, commission a hardening review first and a penetration test after the settings have been reviewed and corrected. A penetration test against a platform that still runs default administrative paths, public REST endpoints or weak TLS mostly rediscovers known configuration gaps, and it spends the tester's time on problems your team could have closed beforehand. NIST SP 800-115 describes system configuration review as comparing settings against security configuration guides or checklists, and it prefers automated checks where feasible because they are fast and repeatable. Start from each vendor's own guidance: Keycloak's production configuration guide, GitLab's hardening recommendations, Nextcloud's admin manual hardening section and Odoo's guidance on running behind a TLS-terminating web server.

A hardening review answers a different question than an attack. It shows whether the platform is configured as the vendor intends, for example whether all Keycloak traffic uses TLS, whether the Administration REST API and Console sit on a separate hostname or context path, and whether non-public REST APIs are blocked at the reverse proxy. It cannot show whether a chain of individually acceptable settings, a weak trust relationship with an identity provider or an authorization flaw in a custom Odoo module can be exploited. That is the work of a penetration test, which uses real exploits and carries more operational risk, so scope it, schedule it and agree rules of engagement in writing first. NIST SP 800-115 notes that organizations often combine techniques to reach an in-depth assessment at an acceptable level of risk, and that is the usual answer here.

Commission a hardening review alone when the deployment is new, the team is still building its baseline and no custom code or integrations are exposed. Add a penetration test when the platform handles regulated data, faces the internet, or carries custom extensions, and confirm with the platform owner how it will receive security fixes: GitLab backports fixes to the previous two monthly releases and always ships high and critical fixes as patch releases, while Nextcloud reports vulnerabilities through HackerOne. Ask any provider to confirm which findings come from configuration checks and which need exploitation, and to re-test after fixes are applied.

Hardening review and penetration test side by side

This table compares a configuration-led hardening review with a penetration test for a self-hosted Keycloak, GitLab, Nextcloud or Odoo deployment. Read each row as a question to put to your provider, and note that the two methods answer different questions and are usually combined.

DimensionHardening reviewPenetration test
GoalConfirm settings match vendor hardening guidance, such as Keycloak's production configuration guide or GitLab's hardening recommendationsFind out whether an attacker can get in, move laterally or reach data despite the settings in place
ScopePlatform configuration, reverse proxy rules, TLS, admin console exposure, CI/CD and OS settingsDefined hosts, endpoints, accounts and workflows agreed in a rules of engagement document
MethodComparison of settings against guidance or checklists, with automated checks where feasible, per NIST SP 800-115Manual exploitation attempts against the running system, with automated tools as support, per PTES phases
Access neededRead access to configuration files, admin settings and hosting layer; typically no exploit attemptsTest accounts at agreed privilege levels, network access to the target, and written authorization for exploit activity
Risk to the running systemMinimal, as review techniques are passive according to NIST SP 800-115Higher, because exploits run against production systems and data; confirm whether a staging copy is available
What it provesWhich settings deviate from the reference guidance, and which ones are still at defaultsWhich weaknesses are exploitable in practice, including business logic flaws that settings checks cannot show
Typical outputChecklist-based findings mapped to settings, with remediation steps per configuration keyNarrative report with validated attack paths, evidence and retest recommendations
Fit with complianceSupports hardening evidence, but auditors may also ask for an independent penetration test under the relevant frameworkMatches requirements that ask for penetration testing, though CIS Safeguard 18.4 also expects detections to be validated afterward
Effort driversNumber of components, custom configuration, and access to the team that owns each setting; confirm the scope in the written quoteNumber of user roles, integrations, environments and authentication paths; confirm the test window and retest rules in the written quote

What a hardening review finds and misses, and what a penetration test finds and misses

A hardening review compares a deployed platform's settings with vendor guidance and checklists. A penetration test tries to break in. The six areas below show where each method earns its cost, and where each one leaves gaps that the other must cover.

Configuration drift from vendor guidance

A hardening review compares live settings with vendor production guidance. For Keycloak, that means confirming that all traffic uses TLS and that the Administration REST API and Console sit on a separate hostname or context path. Non-public REST endpoints should be blocked at the reverse proxy. Drift happens when an upgrade, a new proxy rule or a quick fix reopens a path nobody re-checked. Ask the reviewer to show the proxy configuration itself, not a summary. A tester may reach the same endpoint, but a review can prove the rule exists and covers every route.

Default and insecure application settings

Self-hosted platforms ship with defaults that suit a trial but not a regulated production deployment. GitLab's hardening guidance covers application settings, CI/CD settings, the gitlab.rb configuration file and operating system settings. NIST SP 800-115 describes configuration review as comparing settings with checklists, and automated checks are preferred where feasible because they are fast and repeatable. Ask the review to cover every layer, including the host. A penetration test will rarely confirm that a CI/CD default is wrong, because the flaw does not show until someone reads the setting.

Patch currency gaps

Knowing whether a deployment runs a fixed version is a version and inventory question, not an attack. GitLab backports security fixes to the previous two monthly releases in addition to the current stable release, and high and critical issues always get a patch release. A deployment that falls outside that window is exposed to known issues that a tester may never re-exploit during a short engagement. Confirm the vendor's support window for each component, record the running version of every component, and set a process to apply patch releases. Nextcloud publishes security advisories that serve the same purpose.

Logic flaws and chained attacks

A configuration check cannot prove that a sequence of individually acceptable steps leads to a takeover. A tester might combine a low-privilege account, an exposed workflow and a permissive integration to reach an administrative function. Only an attacker-style test shows whether the chain works in your environment. Scope the test to include authenticated paths, realistic accounts and the integrations your platform actually uses. Ask the tester to report each step of every chain, so that your team can fix the earliest link rather than only the final symptom.

Authenticated business logic issues

CIS Safeguard 16.13 notes that for critical applications, authenticated penetration testing is better suited than code scanning and automated testing to finding business logic vulnerabilities. Examples include approval steps that a user can skip, record access that ignores tenant boundaries, or workflow transitions that run out of order. A hardening review will confirm the access control settings exist but cannot judge whether they match intended business rules. Provide the tester with test accounts at each role, a written description of expected workflows, and a list of approvals that must hold.

Blind spots of each method

A hardening review sees configuration and version data but not how an attacker would move through it, so it can miss exploitable paths that look correct on paper. A penetration test sees exploitable behavior at one point in time but cannot prove every setting is correct, and it cannot confirm that a fix survives the next upgrade. NIST SP 800-115 notes that organizations often combine techniques to reach an acceptable level of risk. Plan for both, and schedule the review to re-run after each major upgrade, while the test covers the attack paths that a checklist cannot describe.

How the review and the test differ for Keycloak, GitLab, Nextcloud and Odoo

A hardening review of Keycloak starts from the production configuration guide, which says all communication to and from Keycloak requires TLS and recommends exposing the Administration REST API and Console on a different hostname or context path. The reviewer checks whether those paths are separated, whether the reverse proxy blocks REST endpoints that are not meant to be public, and whether TLS is enforced end to end. A penetration test takes the opposite starting point: it maps which endpoints answer from the internet or from a user network, then tests whether authentication, session handling and realm settings hold up when a tester tries to reach administrative functions from the wrong side of the proxy.

  • Hostname separation: the review confirms the admin console and admin API sit on a separate hostname or path; the test confirms whether they answer from the public listener.
  • Proxy rules: the review reads the reverse proxy configuration against the endpoints the guide says should be blocked; the test probes what the proxy actually forwards.
  • TLS coverage: the review checks every hop, including internal ones; the test looks for plaintext fallbacks it can observe.
  • Realm and client settings: the review compares settings with the guide and your own policy; the test checks whether those settings can be bypassed in practice.
  • Version and patch level: confirm with the vendor which release line your deployment runs and how security fixes are delivered for it.

Before booking a test, confirm the Keycloak version and the exact proxy configuration in use, since both change what a tester can reach.

How vendor patch cadences change the testing calendar

Vendor patch cadence decides how often your hardening review has to run, and that cadence differs by platform. GitLab backports security fixes to the previous two monthly releases in addition to the current stable release, and it addresses high and critical security issues with a patch release. Nextcloud routes vulnerability reports through HackerOne and publishes security advisories, so the security feed is the signal to watch. Odoo's security page recommends on-premise instances sit behind a web server such as nginx for encryption and request proxying, which means the proxy configuration is part of what you must re-check after each platform update. Confirm the current release and advisory channels for each product you run before you set a schedule.

A configuration review is tied to a version and a settings baseline, so it has to be repeated after every upgrade that changes defaults, adds a module or alters a reverse proxy rule. In practice, a Keycloak deployment should be re-checked when hostname or proxy settings change, and a GitLab instance should be re-checked when gitlab.rb or CI/CD settings move. A penetration test is a point-in-time exercise against the system as it stood on the test dates, so its findings age as soon as a patch lands. That makes the two methods complementary rather than interchangeable, and the calendar should show configuration checks tied to change events and tests tied to a fixed planning cycle.

NIST SP 800-115 suggests regularly scheduled network and vulnerability scanning, interspersed with periodic penetration testing. For a self-hosted platform, that suggests a rhythm where scanning and configuration checks follow each vendor release, while a penetration test is scheduled after a significant upgrade or architecture change. Ask your DeployOpen team, or any provider, to agree in writing which triggers start a re-review, who receives vendor advisories, and how a high or critical patch is tested before it reaches production. This turns vendor cadence from a surprise into a planned input.

How to sequence the work on a new deployment

The sequence turns a fresh self-hosted platform into one that has been checked against vendor guidance, attacked in its hardened state, and set up to stay checked after every upgrade. Each step closes with a record that the next step depends on.

  1. Fix scope and rules of engagement

    The client and DeployOpen agree in writing which platforms, hostnames, environments and accounts are in scope, which techniques are allowed, who may be contacted during testing, and how testers stop if production is affected. NIST SP 800-115 provides a rules of engagement template in Appendix B that teams can adapt. The step is done when the signed scope lists each platform version, its exposure (internal or internet-facing), and the named contacts for testing windows and escalation.

  2. Run the configuration review

    The reviewer compares the live settings of Keycloak, GitLab, Nextcloud or Odoo against the vendor's hardening guidance and a recognized benchmark where one exists. Keycloak's production configuration guide, GitLab's hardening recommendations and Odoo's on-premise guidance all serve as baselines. Automated checks are preferred where they are repeatable, and the reviewer reads the results by hand for anything the checks cannot judge. The step closes with a findings register that cites each setting, its expected value and the observed value.

  3. Remediate and record evidence

    The platform owner fixes the findings, and each fix is recorded with the change made, the approver and the date. Items that cannot be fixed are logged as accepted risks with an owner and a review point. Evidence includes configuration exports, reverse proxy rules that block unintended public API paths, and TLS settings. The step is done when the reviewer re-checks every high-priority item and confirms the evidence matches the live system.

  4. Run penetration testing on the hardened environment

    Testers attack the platform after the configuration review and remediation are complete, using the scope and rules of engagement from the first step. The test covers authentication, authorization, business logic and exposed interfaces that a settings check cannot judge, which matches the point CIS Controls v8.1 Safeguard 16.13 makes about authenticated testing for business logic. The step closes with a report that separates confirmed exploits from observations and lists the evidence for each.

  5. Retest and close the loop

    The client and DeployOpen fix the penetration test findings and the testers verify each fix against the original attack path. Where a fix changes configuration, the reviewer also re-runs the affected checks so the register and the test results agree. Findings that remain open are carried into the next cycle with a documented reason. The step is done when every finding has a status of fixed, accepted or open, with the verification noted.

  6. Confirm detection coverage

    The client's security team checks whether its monitoring logged the techniques the testers used. CIS Controls v8.1 Safeguard 18.4 asks for this validation after each test. Gaps are turned into detection or logging tasks in the monitoring backlog. The step closes when the team can show which test activities generated alerts and which did not, and what it changed as a result.

  7. Schedule recurring review after upgrades

    The client sets a recurring schedule for configuration review and penetration testing, and ties a configuration review to each platform upgrade. Vendor patch cadences shape the trigger, so the owner checks GitLab's maintenance policy or Nextcloud's security advisories when planning. NIST SP 800-115 suggests periodic penetration testing combined with regularly scheduled scanning. The step is done when the calendar, owners and triggers are recorded in the operations runbook.

What you receive and what drives cost for each approach

A hardening review and a penetration test produce different evidence and cost different amounts of effort for different reasons. The cards below set out what lands on your desk, who is paid for each part, and which choices move the quote up or down.

Configuration findings mapped to vendor guidance

A hardening review compares the live settings of Keycloak, GitLab, Nextcloud or Odoo against the vendor's own hardening material and against any relevant checklist. Each finding should name the setting, the observed value, the guidance it departs from and the proposed change. Ask for a finding list that shows the evidence for each setting, such as the file, the configuration key or the screen captured, so your team can verify it without repeating the work.

Penetration test report with validated findings

A penetration test report records only what the tester could actually exploit or chain, with the access level used, the steps taken and the impact shown. NIST SP 800-115 describes penetration testing as using real exploits against production systems and data, which is why it carries more risk and more planning than a review. Ask who wrote each finding, which accounts were used and whether any test ran against production data.

Remediation evidence and retest letter

Remediation evidence shows that a fix was applied, such as a changed gitlab.rb setting, a reverse proxy rule or a patched release. A retest confirms whether a validated finding is closed and states what was and was not retested. Ask whether the retest is included in the fixed-scope agreement, how many retest rounds are covered, and what the letter says about residual risk. Retest rounds are a common source of unplanned cost.

Who is paid for what

In a fixed-scope engagement, DeployOpen's fee covers planning, testing, reporting and any agreed retest, based on the written scope. Your internal team pays in time for access approvals, change windows, staging copies and staff who answer questions during testing. Any third-party tool licenses, cloud resources for test environments and the cost of your own remediation are separate lines. Ask the quote to show each of these as its own item.

Scope size, platforms and authenticated roles

The main cost drivers are the number of platforms, the number of user roles tested with credentials, and the depth of the configuration review. A single Nextcloud instance with one admin role costs less to assess than a Keycloak realm, a GitLab instance with CI/CD runners and an Odoo deployment tested across administrator, finance and portal roles. Ask the quote to list each platform, each role and each environment in scope, with the reasons any are excluded.

One-time work versus recurring work

A hardening review and a penetration test are usually one-time assessments, but the configuration baseline and upgrade checks recur with each vendor release. GitLab's maintenance policy backports security fixes to the previous two monthly releases as well as the current stable release, so a baseline written today needs refreshing as versions move. Ask whether the quote covers only the assessment or also a repeat review after upgrades, and whether a retainer is needed for continuous checks.

Questions buyers ask before choosing a hardening review or a penetration test

Does a hardening review alone satisfy a penetration testing requirement?

Usually not, if the requirement is a penetration test. PCI SSC's Penetration Testing Guidance (v1.1, September 2017) describes penetration testing as essentially a manual process and says that simply running an automated tool does not satisfy the penetration testing requirement. A configuration review compares settings with vendor guidance, which answers a different question: whether the platform is configured as documented. NIST SP 800-115 treats review techniques as passive and low risk, while penetration testing uses real exploits, so the two produce different evidence. Ask your assessor which one the requirement names before you buy either.

How do the CIS annual penetration test safeguards affect planning?

CIS Controls v8.1 Safeguards 18.2 and 18.5 ask for external and internal penetration tests at least annually, so a hardening review cannot replace them on that schedule. Safeguard 18.4 asks that security measures be validated after each test so that detections cover the techniques the testers used. Plan the test date, then schedule the detection validation that follows it. Safeguard 16.13 adds that authenticated penetration testing is better suited than automated scanning for finding business logic flaws in critical applications, which matters for Odoo workflows or GitLab permission models.

How much work does our team have to provide?

Expect to provide accounts, access paths, and decisions, not only time. For a hardening review, you supply the deployed configuration files, reverse proxy settings, identity realm settings, and the version and patch records for each platform. For a penetration test, you add a written rules of engagement, named contacts, test accounts at each role level, and an agreed window for any load or disruptive work. NIST SP 800-115 provides a rules of engagement template in Appendix B that many teams adapt. The more of this is ready before kickoff, the less time your team pays to wait on access.

Can our own team run the hardening review internally?

Yes, for the configuration review, if the team knows the platform well and has a checklist to compare against. NIST SP 800-115 describes system configuration review as comparing settings with security configuration guides or checklists, and it says automated checks are preferable where feasible because they are fast and repeatable. Keycloak's production guide, GitLab's hardening recommendations, Nextcloud's hardening section and Odoo's on-premise guidance give the baseline. Internal teams often struggle with independence, because the people who built the system also judge it. A penetration test by a separate team is usually the better fit for that second opinion, and some frameworks ask for an independent tester.

How long does each one take, and what drives the duration?

No duration figure is supplied here, so treat any number from a vendor as a claim to confirm against written scope. A hardening review scales with the number of components (identity realm, GitLab runners, Nextcloud apps, Odoo modules), how many configuration sources exist, and whether checks are automated. A penetration test scales with the number of applications and roles, the depth of authenticated testing, the number of retest cycles, and the time needed to agree rules of engagement. Ask for the duration as part of a fixed-scope written plan, which DeployOpen agrees before work starts, and ask what would move it.

How do the two fit with DORA Article 25 and NIST SA-11 or CA-8?

They sit in different places in the same program. DORA Article 25(1) lists several test types, including vulnerability assessments and scans, open source analyses, source code reviews where feasible, scenario-based tests and penetration testing. Article 26 requires threat-led penetration testing at least every 3 years for entities identified by their competent authority. NIST SP 800-53 Rev. 5 control SA-11 Developer Testing and Evaluation includes static and dynamic analysis, manual code reviews and penetration testing as enhancements, while CA-8 Penetration Testing applies to the high baseline in SP 800-53B Release 5.2.0. A practical sequence is configuration review first to remove known gaps, then penetration testing to check what remains. Confirm with your competent authority or assessor which tests your obligation names.

Which order does DeployOpen recommend for a new self-hosted deployment?

For a new deployment, we recommend a configuration review against the vendor's hardening guidance before the first penetration test, so the tester spends time on design and logic flaws rather than known defaults. OWASP's Web Security Testing Guide v4.2 (released 3 December 2020) says no single technique is enough and recommends a mix with more weight on early phases. Once the platform is in production, schedule penetration testing on your required cadence and repeat the configuration review after each major upgrade. DeployOpen plans, implements and documents these steps inside your own infrastructure, and ongoing testing can run under a retainer.

How the two approaches combine over time

No single technique is enough. NIST SP 800-115 says organizations often combine techniques to reach an in-depth assessment at an acceptable level of risk, and the OWASP Web Security Testing Guide v4.2 makes the same point, recommending a mix that puts more weight on early development phases. For a self-hosted Keycloak, GitLab, Nextcloud or Odoo deployment, that means the configuration review and the penetration test answer different questions and should both run, in a sequence that fits your change cycle. Confirm with your assessor which of these techniques they will count toward your own control requirements before you plan the calendar.

Validate detections after every test. CIS Controls v8.1 Safeguard 18.4 asks that security measures be validated after each test so that detections cover the techniques the testers used. Ask your testing provider to log the technique used for each finding, and check that your monitoring stack, whether it is a SIEM or host-based tooling, alerted on the access paths the tester took, such as an exposed Administration REST API on Keycloak that was not blocked at the reverse proxy. A finding that goes unnoticed in production is a detection gap, and it should be tracked separately from the vulnerability itself.

A hardening baseline makes testing more productive. Illustrative example only: suppose a team first checks its Odoo instance against the nginx and encryption guidance in Odoo's security documentation, fixes the exposed database manager and default settings, and then commissions a penetration test. The tester spends the engagement on business logic, role boundaries and authenticated paths instead of re-finding known misconfigurations, which is the kind of work CIS Safeguard 16.13 says authenticated testing is better suited to. Each platform release should trigger a targeted recheck of the baseline, since GitLab's maintenance policy describes patch releases for high and critical issues that can change settings.

How DeployOpen scopes, delivers and supports hardening reviews and penetration tests

DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks, and runs security testing alongside that work. For a self-hosted Keycloak, GitLab, Nextcloud or Odoo deployment, we agree in writing which configuration review and which attack-based testing fit your risk, then deliver each against that written scope.

How we scope

  • Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
  • The written scope names the platform versions, hostnames, environments and configuration areas that a hardening review covers.
  • Penetration test scope sets the authenticated access, target layers and rules of engagement, and states which production actions are permitted.

How we build

  • We compare the deployed settings with the vendor hardening guidance for each platform, recording each deviation and its reason.
  • Attack-based testing follows a published methodology agreed at scoping, so results can be reproduced and compared across test cycles.
  • Where a platform needs changes, we implement them in your environment and retest the affected controls before closing the item.

How we hand over

  • You receive written findings that separate configuration deviations from exploitable weaknesses, with evidence and suggested fixes for each.
  • Documentation covers the configuration baseline, the test results and the remediation status, in a form your auditors or assessors can read.
  • We confirm with you which evidence your independent auditor or certification body needs, since DeployOpen prepares evidence but does not issue attestations.

How we support

  • Ongoing support, platform upgrades and continuous testing are available as retainers, separate from the fixed-scope engagement.
  • A retainer can re-run the configuration review after each vendor release, so changes in patch cadence are checked against your baseline.
  • Ask us to confirm the retainer scope, response expectations and reporting cadence in the quote before you commit.

Sources and further reading

Related

Talk to our team.

Tell us what you're working on, whether it's a deployment, an audit, a security test or a cyber range. You'll speak with an engineer who can help you scope it.

  • 30-minute call: free, with no obligation.
  • NDA on request: we can sign before you share details.
  • Clear next steps: a scope and plan after the call.
Book a callhello@deployopen.com