IoT penetration testing vs firmware analysis

IoT penetration testing vs firmware analysis: which one a device maker needs first

Firmware analysis examines the device image alone, while IoT penetration testing covers the whole connected system: hardware interfaces, firmware, the companion app, the cloud back end and the network paths between them. A device maker with a launch date should start by mapping which of those components it actually ships and operates, because that map decides the scope. A standalone firmware review is usually the right first step when the product is mostly a sealed device with a single update channel and the team needs early visibility into what is inside the image. Full penetration testing is the right choice when the app, the account system and the cloud API are part of what customers buy, which is the case for most connected products launching today.

The two approaches map onto different stages of the OWASP Firmware Security Testing Methodology (FSTM), which lists nine stages from information gathering and obtaining firmware through filesystem extraction, emulation, dynamic and runtime analysis and binary exploitation. Firmware work covers the early stages well, and it can run on a development image before hardware is frozen. The OWASP IoT Security Testing Guide (ISTG) takes a wider view, with an IoT device model and a catalog of test cases for each component, which is why its results are easier to compare across a product line. Confirm with the testing provider which FSTM stages and ISTG test cases are in the written scope, and ask how the team will reach the device if the hardware interfaces or the emulation work are not included.

Combining the two is common when the product has a real attack surface on several layers. A practical sequence is to run firmware analysis on the first release candidate to find hard-coded secrets and unsafe services, then run a full IoT assessment on the production build, once the app and cloud are stable. Regulatory expectations point the same way: ETSI EN 303 645 V3.1.3 (2024-09) groups its provisions around software integrity, exposed attack surfaces and input validation, which span the firmware and the services around it. The European Commission's CRA FAQ also lists firmware analysis and penetration testing among the additional tests a manufacturer may need, so ask early whether the schedule can absorb both before the launch date.

How the two approaches compare on scope, method and cost drivers

This table compares a full connected-device penetration test with analysis of the firmware image alone on goal, access, method, findings, gaps and deliverables. Read each row against your device's architecture and launch timeline, since the right choice depends on which surfaces ship with the product.

Factor	IoT penetration test (hardware, firmware, apps, cloud)	Firmware analysis alone
Primary goal	Show that an attacker cannot compromise the product as shipped, across every interface it exposes	Find weaknesses inside the firmware image, such as hardcoded credentials, outdated components and unsafe services
Scope	Device hardware, firmware, companion app, cloud back end and update channel	One or more firmware images and their extracted filesystem
Access required	Physical device, test accounts, app builds and a permitted cloud test environment	A firmware image obtained from the vendor, a release package or a public download
Method at a buyer level	Stage-based test cases from the OWASP ISTG device model, with hardware and runtime work added	Stages from the OWASP FSTM, from obtaining and extracting firmware through emulation and binary analysis
Typical findings	Exposed debug ports, weak pairing, insecure app-to-device traffic, cloud authorization flaws, update weaknesses	Default or embedded secrets, vulnerable libraries, unnecessary services, unsigned or weakly protected images
What it misses	Full code coverage of every binary, unless the firmware analysis is also scoped	Runtime behavior, the app, the cloud service and physical attacks on the device
Deliverables to expect	Test cases mapped to the ISTG model, reproduction steps, severity ratings and retest results	Findings tied to image versions, extracted filesystem evidence and a list of affected components
Duration drivers	Number of interfaces, app platforms, cloud components and test accounts provided to the team	Number of firmware images, filesystem complexity and whether emulation is achievable
Fit with ETSI EN 303 645	Supports evidence for provisions on default passwords, attack surface and software integrity when the full product is in scope	Supports checks on default passwords and software integrity, but not on the wider attack surface
Fit with CRA expectations	Matches the CRA FAQ examples of penetration testing and network traffic analysis when the product is in scope	Matches the CRA FAQ example of firmware analysis for components integrated into a product
Choose when	A device is shipping or a connected product needs proof of its full attack surface before launch	A vendor needs a fast review of a component image or a supplier firmware before integration

What firmware analysis finds that a live test may miss, and the reverse

Firmware analysis and live IoT testing look at the same device from different angles, so each one misses things the other catches. The cards below pair what each method tends to surface with what it tends to overlook, so a device maker can decide which gaps to close before launch.

Hardcoded keys and update flaws

A firmware image often holds hardcoded credentials, certificates, private keys and update logic that trusts unsigned or weakly verified packages. Static analysis of the extracted filesystem finds these without needing the device to be on a network. A live test may miss them if the secret is only used during boot, behind a debug path, or in a code branch the tester never triggers. Check whether the update client verifies signatures and whether the key store is protected at runtime.

Runtime services and open interfaces

A live test sees what the device actually exposes: open ports, web admin panels, UPnP, Bluetooth and MQTT listeners, and the behavior of each service under malformed input. Firmware analysis can list the binaries and init scripts that might start a service, but it cannot prove one is enabled in the shipped configuration. Confirm the running state on a device, and compare it with the default configuration and the ETSI provision 5.6 Minimize exposed attack surfaces.

App and cloud API authorization

Companion apps and cloud APIs often enforce access rules that the device never checks. A tester working through the mobile app and backend can find broken object-level authorization, where one account reads another user's device data. Firmware analysis of the device alone rarely reaches these flaws, because the authorization decision happens on a server the image does not contain. Include the app and cloud in scope, and ask whether the vendor's own test plan covers account-to-account access.

Hardware debug and side-channel exposure

Exposed JTAG, UART or SWD pads can hand an attacker a root shell or memory dump, and side-channel leakage can reveal keys during cryptographic operations. These issues depend on the physical board, so a remote live test often misses them, and a firmware image cannot show whether a debug header is populated or fused off. Inspect production units for debug access, confirm that debug interfaces are disabled, and treat side-channel analysis as a separate test that the CRA FAQ lists among possible additional security tests.

Software integrity and provision gaps

ETSI EN 303 645 provision 5.7 Ensure software integrity asks whether the device verifies the code it runs, and a firmware review can check signing, secure boot and rollback protection in the image. A live test may only confirm that an update is accepted or refused. Provision 5.6 Minimize exposed attack surfaces is similar: firmware review shows what could run, while live testing shows what does. Map each provision to the evidence from both methods, and document any gap that neither method closes.

How the FSTM stages and ISTG test areas map onto one engagement

This sequence shows where a firmware image analysis stops and a full IoT penetration test continues, so buyers can see which activities each scope buys. The exact order depends on the scope agreed in writing before work starts.

Agree scope and gather information
The client and the tester fix the target device models, the components in scope (hardware, firmware, companion app, cloud back end and network interfaces), and the rules of engagement. The tester then collects public documentation, vendor release notes, certification records and any prior test reports, mapping the FSTM information gathering stage and the ISTG device model to the agreed boundaries. This step is done when the scope document lists every component to be tested and every component explicitly excluded. Firmware-only analysis can stop at firmware and its update mechanism, while a full IoT penetration test covers all components.
Obtain and analyze the firmware image
The tester obtains firmware through the vendor's update channel, a client-supplied image or extraction from the device, and records its source and version. Next they identify the format, headers, embedded bootloader and signing or encryption indicators, which covers the FSTM obtaining and analyzing firmware stages. This step closes when the image hash, source and identified format are documented. Firmware-only work starts and often ends here, while a full engagement uses the image as an input to later hardware and runtime work.
Extract and analyze the filesystem
The tester unpacks the firmware into its filesystem and reviews contents for hard-coded credentials, private keys, certificates, default accounts, outdated libraries and unnecessary services. This maps to the FSTM extracting and analyzing filesystem stages and to ISTG test cases on software and credentials. The step is complete when each finding is logged with its file path and a short validation note. This is a core activity in firmware analysis alone, and a full engagement uses the same findings to direct live testing.
Emulate the firmware
Where the architecture allows, the tester runs the filesystem in an emulated environment so services can be started and probed without the physical device. Emulation may fail for hardware-dependent components, and the tester records which services did and did not start. This FSTM stage supports dynamic analysis and is usually a bridge between the two scopes. It is done when the emulation outcome and its gaps are written down, so the client knows which results still need hardware validation.
Run dynamic and runtime analysis
The tester exercises running services, network interfaces and update paths on the emulated or physical device, observing behavior under crafted input. This covers the FSTM dynamic and runtime analysis stages and the ISTG network, interface and software test areas. It is performed on a physical device or a lab replica under agreed conditions. The step closes when every service or interface in scope has a recorded outcome. Only a full IoT penetration test normally includes live device and network testing.
Test hardware and companion app paths
Where hardware and the mobile or web app are in scope, the tester examines debug interfaces, storage, pairing, authentication between app and device, and cloud API calls. This corresponds to ISTG test areas for hardware, mobile and cloud components. Work needs physical access to the device and the app's test builds, both fixed in the scope document. The step is done when each interface and API has a pass, fail or not-tested status with evidence. Firmware analysis alone does not reach this step.
Perform binary exploitation if in scope
If the agreed scope includes exploitation, the tester attempts to turn validated memory-safety or logic weaknesses in binaries into controlled effects, within the rules of engagement. This FSTM stage carries the highest risk to the device, so the client should confirm in writing whether exploitation is allowed and on which units. The step closes with a reproducible proof of concept or a recorded reason it could not be reproduced. Many engagements exclude this step, and the scope should say so.
Document results against the ISTG and FSTM coverage map
The tester consolidates findings, maps each test to its ISTG test case or FSTM stage, and states which areas were not covered and why. The report should let the client confirm that the chosen scope matches what was tested and which ETSI EN 303 645 provisions the results inform. This step is complete when the coverage map and findings are reviewed with the client and any unreachable areas are listed for a decision.

What ETSI EN 303 645 and the CRA ask for before launch

ETSI EN 303 645 V3.1.3 (2024-09) is the baseline standard for consumer IoT security, organized into 13 provision groups that run from 5.1 No universal default passwords to 5.13 Validate input data. For a device maker, the useful question is which provisions a firmware image can prove. Firmware analysis can evidence provisions such as 5.1 (no universal default passwords hardcoded in the image), 5.6 Minimize exposed attack surfaces (open services and debug interfaces visible in the filesystem and binaries) and 5.7 Ensure software integrity (signed update mechanisms and verification logic). It cannot, on its own, show how the device behaves once running, how the companion app handles accounts, or how the vendor cloud protects data. Those need runtime, app and cloud testing, so plan them as separate evidence rather than assuming the firmware review covers them.

The European Commission's CRA FAQ lists additional security tests a manufacturer may use when integrating components, including fuzz testing, penetration testing, firmware analysis, side-channel analysis, red-team exercises and network traffic analysis. The list is illustrative of methods, not a checklist that every product must pass. Map each method to a risk in your own threat model and record why the chosen tests are sufficient for the product's exposure. A practical first step is to ask which test types your notified body, if one is involved, expects to see as evidence, and to confirm the scope of any conformity route with legal counsel before committing budget to a specific test set.

Before launch, confirm with your notified body or legal counsel which obligations apply to your product, its category and its market, because the CRA's requirements and timing depend on the product's classification and the conformity route that applies. Ask them to state in writing which ETSI provisions they accept as evidence, whether a firmware-only review is acceptable for any of them, and what documentation they will review. Ask your test provider to map each finding and each test to a named provision or CRA requirement so the evidence file can be assembled without rework.

Which approach fits a startup, an established vendor or an integrator

A first-generation device with a companion app and a vendor cloud usually has its largest unknowns in the places a firmware image cannot show: the radio interfaces, the debug ports on the board, the mobile app's handling of credentials and the cloud API the device calls. For this situation, the usual starting point is a full IoT penetration test that follows the OWASP IoT Security Testing Guide, because it covers the device, app and cloud together and gives the maker a first view of exposure across all of them. A firmware analysis alone would miss the pairing flow and the cloud authorization logic that often matter most.

Test units: Supply several production-equivalent units, including one that can be reset to factory state and one that can be opened for hardware access.
App builds: Provide the exact mobile app versions that ship to users, with a staging account and a note of any debug flags.
Cloud staging access: Give the tester a staging tenant that mirrors production APIs, with agreed rules for test accounts and rate limits.
Firmware images: Share the release image for each supported hardware revision so the tester can compare what the device runs with what the app expects.
Add firmware analysis later: Once the first release is stable, a firmware-focused review helps with update signing and secret storage before the next hardware revision.

Before booking, confirm with the test provider whether its scope covers the radio interfaces and the cloud API, since many first-release gaps sit there.

What you receive from each approach and what it proves to an assessor

A penetration test and a firmware analysis produce different evidence, and neither produces a certificate. The cards below cover what lands on your desk from each route, how the output maps to ETSI EN 303 645 and CRA expectations, and what support follows the findings.

Penetration test findings with reproduction steps

A device penetration test should return findings that name the component tested (hardware interface, firmware, companion app, cloud endpoint), the exact steps that reproduced each issue, and the evidence captured, such as requests, captured traffic or a serial console log. For an assessor, reproducibility matters more than severity labels alone. Check whether each finding references the test case it came from, so the work can be compared with the OWASP IoT Security Testing Guide model and repeated after a fix.

Firmware analysis report with extracted components

A firmware analysis report should list what was extracted from the image: the filesystem layout, embedded binaries, libraries and their versions, hard-coded credentials, certificates and init scripts that start network services. It should also flag unsafe findings such as weak default configuration or outdated components. Ask whether the report follows the OWASP Firmware Security Testing Methodology stages, and whether the extraction and emulation steps can be rerun by your own team.

Retest and remediation support

Findings only reduce risk once fixed and verified. Confirm whether the engagement includes a retest of closed findings, how long the tester keeps the device, firmware images and test artifacts, and who approves a fix for a shipped product. Firmware fixes often require a new signed image, so ask how the update path is tested before the retest. Retests should be written into the fixed scope, with deliverables and timing agreed before work starts.

Mapping to ETSI provisions and CRA evidence

A useful mapping traces each finding or extracted component back to the relevant ETSI EN 303 645 provision, such as the one on minimizing exposed attack surfaces or ensuring software integrity, and to the security tests the Commission's CRA FAQ lists for integrated components. Ask the provider to show this mapping in the report and to state which provisions were not tested. The mapping is evidence for your own technical file, not a conformity finding.

What no approach certifies

Neither a penetration test nor a firmware analysis issues a certificate or a conformity declaration. The report is input to a conformity decision made by the manufacturer and, where a scheme requires it, by an independent body. Ask the provider to state this in the statement of work, and confirm with the certification body or notified body which evidence it expects before the test starts.

How much does IoT penetration testing cost compared with firmware analysis?

Is firmware analysis alone enough before a connected device launches?

Usually not on its own. Firmware analysis examines the image a device runs, so it can find hardcoded credentials, unsafe services and outdated components inside that image. It does not show how the device behaves on the network, how its companion app handles authentication, or how its cloud back end exposes data. The OWASP Firmware Security Testing Methodology covers stages from obtaining firmware through emulation and runtime analysis, while the OWASP IoT Security Testing Guide covers the wider device model, including hardware, apps and cloud. Confirm with the test provider which components their scope covers, and compare that list against the device's actual attack surface.

What does an IoT penetration test need from the device maker?

The work depends on access, so expect to supply the following before the test starts:

Firmware images for each model and version in scope, plus the update package used to deliver them
Test devices, including spares if destructive hardware work is planned, and any debug access the maker is willing to provide
Current builds of the companion mobile or web apps, with test accounts
Cloud or back-end access, such as staging environments, API documentation and test tenants, under written rules of engagement

Firmware analysis needs far less: mainly the images and, where relevant, documentation. Ask the provider to list every input in the written scope so that delays do not surface mid-engagement.

What drives the cost of each approach?

Cost follows the number of components the test covers, not just the size of the firmware. Penetration testing across hardware, firmware, apps and cloud involves more specialists, more test devices and more environments, and it adds time for setup and reporting. Firmware analysis is narrower, so its effort tracks the number of images, the complexity of the filesystem and how much emulation or binary work is needed. Both are usually fixed-scope engagements priced from a written scope. Ask each provider for an itemized quote showing the components, the number of firmware versions, the devices supplied and whether retesting after fixes is included. DeployOpen does not publish a price list; a quote follows from the agreed scope.

How long does each approach take, and what changes the timing?

No supplied source gives a duration for either approach, so confirm timing with each provider against the written scope. The main timing drivers are the number of devices and firmware versions, how quickly the maker delivers images, app builds and test accounts, whether hardware work is needed, and how many findings require retesting. Access delays are often the largest source of slippage, so agree on delivery dates for each input in advance. Ask for milestones that separate setup, testing, reporting and retest.

Can an in-house team run IoT penetration testing or firmware analysis?

Yes, if the team has the skills and the time. Firmware analysis is the easier of the two to bring in-house, since it can start from the images and a standard set of stages. Full penetration testing needs hardware, wireless, mobile and cloud skills, and independence matters: a team that built the product may miss what an outside tester would try. Many makers therefore keep firmware checks in their build pipeline and buy an external penetration test before launch. Ask any provider how they document the methodology and what evidence the report contains, so that internal and external results can be compared.

How should a maker combine both approaches across firmware updates?

Run firmware analysis on every release and use full penetration testing at major milestones, such as a new hardware revision, a new companion app or a significant change to the cloud service. The OWASP Web Security Testing Guide notes that no single technique is enough and recommends a mix, so the two approaches complement each other. Agree on a rule for which changes trigger a full test, and keep the findings register so that fixes can be verified in later releases. Scheduled retesting belongs in the initial contract, because a one-off test does not cover later firmware.

What should a maker confirm about CRA obligations and ETSI coverage?

The European Commission's Cyber Resilience Act FAQ lists fuzz testing, penetration testing, firmware analysis, side-channel analysis, red-team exercises and network traffic analysis as possible additional tests when integrating components, so the choice of tests is not fixed by a single method. ETSI EN 303 645 V3.1.3 (2024-09) sets 13 provision groups for consumer IoT, including minimizing exposed attack surfaces and ensuring software integrity, and a test plan can map to those provisions. Confirm with a legal or compliance adviser which obligations apply to the product, which conformity route applies and what evidence the regulator expects. DeployOpen prepares controls and evidence for assessments but does not issue certifications or attestations; independent bodies do.

How to combine firmware analysis and penetration testing across releases

Combine the two methods on purpose, not as a one-off choice. NIST SP 800-115 notes that organizations often combine assessment techniques to reach an in-depth assessment at an acceptable level of risk, and the OWASP Web Security Testing Guide makes the same point from the development side: no single technique is enough, and the mix should carry more weight in the early development phases. For a connected device, that means firmware analysis and penetration testing answer different questions at different points in the product lifecycle, and a maker who runs only one of them before launch leaves a gap.

A practical sequence starts with firmware analysis on early builds, while defects are still cheap to fix. Use the OWASP Firmware Security Testing Methodology to check the image itself: hard-coded credentials, unnecessary services, weak update signing and exposed debug interfaces. Then run a full penetration test against a release candidate that includes the real hardware, mobile app and cloud back end, using the OWASP IoT Security Testing Guide so results can be compared across test rounds. Before that engagement, confirm in the written scope whether the tester may use the device's debug ports and whether the test environment matches production configuration.

After launch, repeat firmware analysis for every significant firmware release, and schedule a fresh penetration test when the device's network exposure, authentication model or cloud interfaces change. Ask the provider to state which findings were reproduced on the new image and which were only confirmed by static analysis, because that distinction drives the retest budget. Keep the firmware analysis reports, build hashes and penetration test retest results together, since a regulator or auditor reviewing your EN 303 645 or CRA evidence will ask which image each finding applied to.

How DeployOpen scopes and delivers IoT security testing

DeployOpen plans and runs IoT security testing as a fixed-scope engagement agreed in writing before work starts. The scope states which device components, firmware images, apps and cloud services are in and out, so the method (full device testing, firmware analysis or both) matches what the device maker needs before launch.

How we scope

Written scope names the device components, firmware versions, companion apps and cloud back ends in and out of testing
Scope states whether testing covers hardware, firmware, apps and cloud, or firmware image analysis alone
Deliverables, timelines and rules of engagement are agreed in writing before any device or firmware is touched

How we build

Test cases are organized against the OWASP IoT Security Testing Guide device model so results are comparable across components
Firmware analysis follows the OWASP Firmware Security Testing Methodology stages agreed in scope
Provisions of ETSI EN 303 645 are mapped to test evidence where the client's market requires that baseline

How we hand over

Findings report documents method, tested components, evidence and reproduction steps for each issue
Mapping to ETSI EN 303 645 provisions and CRA-related evidence is delivered as preparation material, not as certification
Handover includes a walkthrough so the client's engineers can rerun key tests after fixes

How we support

Ongoing support, firmware upgrades and continuous testing are available as a retainer after the initial engagement
Retesting after fixes is scoped in writing, so the client knows which findings are verified and which remain open
Clients should confirm with DeployOpen which compliance evidence an independent auditor or certification body will need

IoT penetration testing vs firmware analysis