Testing custom NiFi processors with nifi-mock

Why teams test custom NiFi processors with nifi-mock

Teams test custom NiFi processors with nifi-mock so that logic defects are caught on a developer machine or in a build pipeline before the processor is deployed to a shared cluster. A team that has written or inherited a custom processor usually needs to know three things: whether the processor validates its properties as intended, whether it routes FlowFiles to the correct relationships with the correct attributes and content, and whether it fails safely when something goes wrong. The TestRunner in nifi-mock answers those questions in an ordinary unit test. It lets the test enqueue FlowFiles, set property values, add and configure controller services, and then assert on the FlowFiles transferred to each relationship and on the messages the processor logged.

This practice is most useful when the processor contains branching logic, record parsing, attribute mapping or error routing, and when it will be changed or handed to someone else. Teams also use it to pin down behavior that a later NiFi upgrade could alter, since a test suite that runs in CI will show the difference quickly. The test code should mock the external systems the processor calls, such as an HTTP endpoint or a database client exposed through a controller service, so that the assertions focus on the processor's own behavior. Confirm the nifi-mock version that matches your target NiFi release before you start, because the test harness must align with the NiFi API your processor compiles against.

Unit tests with nifi-mock stop short of several things. They do not prove that a real external system accepts the data, that the processor performs under production volumes, or that it behaves correctly inside a NiFi cluster with node coordination, load balancing and provenance enabled. They also do not exercise the NAR packaging, the classloader arrangement or the way the component appears in the user interface. Plan separate integration tests against representative environments for those risks, and treat a passing unit suite as evidence about the processor's logic only.

What TestRunner lets you verify

TestRunner gives you a harness to drive a processor or controller service without a running NiFi instance. The cards below cover the capabilities that matter most when you write unit tests for a custom component, from feeding it input to checking what it sends downstream.

Enqueue FlowFiles with content and attributes

Each test starts by putting FlowFiles on the input queue, with content and attributes that match what the flow will really send. Use realistic-looking payloads for each branch you care about, such as valid records, malformed records and empty content. Check that attribute names match the expression language your processor reads, because a typo here passes silently in the test and fails in production.

Set and validate processor properties

Set each property the processor declares through the runner, then confirm the runner reports the configuration as valid or invalid as you expect. Test both the required-property failures and the cross-property rules, since those rules are where configuration mistakes usually hide. Record which property values the tests use so a later change to defaults does not quietly change what the tests cover.

Add and enable controller services

When a processor depends on a controller service, add the service to the runner, configure it, enable it and then link it to the processor's property. Confirm the service is enabled before the run, because a processor that references a disabled service fails validation. Test the processor against a fake or in-memory implementation of the service, not against a live external system.

Run the processor and assert relationships

Trigger the processor a set number of times and then assert how many FlowFiles reached each relationship, such as success, failure or original. Assertions on relationships catch routing bugs that content checks would miss. Also check that the processor does not route a FlowFile to a relationship that is not defined in its code.

Assert output content and attributes

Read the FlowFiles that landed on each relationship and compare their content and attributes with the expected values. Compare parsed structures where possible, not raw strings, so that harmless formatting differences do not break the test. Include attributes that downstream processors depend on, since a missing attribute is a common cause of failures several steps later.

Mock external systems behind the service

Replace calls to databases, HTTP APIs and message brokers with mocks or fakes so the unit test stays fast and repeatable. Decide which behavior each fake must reproduce, including error responses and timeouts, and test those paths explicitly. Confirm with your team which external behavior is covered by integration tests outside the unit suite.

How a processor test runs through TestRunner

A TestRunner is created for the processor class under test, and every later step is a call on that runner. The test sets properties with setProperty, enqueues input FlowFiles with enqueue (content plus attributes), and then calls run, which drives the processor through its normal trigger cycle against an in-memory session rather than a live flow. Controller services are attached before the run with addControllerService and enabled with enableControllerService, so the processor under test sees the same lookup path it will use in production. After run, the assertions work on what the framework recorded: FlowFiles routed to each relationship are retrieved by relationship name, their content and attributes are checked, and the runner's counts for input, output and penalized FlowFiles show whether anything was left behind. Confirm the exact method names and assertion helpers against the nifi-mock version you pin, because the API has changed across NiFi releases.

Session commit and rollback are visible only through what the test can observe, so a processor that throws mid-trigger must be tested for both outcomes. A committed session shows FlowFiles on their relationships; a rolled-back session leaves the input queued and typically produces no output, and a test that only checks the happy path will miss a processor that leaks FlowFiles or writes partial results before failing. Assert the queue counts after the run as well as the outputs. Validation and lifecycle behavior need explicit checks because the runner does not exercise them by default: call the runner's validation (for example, asserting that a missing required property or a custom validator failure makes the processor invalid) and test @OnScheduled and @OnStopped logic in their own tests, since a field initialized in @OnScheduled will not be set just because run succeeded. Treat a passing run as evidence of trigger logic only, and verify lifecycle and validation separately.

Choosing what to mock and what to test for real

Mocking an HTTP or database client is the cheapest way to test how a processor reads its properties, builds requests, routes FlowFiles to success or failure and writes attributes. In nifi-mock, you replace the controller service that wraps the client with a mock implementation that returns canned responses or raises chosen exceptions, then assert on the FlowFiles the processor emits. This fits logic-heavy processors, where the interesting bugs live in parsing responses, mapping records or handling error branches.

Logic coverage: Good for verifying branches such as retries, error relationships and attribute mapping when the client returns a given status or row set.
Not proven: Network timeouts, connection pool exhaustion, TLS handshakes and authentication behavior are not exercised, because the mock never opens a socket.
Driver quirks: Type conversions, character encoding, transaction semantics and SQL dialect differences stay unverified until a real database runs the query.
Maintenance cost: Mocks drift from the real client API after upgrades, so each change to the wrapped library needs a matching change to the test double.
Ownership: The team writing the processor owns the mock, and the mock's assumptions about responses should be copied from real captures, not invented.

Before relying on a mock, check which responses the real service actually returns for your error cases and confirm them against the vendor's API documentation.

Building the test suite in sequence

The sequence moves from confirming the target NiFi line and runtime to a CI job that blocks merges when a processor's behavior changes. Each step produces a artifact you can review before the next one starts, so failures point to a specific layer of the processor.

Confirm the target NiFi line and runtime
The processor developer checks which NiFi line the extension must run on, 1.x or 2.x, and the Java runtime that line requires. NiFi 2 uses Java 21 as its minimum runtime, and the test dependencies, API versions and build plugins must match that line. NiFi 1.x and 2.x differ in APIs and packaging, so a test setup written for one line will not reliably validate the other. The output is a short written note in the repository README stating the NiFi version, the Java version and the Maven profile used for testing. Confirm the exact version against the NiFi downloads page before pinning it.
Set up Maven test dependencies
The developer adds the nifi-mock test artifacts and a JUnit version compatible with the chosen NiFi line to the processor module's test scope, then pins the NiFi API version in the parent POM so the test and production code resolve the same API. Test-scoped dependencies should not leak into the NAR that ships to production. The output is a successful mvn test run on an empty test class that creates a TestRunner for the processor, which proves the classpath resolves. Confirm the artifact coordinates against the nifi-mock documentation for your version.
Test property validation first
The developer writes tests that set invalid and missing property values and assert that the processor reports the expected validation errors, and that valid combinations pass. Property validation is the cheapest layer to test and it catches configuration mistakes before any FlowFile is involved. Include tests for dependent properties, allowable values and expression language support where the processor uses them. The output is a test class where each property rule has at least one passing and one failing case, and the suite fails if a validator is removed.
Test happy-path FlowFile routing
The developer enqueues FlowFiles with representative attributes and content, runs the processor, and asserts that output lands on the success relationship with the expected content and attributes. Assertions should check attribute values and content, not only the count of files routed. Use small, named fixtures so a failing test shows which input caused the change. The output is a passing test class covering each supported input shape, and the team can read the expected output for each fixture from the test code alone.
Test failure relationships
The developer forces the processor into its error paths, such as malformed content, missing required attributes or a failing external call, and asserts that FlowFiles go to the failure or retry relationship with error attributes set as designed. Confirm that the processor does not drop FlowFiles silently and that penalized or yielded outcomes are asserted where the design depends on them. The output is a test class covering every relationship the processor declares, so an unrouted relationship will be visible in the test report.
Mock controller services and external systems
The developer adds mock controller services through the TestRunner for any connection pool, client or cache the processor uses, and replaces external endpoints with in-memory fakes or stubs. The mock contract is documented in the test package: which methods are stubbed, what each returns and which failure modes are simulated. This keeps tests deterministic and independent of a live database or API. The output is a documented mock contract and passing tests that cover both successful and failing responses from the service. Real integration checks belong in a separate suite run against a test environment you control.
Wire the suite into CI
The team adds a CI job that builds the processor with the pinned Java runtime and runs the full test suite on every change to the processor module. The job must fail the build when any test fails, and the team should confirm this by temporarily breaking one assertion on a branch. Test reports and the NAR build artifact should be stored so reviewers can see which relationships and validators were covered. The output is a CI job that blocks merges on failing tests, verified by one deliberate failure.

What drives testing effort and how to keep it down

This table lists the factors that make a NiFi processor test suite larger or smaller, with the effect of each factor and the lever that reduces it. Read it before scoping, because effort follows the written scope: the more relationships, external systems and controller services a component touches, the more fakes and assertions the suite needs.

Driver	Effect on effort	How to keep it down
Relationships and property combinations	Each relationship needs at least one test that routes a FlowFile to it, and each mutually exclusive property set multiplies the cases to cover.	Keep the property model small, validate combinations in one place, and test error and failure relationships with focused cases instead of full matrices.
External systems the processor calls	Every HTTP endpoint, database, queue or API client needs a fake or stub, and each fake must match the response shapes the processor parses.	Isolate client calls behind a single interface or controller service so one fake covers many tests, and keep recorded responses small and representative.
Controller services that need fake implementations	A processor that looks up a controller service needs a test implementation registered with TestRunner, plus tests for its validation and enablement behavior.	Reuse existing mock or test services where they fit, and confirm with your team whether the service is already covered by its own test suite.
Record readers and writers	Record-based processors need schemas, sample records and checks on field types, nulls and schema drift, which adds fixtures and cases quickly.	Use one shared schema fixture set, test type handling with a small number of representative records, and keep schema-specific logic out of the processor where possible.
FlowFile content and attribute handling	Tests must build input content and attributes, then assert on content, attributes and provenance-relevant changes, so large or binary payloads raise setup and assertion work.	Use small synthetic payloads, assert on the attributes that matter, and keep binary handling in a few dedicated tests.
Team familiarity with TestRunner	Teams new to TestRunner spend time on setup patterns, lifecycle order and how controller services are registered, which slows the first suite.	Ask for one worked example in the scope, agree on a test template, and pair the first few tests with someone who has built NiFi processors before.
Reliance on framework behavior	If the processor depends on NiFi state, scheduling or session commit behavior, tests need more careful setup and may need integration-level checks as well.	Keep unit tests on processor logic, and state which behaviors need a running NiFi instance so they are scoped and estimated separately.
Continuous integration constraints	Running tests in CI requires build tooling, dependency resolution for NiFi artifacts, and a decision on how long the suite may take per run.	Confirm the build tool version, cache dependencies, and split slow integration checks into a separate pipeline stage.

Questions teams ask before testing NiFi extensions

What does it cost to write a nifi-mock test suite for a custom processor?

The cost comes mostly from the hours of engineering time needed to write and maintain the suite, and it depends on how many components are in scope and how much external behavior must be simulated. Cost rises with the number of processors and controller services, the number of relationships and property combinations to cover, the number of external systems that need mocks (databases, HTTP APIs, message brokers), and how much existing code must be refactored so it can be tested in isolation. One-time costs are the initial suite, test data and CI wiring. Recurring costs are updates when processors change and when NiFi versions move. Ask any partner for a written scope listing components, test categories, mocked systems, CI integration and handover material, then compare the quote against that list rather than against a headcount guess.

How long does a first nifi-mock test suite take to build?

Duration is set by the number and complexity of the components under test and by how much the team already knows about the codebase and the flows that use it. A single processor with a few properties and clear relationships is a much smaller job than a set of processors that share a controller service and call an external API with paging and retries. The main drivers are the number of property combinations and error paths, the effort needed to build realistic mocks or fixtures, whether the build and CI pipeline already runs Maven tests, and how quickly reviewers can confirm expected outputs. Ask for a phased plan with a first milestone that covers the highest-risk component, and confirm which dates depend on the client providing access and sample data.

What must our team provide before testing starts?

You need to provide the source code for the custom processors and controller services, the target NiFi version and Java runtime, and a clear list of the external systems each component talks to. Sample data is needed in representative form, such as anonymized records or synthetic FlowFiles that match the real content types and schemas. Your team should also provide CI access, meaning a repository, a build agent that can run Maven tests, and permission to add a test stage. Ask any partner to confirm these items in writing, including who approves data use and where test artifacts are stored, so that gaps are found before the work is scheduled.

Can our in-house team write these tests without a partner?

Yes, if the team already builds and maintains custom NiFi extensions and understands Maven, the NiFi API and the TestRunner model. A team new to NiFi extension development usually needs time to learn how properties, validators, relationships and controller service lifecycles behave under test. An outside partner is most useful when the team lacks capacity, when the suite must be set up quickly for a release, or when it needs a reviewed baseline and a handover plan. Whichever path you take, decide who owns the suite after delivery, who reviews changes to tests, and how failing tests are triaged.

Do unit tests with nifi-mock replace integration tests?

No. nifi-mock runs processors and controller services in a simulated framework, which is well suited to checking property validation, routing logic, attribute handling and output content. It does not prove that a flow works against a real database, a real broker or a real NiFi cluster with its repositories and security settings. Keep unit tests for fast feedback on each change, and add a smaller set of integration tests against a running NiFi instance and real or containerized dependencies before production releases. Confirm with any partner which tests run in each pipeline stage and which environments they require.

How do we keep nifi-mock tests working across NiFi upgrades?

Tests break across upgrades when the NiFi API, the components a processor depends on, or the runtime changes. NiFi 2 changed several areas that custom code may touch, so review the NiFi migration guidance for the versions you move between and run the suite against each target version before upgrading production. Pin the NiFi and nifi-mock versions in the build, keep test dependencies separate from runtime dependencies, and record which version each test was written against. Ask a partner how upgrade testing is scheduled and whether it is covered by a support arrangement or a separate retainer.

What should we confirm in a quote for this work?

Ask for a written scope that names each processor and controller service, the test categories included (property validation, routing, error handling, external system mocks), the external systems to be mocked, the NiFi and Java versions targeted, and the CI platform used. The quote should state deliverables such as test code, documentation, a CI configuration and a handover session. Confirm what is excluded, such as integration environments, performance tests or upgrade runs, and whether ongoing maintenance is a separate retainer. DeployOpen works from a fixed written scope, so ask for the timelines and acceptance criteria to be written into that scope before work begins.

How the test suite connects to the rest of the platform

A nifi-mock test suite only pays off when it exercises the same artifact that runs in production and fits the way flows are built, versioned and upgraded. The cards below cover the links that most often break when a team treats unit tests as an isolated codebase.

Package the processor as a NAR

Run the TestRunner suite against the same NAR that ships to the cluster, not a separate test-only classpath. Confirm that the processor is declared in the NAR's service and extension manifests, because a class that loads in tests but is missing from the packaged archive will pass CI and fail at deployment. Check that the test dependencies stay out of the runtime NAR so the production artifact stays small and predictable.

Wire tests into the CI pipeline

Make the nifi-mock suite a required step that runs on every merge request, before the NAR is built for deployment. Pin the Java runtime used in CI to the version your target NiFi line requires, since the NiFi administration guide lists Java 21 as the minimum runtime. Fail the build on test errors rather than on warnings so that a broken processor never reaches an artifact repository.

Keep flow design out of unit tests

Unit tests should set properties directly on the processor under test, not load a flow.json.gz or depend on the parameter contexts of a particular environment. Decide which values come from parameters at deployment time and test that the processor validates them, not that a named parameter resolves to a specific value. Otherwise a renamed parameter in a promoted flow breaks tests that had nothing to do with the change.

Align tests with Git-versioned flows

When flows are versioned through a Git-based Flow Registry Client, the processor version a flow references must match the NAR version the tests were run against. Tag releases so that a flow version, the NAR version and the test run that approved them can be traced together. Ask whether your change process requires that link to be recorded for audit evidence.

Plan tests around the upgrade path

Migration guidance for NiFi 2 changes APIs and configuration that custom processors may depend on, including the removal of the variable registry expression language scope. Keep a test profile for each NiFi line you intend to run, and treat any failing assertion after a framework bump as an upgrade finding rather than a test to delete. Confirm the exact API changes against the migration guidance for your source and target versions before planning the work.

Separate controller service coverage

Processors that depend on controller services need tests for the service's own lifecycle, validation and sharing behavior, which the controller service pattern covers separately. In the processor suite, add the service through the TestRunner and verify how the processor reacts when it is disabled or misconfigured. Keep the external system mocked at the service boundary so the processor tests do not require a live database or API.

What to hand over at the end of a testing engagement

The handover is the set of artifacts your team needs to run, extend and trust the nifi-mock suite after the engagement ends. The core deliverable is the test source tree committed to your repository, with a README that explains how to run the suite locally and in your CI system. The README should name the Java and Maven versions used, the commands that run the full suite and a single test class, and how controller service mocks are registered with the TestRunner. Ask for the CI job definition as a file in your pipeline repository, not as a description, so the job can be changed by your own engineers.

The handover should also include a written list of what is mocked and what is not. For each processor, list the external systems it calls (HTTP endpoints, databases, message brokers, object stores) and mark whether each is replaced by a mock, a stub or a real local instance. For each controller service, provide a mock contract that states the methods the processor relies on, the return values and exceptions the mock produces, and the properties that the mock accepts or ignores. A reader should be able to tell which assertions prove processor logic and which only prove that a mock was called. The known gaps section must name the behaviors that only integration tests can confirm, such as real authentication handshakes, TLS configuration, network timeouts and behavior under a running cluster. Confirm the format and level of detail for these documents with your team before the work starts.

Your responsibilities start before the engagement. You need to provide access to the source repository, the CI pipeline and permission to add or change jobs, along with a test environment or test accounts for any external system that an integration test will reach. Name a technical owner who can review pull requests and approve the mock contracts. Acceptance criteria belong in the written scope: confirm which processors and controller services are covered, that the suite passes in your CI system from a clean checkout, and that the known gaps document is complete. Agree in writing how defects found during acceptance are handled, and whether acceptance is confirmed by your owner in writing.

How DeployOpen scopes and delivers NiFi extension testing

DeployOpen treats processor testing as part of the extension, not an afterthought. We scope the test suite against the processors and controller services your flows depend on, build it with the TestRunner harness, and hand over a suite your own engineers can run and extend.

How we scope

Inventory every custom processor and controller service, with the properties, relationships and external systems each one touches.
Agree in writing which behaviors need unit tests, which need mocked external systems, and which are better covered by integration tests in a real NiFi.
Define the CI environment, Java and build tooling, and the pass criteria the team will use to accept each extension.

How we build

Write TestRunner tests that enqueue FlowFiles with attributes and content, set properties, and assert routed relationships and output content.
Add controller service stubs and mocks for databases, HTTP APIs or message brokers so tests run without live dependencies.
Wire the suite into the build pipeline so every change to the extension runs the same tests before a NAR is produced.

How we hand over

Deliver the test code in your repository with a short guide to running it locally and in CI, including how to add a new case.
Document the mocked boundaries, their known limits, and the behaviors that still require a live NiFi test before release.
Walk your developers through the suite so they can maintain it when processors change.

How we support

Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
Ongoing support, upgrades and continuous testing are available as a retainer, including updates when NiFi versions change.
Ask for a quote based on the written scope, which sets out who does which part of the build and what your team receives.