Calling REST APIs from Apache NiFi with InvokeHTTP

How InvokeHTTP calls REST APIs from Apache NiFi

When a team needs NiFi to pull from or push to a REST API on a schedule, InvokeHTTP is the processor to use. It takes a configured URL and HTTP method, and for PUT, POST and PATCH it sends the FlowFile content as the request body. Its Attributes to Send setting can convert FlowFile attributes into HTTP headers, which lets upstream processors pass request metadata, identifiers or pagination cursors without rewriting the payload. InvokeHTTP replaced the deprecated GetHTTP and PostHTTP processors, so any flow still built on those components needs to be rebuilt around it before a NiFi 2 move. The migration also changes where connection settings live. Proxy settings no longer sit on the processor. They move to a Proxy Configuration Service, which means one proxy definition can be shared across every processor that needs it. For HTTPS, InvokeHTTP uses the Java runtime default certificate authorities when no SSL Context Service is attached. Teams calling internal APIs that use a private certificate authority should decide early whether to add a truststore through an SSL Context Service or to extend the runtime trust store, and should confirm the choice with their security team before the flow goes to production.

Before building the flow, confirm three things with the API owner: the exact endpoint and method for each operation, the authentication scheme, and the response codes that signal rate limiting or transient failure. NiFi 2 includes the StandardOauth2AccessTokenProvider and JWTBearerOAuth2AccessTokenProvider controller services for obtaining OAuth 2.0 access tokens, so token handling can stay out of custom scripts. The reader should verify which grant type the provider supports for their identity platform and how long tokens remain valid, since that determines how often the flow refreshes them. Plan the flow around failure before the first run. An API that returns paged results or throttles requests will fail in ways a simple single-call test will not reveal, so run the flow against a test tenant with realistic volumes. Record which response codes route to retry, which route to a failure path with the response body preserved, and where the cursor or page state is stored between runs. These decisions drive the operational cost of the integration, and they are easier to change in a design review than after the flow has been running against production data.

What you need before building the first InvokeHTTP flow

Most InvokeHTTP failures trace back to the runtime, the API contract, network paths or certificate trust, not to the processor itself. Confirm these four areas with the owners of each system before you place the first processor on the canvas.

NiFi 2 runtime and Java version

Confirm you are building on NiFi 2, not 1.x, because the 1.x line cannot be maintained further and the migration path starts from 1.27.0 before moving to 2.0.0. The NiFi administration guide lists Java 21 as the minimum runtime, so check the JVM on every cluster node, not only the one you log into. Mixed Java versions across a cluster are a common cause of unexplained processor behavior.

API documentation and test credentials

Obtain the target API's reference, its authentication method, rate limits and error response format before you design the flow. Ask the API owner for a non-production sandbox and a dedicated test identity with the narrowest scope that still lets you read or write the required resources. Record which endpoints are idempotent, because that decides how retries are safe later.

Network routes and egress rules

Confirm that the NiFi nodes, not only a workstation, can reach the API host on the required port, and that any proxy the API requires is reachable from every node. Ask the network team whether egress is allowed by destination or by domain name, and whether DNS resolves the API name the same way on each node. A flow that works on one node and fails on another often points to a node-specific route or proxy setting.

Certificate authority chain and trust

When no SSL Context Service is configured, InvokeHTTP uses the Java runtime default certificate authorities for HTTPS. If your organization or the API uses an internal CA, confirm that the full chain is trusted by the Java runtime on every node, or plan an SSL Context Service with your own truststore. Ask the API owner which intermediate certificates they serve, since a missing intermediate is a frequent cause of handshake failures.

Proxy and outbound policy decisions

If outbound traffic must pass through a forward proxy, decide whether it applies to all flows or only to specific processors before anyone configures credentials. Proxy settings for InvokeHTTP live in a Proxy Configuration Service rather than on the processor, so plan one shared service with a named owner. Confirm with your security team which proxy authentication method is permitted.

Configuring InvokeHTTP step by step

This sequence takes an InvokeHTTP processor from an empty canvas to a tested call against an API, with authentication, proxying, certificate trust and relationship routing each set and checked in order.

Create the processor and set URL and method
Add an InvokeHTTP processor to the canvas inside the process group that owns the integration, then set the HTTP Method and HTTP URL properties. Use a parameter for the base URL so the same flow can point at a sandbox and a production endpoint without editing processor settings. Confirm the method matches the API contract for the endpoint you are calling, since GET, POST, PUT, PATCH and DELETE each imply different request semantics. Check that the processor shows no validation errors before continuing. The processor is ready when it validates with the URL resolved from the parameter context.
Decide the request body source
For PUT, POST and PATCH, InvokeHTTP sends the FlowFile content as the request body, so the upstream processors must produce the exact payload the API expects, such as JSON from a record writer or a converted document. For GET and DELETE, leave the body empty and carry any query data in the URL or attributes. Confirm the Content-Type the API requires and set it explicitly rather than relying on the content of the FlowFile. Send a small sample FlowFile to a test call and inspect the request body in the server logs or the sandbox service. The body source is settled when the sandbox receives the same bytes that the FlowFile contains.
Map attributes to request headers
Set the Send Message Body and Attributes to Headers properties so that selected FlowFile attributes become HTTP headers. Use a regular expression for the attribute names that should be sent, and keep it narrow so internal attributes, such as file paths or upstream identifiers, do not leak to external services. Add static headers, such as Accept, through the dynamic property mechanism where the API requires them. Confirm with the API owner which headers are mandatory and which are forbidden. The step is complete when a test FlowFile carrying a sample attribute produces the expected header on the sandbox request.
Add an OAuth 2.0 access token provider
If the API uses OAuth 2.0, create either a StandardOauth2AccessTokenProvider or a JWTBearerOAuth2AccessTokenProvider controller service, depending on the grant type your identity provider supports, and enable it. Enter the token endpoint, client identifier and secret or signing material from the identity administrator, storing secrets in sensitive parameters. Reference the provider from the InvokeHTTP processor so that it obtains and refreshes the bearer token for each call. Ask the identity team which scopes and audience values the API requires before you start. The check is a sandbox call that succeeds with the token attached, and an invalid credential that fails with an authorization error.
Attach a proxy configuration service
If outbound HTTP traffic must pass through an egress proxy, create a Proxy Configuration Service with the proxy host, port and any authentication details your network team provides, then assign it to InvokeHTTP. Proxy settings live in this service rather than on the processor, which allows the same configuration to be reused across processors. Confirm with the network team whether the proxy also intercepts TLS, since that affects the certificate step that follows. Run a test call from a NiFi node and check the proxy access log to confirm the request passed through the intended path. The step is complete when the call succeeds and the proxy logs show the destination host.
Set the SSL context for private certificate authorities
InvokeHTTP trusts the Java runtime default certificate authorities for HTTPS when no SSL Context Service is set. If the API uses a certificate issued by a private or internal CA, create a StandardSSLContextService with a truststore containing that CA chain and assign it to the processor. Store the truststore password as a sensitive value and keep the file on a path readable by the NiFi service account. Ask your PKI team for the full chain, including intermediates, rather than only the leaf certificate. The check is a successful handshake from every node in the cluster, which you can confirm by running the test call on each node or from a flow that exercises all of them.
Route success, retry and failure relationships
Connect the Response relationship to downstream processing for successful responses, and connect Retry and Failure to separate paths. Route Retry to a Wait or penalized loop that re-sends the FlowFile after a delay, which is the place to handle HTTP 429 and transient 5xx responses from rate-limited APIs. Route Failure to a queue that alerts an operator or writes the request and response to a dead-letter location for review. Check the response status code and the invokehttp.status.code attribute on a sample of outputs to confirm each status reaches the intended relationship. The step is complete when a deliberately rejected request lands in Failure and a throttled one lands in Retry.
Test the flow against a sandbox endpoint
Run the complete flow against a sandbox or staging instance of the API before connecting it to production data. Send a representative set of FlowFiles covering a successful call, an authentication failure, an invalid payload and a throttled response, if the provider can simulate one. Inspect response attributes such as the status code, the response body and the request headers that InvokeHTTP recorded, and compare them with the API documentation. Confirm that the bearer token is refreshed when it expires during a longer run. The test is complete when every path produces the expected result, and the findings are documented for the operations team.

How FlowFile content and attributes become an HTTP request

InvokeHTTP builds each request from two inputs: the FlowFile content and the FlowFile attributes. The URL and HTTP method are set on the processor, or driven by attributes if you configure them that way. For PUT, POST and PATCH, the FlowFile content is sent as the request body. For GET and DELETE there is no body, so any content must be handled upstream, for example by moving the data into a query parameter or a header. Before you build the flow, decide which method the API expects for each call and confirm in the InvokeHTTP component documentation how the processor treats the body for that method in the NiFi 2 release you run.

Attributes are the second input. InvokeHTTP can convert FlowFile attributes into HTTP headers, which is how values such as a tenant identifier, an idempotency key or a content type reach the API without being written into the body. Attribute names that match your header naming convention should be set in an UpdateAttribute or equivalent processor before InvokeHTTP, and you should check which attributes are forwarded by default and which are filtered by the processor's attribute-to-header settings. Avoid placing secrets in attributes that travel with the FlowFile through provenance and queues. Use a credential-bearing controller service, such as the OAuth 2.0 access token provider, to add authorization instead.

On the way back, the response status code, status message and response body are written to the FlowFile and its attributes, and the processor routes the FlowFile to success, failure, retry, no retry or original relationships depending on the outcome. Routing decisions in your flow usually depend on the status code attribute, so confirm the exact attribute names and the status ranges that map to each relationship in the component documentation for your NiFi 2 release. Test with a non-production endpoint that returns each class of response you expect, including a rate-limit response, and record what lands in the attributes before you build routing logic on top of them.

Mapping FlowFile attributes, headers and API fields

This table maps where each value in an InvokeHTTP call comes from and where it lands, whether that is a FlowFile attribute, an HTTP header, a query parameter, a body field or a record schema field. Read each row as a contract between your flow and the API, and confirm every name, type and required flag against the provider's API reference before you build the processor configuration. The main takeaway is that most failed integrations trace back to one mismatched field name or an attribute that was never set upstream.

Source or target	HTTP header or parameter	API field	Type	Owner
Attribute set by upstream processor (for example, an API token attribute)	Authorization header via Dynamic Property	None (credential)	String, sensitive	Platform team
Attribute such as a tenant or account identifier	Custom header or path segment	Path or header field named by the API	String	Integration developer
Attribute holding a page cursor or last-seen timestamp	Query parameter (for example, a since or cursor parameter)	Filter or pagination field	String or timestamp	Integration developer
FlowFile content for PUT, POST or PATCH	Request body (no header mapping)	Record root or envelope object	JSON, XML or CSV	Data owner with API team
Record field in the body, mapped through a record reader schema	Not applicable	Request body field names	Typed per schema	Data owner
Response status code or response header attribute	Response header such as Retry-After	Not applicable	Integer or string	Integration developer
Response body fields extracted into attributes or records	Not applicable	Nested response fields such as next page link or item list	String, number or array	Integration developer
Schema used by record reader and writer for response records	Not applicable	Full set of response fields the flow keeps	Schema definition	Data owner

Choosing an authentication pattern for the API

A static API key or long-lived bearer token is the simplest model. InvokeHTTP sends it as a header on every request, usually built from a FlowFile attribute or a parameter. The credential does not expire on its own, so the flow needs no token refresh logic, but every rotation becomes a manual change that must be coordinated with the provider and the NiFi operators. Store the value as a sensitive parameter in a parameter context rather than as a plain processor property, and keep the parameter context restricted to the people who maintain the flow.

Storage: Use a sensitive parameter so the value is encrypted at rest in the flow and masked in the UI.
Rotation: The application owner or API administrator rotates the key, then the platform team updates the parameter and reviews the change through your normal change process.
Expiry mid-run: A revoked key fails every request at once, so InvokeHTTP routes FlowFiles to the failure relationship until the value is updated.
Scope: Confirm whether the provider issues keys with limited scopes or per-environment keys, since a single broad key widens the blast radius.
Audit: Confirm whether the provider logs which key made each call, because a shared key hides which flow sent a request.

Confirm with the API provider how keys are revoked and whether a second key can be issued before the first is retired.

Designing pagination without losing or duplicating records

The core risk in paginated API ingestion is that a page is either skipped or read twice, and InvokeHTTP does not prevent either by itself. Each request it makes is stateless, so the flow must carry the position forward. Choose the pattern the API actually offers: offset pagination maps to a counter stored in a FlowFile attribute and incremented after each successful response, cursor pagination maps to the opaque token read from the response body with EvaluateJsonPath and written back into the next request as a query parameter or header, and link-header pagination maps to the URL found in the Link response header, which can be read from the response attributes InvokeHTTP writes and fed back into the URL property. Each pattern loops from the response branch back to the request, so the loop must have one exit path that is reached only after the stop condition is confirmed.

Cursor state must survive restarts, so do not rely on a FlowFile attribute alone. A FlowFile that is queued when NiFi restarts, or that is dropped by a failure branch, takes its cursor with it. Store the last confirmed cursor in NiFi state with UpdateAttribute-driven state handling or a processor with state management, and advance it only after the downstream write has succeeded. In a cluster, state scoped to the local node stays on that node, so a different node picking up the next scheduled run starts from nothing. Decide whether the pagination processor runs on the primary node only, and set the state scope to match that decision. Test a forced restart in the middle of a run and compare the record count against the source's own total. Limits such as maximum page size, rate ceilings and whether a cursor expires must be confirmed from each API's own pagination documentation, not assumed from another vendor.

Handling rate limits and transient failures

Rate limits and server errors are normal behavior for most public APIs, so the flow needs a deliberate answer for each response. The cards below cover how to read the server's wait signal, where to pause work, when to stop retrying, and how cluster nodes share the request budget.

Honor Retry-After before retrying

When an API returns 429 or 503, it may include a Retry-After header that states how long to wait. Read that header first and use it as the minimum delay before the next attempt. InvokeHTTP exposes response headers as FlowFile attributes, so a downstream RouteOnAttribute or expression can carry the value into the wait logic. Check whether the vendor sends the header on every limited response, and whether it uses seconds or an HTTP date.

Back off with penalty, not sleep

Choose between penalizing FlowFiles and lengthening the run schedule. A penalized FlowFile waits in its queue while other FlowFiles keep moving, which suits per-request throttling. Increasing the run schedule slows the whole processor and suits a global quota. Avoid script or Wait-style loops that hold a thread, because they block other work on the node. Test the chosen approach with a sustained burst of records, not a single failing request.

Cap retries and count attempts

Set a maximum attempt count and store it in a FlowFile attribute that increments on each failed pass. Without a cap, a permanent 5xx or an expired credential loops indefinitely and fills the queue. Decide which status codes are retryable. Typically 429 and some 5xx codes are, while 400, 401 and 403 usually are not and should go to failure immediately. Confirm the status-code behavior with the target API's documentation.

Route exhausted FlowFiles to dead letters

When the attempt cap is reached, route the FlowFile to a dead-letter path rather than dropping it. Keep the original request body, the URL, the last status code and the response body so an operator can replay the record later. Give the dead-letter queue its own alerting and an owner. Note that FlowFile content and provenance also consume repository space, so set a retention plan for this path before go-live.

Share the quota across cluster nodes

In a clustered NiFi, every node that runs InvokeHTTP sends requests, so the effective rate is the sum across nodes. A per-node limit that looks safe on one node can exceed the API quota on three. Decide whether the processor runs on all nodes or only the primary node, and use a shared rate control or a single-node ingestion point where the quota is strict. Ask the API owner whether limits are per key, per IP or per account.

Common InvokeHTTP failures and fixes

This table pairs each common InvokeHTTP failure with its likely cause, the check to run first and the fix to apply. Read it in order of symptom: find the error you see, confirm the cause with the check, then apply the fix in the processor or controller service named. The main takeaway is that most failures trace to certificate trust, token lifecycle, proxy settings or retry design rather than the API itself.

Failure	Likely cause and check	Fix
TLS handshake error against an internal API	The server certificate is issued by a private CA that is not in the Java runtime default trust store. Check the processor logs for PKIX or certification path errors, and confirm the CA chain with a test call from the NiFi host.	Add the private CA to a truststore, attach a StandardSSLContextService to InvokeHTTP, and confirm the CA is trusted on every cluster node.
401 Unauthorized after hours of running	The access token expired and the flow is reusing a stale Authorization header. Check whether the token comes from a fixed attribute or from an OAuth 2.0 access token provider, and compare the failure time with the token lifetime the identity provider issues.	Obtain the token from StandardOauth2AccessTokenProvider or JWTBearerOAuth2AccessTokenProvider so it refreshes, and send it through the processor's authorization settings rather than a static attribute.
Requests time out or fail to reach the API through a proxy	Proxy settings were configured on the processor or in an old property rather than in a Proxy Configuration Service, or the proxy blocks the target host. Check the proxy host, port and no-proxy list against the URL and test from the NiFi node.	Move proxy settings into a Proxy Configuration Service, attach it to InvokeHTTP, and add internal hosts to the bypass list.
Headers missing or duplicated at the API	Attributes sent as headers match a name pattern you did not intend, or the same header is set by an attribute and by a processor property. Check the outgoing request in the processor's debug output or at a test endpoint that echoes headers.	Restrict the attribute-to-header pattern to explicit names, remove conflicting properties, and set each header once in one place.
429 Too Many Requests in bursts	The flow sends requests faster than the API quota, often because every FlowFile is sent at once or every cluster node calls the same endpoint. Check the response Retry-After header, the request rate per node and the number of concurrent tasks.	Lower concurrent tasks, add a controlled delay between calls, honor Retry-After, and route 429 responses to a retry path with backoff instead of looping immediately.
Pagination loop that never ends	The next-page token or offset is not updated, or the stop condition checks an empty array that the API never returns. Check the attribute that stores the cursor and the response body of the last page.	Store the next cursor in an attribute, stop when the API reports no further pages, and cap the number of pages per run as a safeguard.
Duplicate writes after a retry on POST	The API accepted the first request, but NiFi did not receive the response and retried. POST is not idempotent by default. Check whether the target system deduplicates on a client-supplied key and whether the response was lost or the request failed.	Send an idempotency key the API honors, use PUT with a stable resource identifier where the API supports it, or check for the record before writing. Confirm the API's behavior with its vendor documentation.
Responses succeed but FlowFiles route to failure with no clear error	The response status is outside the expected set, or the body is not what downstream processors expect. Check the Status Code and response attributes on the failed FlowFile.	Route by response code explicitly, add a validation step on the response body, and keep failed FlowFiles with their attributes for replay.

Questions teams ask before building REST API flows in NiFi

What drives the cost and effort of a REST API integration in NiFi?

The main drivers are how many endpoints the flow calls, how the API authenticates, how it paginates and limits requests, and how many downstream systems receive the data. A single read-only endpoint with an API key is a small build. Several endpoints with OAuth 2.0 token refresh, paginated responses, retries for rate-limited calls and record-level error routing is a larger one. Cost has two parts: the one-time design and build work, and the recurring effort to keep flows working as the API and the platform change. Ask any quote to break down endpoints, authentication flows, error handling, testing against a sandbox and handover documentation as separate line items.

How long does a typical NiFi API flow take to build?

Duration depends mainly on the number of API endpoints, the authentication model and how clean the vendor documentation is. Each endpoint needs its request shape, response parsing, pagination logic and failure handling designed and tested. OAuth 2.0 with token refresh, JWT bearer grants or a proxy between NiFi and the API add design and test time. Build time also depends on how quickly the client can provide a sandbox and credentials. No general duration applies, so ask for a timeline tied to a written list of endpoints and the test environment you will use. A fixed-scope agreement sets this out before work starts.

What does the client team need to provide before the work starts?

The build moves faster when the client team provides a few things early:

API documentation, including rate limits, pagination rules and error response formats
A sandbox or test tenant that behaves like production, with test data
Named owners for each credential, with the scopes or roles each one carries
Network paths from the NiFi cluster to the API and any proxy that must be used
A decision on where secrets are stored and who can rotate them

Without a sandbox, testing happens against live data, which raises risk. Without named credential owners, token and key rotation becomes a recurring delay.

Can an internal team build REST API flows in NiFi without a partner?

Yes, a team with NiFi experience and API integration skills can build these flows alone. InvokeHTTP is a standard processor, and OAuth 2.0 token providers and the Proxy Configuration Service are part of NiFi 2. The harder questions are operational: retry logic that respects a provider's rate limits, safe handling of pagination state across a cluster, secret handling, and tests that catch changes in the API. Ask whether the team has someone who will own these flows after launch, whether they have time for documentation and testing, and whether they can run a sandbox. If those are covered, building in-house is a reasonable choice.

How should a team handle changes to the API it calls?

Start by treating the API contract as a dependency. Pin the expected response fields in a test, validate records against a schema before they leave NiFi, and route unexpected responses to a failure path with the raw content kept for review. Subscribe to the provider's change notices and version announcements, and test flows in a sandbox before each provider change takes effect. Store flow configuration in a version control system, using NiFi's Git-based flow registry clients where your platform supports them, so changes can be reviewed and rolled back. Ask any partner how they will detect breaking changes and who approves updates to production flows.

When is it worth bringing in a partner for API integration work?

A partner is worth considering when the integration is business-critical, spans many endpoints or several authentication methods, or must meet audit requirements for data transfer and change management. It also helps when the internal team lacks time, or when the flows must run in a restricted network with no direct access to the internet. DeployOpen plans, builds, documents, tests and supports self-hosted NiFi flows inside your own infrastructure, under a written scope. Ongoing support and continuous testing are available as retainers. Ask for a written scope that lists endpoints, authentication, failure handling, test evidence and handover before comparing options.

Does a partner guarantee that the API integration will pass an audit?

No. DeployOpen prepares systems, flows, controls and evidence for compliance assessments, but independent auditors and certification bodies issue audit reports and certifications. For an API integration, useful evidence includes documented flow design, credential ownership records, logs of data transfers, change history and test results. Ask your auditor which of these they expect to see before the build starts, so the scope includes the right evidence.

Official NiFi references for InvokeHTTP

Start with the InvokeHTTP component documentation on the NiFi components site. It is the authoritative reference for the URL and HTTP Method properties, the rule that PUT, POST and PATCH requests send FlowFile content as the body, and the option to convert FlowFile attributes into HTTP headers. Before you build, read the property descriptions for the version you will run, confirm which attributes are converted to headers and which are excluded, and note how the processor reports response codes and error relationships. Those details decide how your retry and error routing will behave.

The migration guidance on deprecated components for 2.0.0 explains why InvokeHTTP matters if you are moving an older flow. The Migrating Deprecated Components and Features for 2.0.0 page records that InvokeHTTP replaced the deprecated GetHTTP and PostHTTP processors, that HTTPS without an SSL Context Service now relies on the Java runtime default certificate authorities, and that proxy settings moved to a Proxy Configuration Service. Any flow that set proxies or trust stores directly on the old processors needs those settings rebuilt before cutover, so check each HTTP-facing processor against this list.

The components index lists the controller services available in your NiFi 2 release, including the StandardOauth2AccessTokenProvider and JWTBearerOAuth2AccessTokenProvider services used for OAuth 2.0 token acquisition. Confirm in that index which provider matches your identity platform's grant type and token endpoint, and confirm the exact property names on the service page for your version. Treat the index as the place to verify that the services exist in the build you deploy, since controller service names and options change between releases.

How DeployOpen scopes and delivers NiFi API integrations

DeployOpen treats each InvokeHTTP integration as a fixed-scope engagement that starts with the API contract, authentication model and rate limits of the target service. We build the flows inside your own infrastructure, test them against real responses and hand over flows, documentation and a runbook your team can operate.

How we scope

Inventory each API endpoint, HTTP method, request body and response shape the flows must call
Agree the OAuth 2.0 grant, token lifetime and proxy path each integration needs in writing
Define pagination, rate-limit and retry behavior per API, including the status codes that trigger a retry
Confirm which NiFi cluster nodes run each flow and where credentials will be stored

How we build

Build InvokeHTTP flows with method, URL and header attributes parameterized through NiFi parameter contexts
Configure access token providers and the Proxy Configuration Service instead of embedding secrets in processors
Implement pagination loops and backoff routing so rate-limited responses are retried without losing FlowFiles
Version flows in your Git repository through the Flow Registry Client so every change is reviewable

How we hand over

Test each flow against a sandbox or test tenant of the target API, with recorded request and response samples
Document endpoints, credential sources, expected status codes and failure routes in a handover pack
Walk your operators through monitoring queues, reading bulletins and rotating credentials in your environment
Confirm the written acceptance criteria before the engagement closes

How we support

Offer ongoing support, upgrades and continuous testing as a retainer after handover
Review flows when the API provider changes versions, authentication or rate limits
Plan NiFi 2 upgrades and deprecated component changes for each integration before they reach production
Ask your team to confirm support scope, response expectations and ownership of credentials in the quote