When Apache NiFi Site-to-Site data transfer between clusters is the right choice
Use Site-to-Site when data must move from one NiFi instance to another, whether between data centers, into a cloud account, or from an edge site into a central cluster, and you control both ends. Site-to-Site is the preferred NiFi protocol for this job. After you enter the remote URL, the sending flow discovers the input ports the remote side exposes, each port can be limited to specific users, and the transfer can use certificates for encryption, authentication and authorization. Those controls sit in the NiFi flow itself, so the team that owns the central cluster decides who may send data into it and where.
A remote process group fits better than a generic transfer when both sides run NiFi and you want NiFi to handle the mechanics. With a remote process group, the sender detects changes in the receiving cluster's membership and spreads FlowFiles across its nodes, sends data in batches, and compares checksums between sender and receiver to detect corruption. An HTTP endpoint or a message broker can carry the same bytes, but your team then builds and tests the load distribution, batching and integrity checks itself. A broker also adds a platform to run, which may be the right call if other consumers need the same stream. Confirm the throughput needs in your own environment with a test, since NiFi does not publish a general performance comparison for these options.
Before you commit, confirm three things with your network and security teams. First, check that the sending side can reach the receiving cluster's Site-to-Site port through any firewall or reverse proxy in between, since the Administrator's Guide documents Site-to-Site properties and reverse proxy configurations for this traffic. Second, decide which identity the sender presents and which ports that identity may write to. Third, confirm whether both clusters run NiFi versions that support the setup you plan, because a mismatch across a 1.x and 2.x estate can change what you can build. The Apache NiFi User Guide covers the Site-to-Site behavior you will rely on.
Site-to-Site configuration options and when each fits
This table compares the three controls that shape how a remote NiFi cluster is reached: direct node-to-node connections, a reverse proxy for Site-to-Site traffic, and per-port user restrictions. Read each row as a decision, since the first two govern network paths and the third governs which users can send data to a port. Most segmented designs layer these controls, so test each layer separately before relying on the combination.
| Option | Decision it answers | Properties and controls | Trade-offs for segmentation and operations |
|---|---|---|---|
| Direct node-to-node connection | Can each sending node open a socket to each receiving node? | nifi.remote.input.host, nifi.remote.input.socket.port, nifi.remote.input.secure | Simplest path with fewest hops, but every receiving node address and port must be reachable across the boundary. |
| Reverse proxy for Site-to-Site | Must traffic cross a boundary through one controlled entry point? | Proxy routing plus HTTP transport properties such as nifi.remote.input.http.enabled, as documented in the Administrator's Guide | Limits exposed addresses and suits segmented networks, but adds a hop, proxy configuration and a component to monitor. |
| Per-port user restrictions | Which users may send data to a given input port? | Access policies on each input port, set in the NiFi UI | Narrows who can deliver data without network changes, but depends on accurate user and certificate identities. |
| Layered design | How do the controls combine in a segmented deployment? | Proxy for network exposure, port policies for identity, TLS on each hop | Strongest separation, but more components to test, certificates to rotate and logs to review. |
| Verification after a change | How do you confirm each layer works? | Test from the sending side, then check the receiving port policy | Failures can look alike at the network, proxy and authorization layers, so isolate each layer during testing. |
How to set up Site-to-Site between two clusters
This sequence takes two NiFi clusters from an agreed data flow to a verified transfer, with trust, port access and load distribution confirmed at each stage. The result is a remote process group that discovers the receiving ports and delivers a checksum-verified test batch.
Agree the data flow and port layout
The receiving team and the sending team agree in writing which data moves, how often, in what volume, and which input port on the receiving cluster will accept it. Each port should be dedicated to one flow so access can be limited to the users who need it. The sending team should also confirm which NiFi version runs on each side, since both clusters must be on releases that support the Site-to-Site protocol configuration you plan to use. Done means a short design record names the source flow, the receiving process group, each port and the users allowed to send to it. Ask the receiving team to confirm the NiFi version on their side before you proceed.
Issue and trust certificates on both sides
Each cluster needs node and client certificates that the other side trusts, issued by a certificate authority both teams agree on. The receiving cluster must trust the sender's client identity, and the sender must trust the receiver's node certificates. Set the truststore and keystore properties in the Administrator's Guide on every node, and keep the same CA chain across the nodes of each cluster. Verify by confirming that a TLS handshake from a sender node to a receiver node succeeds with the expected certificate subject, before any NiFi configuration depends on it.
Configure receiving input ports and user access
On the receiving cluster, create an input port inside the target process group for each flow, then set its access policy so only the sender's identity can send data to it. Use the certificate-based identity or the user you created for the sender, and grant no broader rights than the port needs. Confirm the port is running and that its access policy lists the sender identity and nothing else. If the port is missing from discovery in a later step, check this policy first.
Add the remote process group with the remote URL
On the sending cluster, add a remote process group to the canvas and enter the URL of the receiving cluster, using the HTTPS address of one or more receiving nodes. Enable transmission only after the remote process group reports a successful connection. Set the transport protocol and the proxy settings if traffic passes through a reverse proxy, following the Site-to-Site properties in the Administrator's Guide. Done means the remote process group shows as connected in the UI and the sender's log contains no authentication or trust errors for that URL.
Confirm port discovery and membership
After the remote URL is saved, NiFi discovers the available input ports and lists them under the remote process group. Confirm that the port you configured appears, and that it is not listed if you expected it to be restricted. Then check the cluster membership: the sender should see every receiving node that is part of the cluster, so that data can be spread across them. Remove a node from the receiving cluster in a controlled test, confirm the sender stops using it, then bring it back and confirm it rejoins the list. Record the outcome for both checks.
Send a test batch and compare checksums
Route a small, known set of FlowFiles from the sending flow into the remote process group, then examine the receiving side to confirm the same number of FlowFiles arrived in the expected port. Site-to-Site compares sender and receiver checksums, so a successful transfer should show no checksum mismatch in either cluster's logs or provenance. Use the data provenance view on both sides to confirm the test FlowFiles are present, with matching content. Keep the test flow, the certificates and the port configuration as the baseline for later change reviews.
Document the setup and the reverse proxy path
If traffic crosses a reverse proxy or load balancer, record the proxy configuration used for Site-to-Site and test the path from the sender network, not only from inside the receiving cluster. Store the certificate inventory, port names, access policies and remote URLs in your change system with an owner for each. Ask your team to confirm the expected renewal process for certificates and the process for rotating a sender identity. The result is a documented setup that someone other than the builder can operate and verify.
Securing Site-to-Site traffic and its endpoints
Site-to-Site moves data across a trust boundary, so each link needs encryption, a verified peer and a limit on who can use it. The cards below pair each endpoint risk with one control: certificates, per-port access, the loopback default, edit rights on remote process groups, reverse proxy exposure and certificate lifecycle.
Certificate-based encryption and authentication
Site-to-Site can use certificates for encryption, authentication and authorization, so each sending and receiving node proves its identity before any FlowFile is accepted. Check that every node in both clusters presents a certificate issued by a CA you trust, and that the receiving side rejects peers it cannot verify. Confirm the TLS settings on the input and output ports match your certificate policy before you move production data across the link.
Per-port access control
Each Site-to-Site port can be limited to specific users, so a remote sender only reaches the ports it needs. Review the users granted access to every input port that receives remote data, and remove grants left over from earlier projects. Because ports are discovered automatically after the remote URL is entered, an unrestricted port is visible to every authenticated peer that can reach the cluster.
Keeping the UI on loopback
When no security configuration is provided, NiFi binds to 127.0.0.1, so the UI is reachable only from the host itself. Keep that default in any test or bootstrap setup until TLS and an authentication provider are in place, and do not change the bind address to a routable interface first. Confirm the final production configuration deliberately, because a Site-to-Site port left open on an unsecured instance widens the exposure beyond the UI.
Restricting edits to remote process groups
A remote process group defines where data goes, so anyone who can edit it can redirect production flow to another host. Limit edit rights on remote process groups to the team that owns the link, and keep them separate from general flow authors. Review changes to remote URLs and transmission settings through your change process, since a single edit can send data to an endpoint your compliance scope does not cover.
Reverse proxy exposure
When Site-to-Site traffic passes through a reverse proxy, the proxy becomes part of the trust path and must forward the client identity NiFi expects. The Administrator's Guide documents reverse proxy configurations for Site-to-Site traffic, so follow it exactly and test a remote sender through the proxy. Restrict which networks can reach the proxy listener, and confirm the proxy does not terminate TLS in a way that hides the peer certificate from NiFi.
Certificate rotation and monitoring
Site-to-Site links fail or silently stop working when certificates expire or are replaced without updating every peer. Plan rotation for node and proxy certificates before they expire, and monitor expiry dates and handshake failures across both clusters. Ask your team to confirm how a rotation reaches every remote node, since a partial rollout leaves some senders unable to deliver while others continue.
Questions teams ask about Site-to-Site implementation
How much effort does a two-cluster Site-to-Site build take?
The effort depends mostly on how many flows cross the link, how many environments sit between the clusters, and how much of the certificate and network work is already in place. A minimal build between two clusters you control involves configuring the remote process group, the receiving input port, access policies and TLS, then testing delivery and checksum behavior. Each additional flow, port, firewall path, proxy hop or environment (test, staging, production) adds verification work. Ask any provider for a written scope that lists the clusters, the ports, the number of flows, the environments and the testing evidence they will hand over. No effort figure is given here because none has been published for this topic.
Can our in-house team build Site-to-Site without outside help?
Yes, many teams can, provided they already run NiFi clusters and have people who understand NiFi's security model. The NiFi User Guide describes Site-to-Site as the preferred way to send data between instances, and the Administrator's Guide covers the properties and reverse proxy settings involved. The harder parts are usually certificate issuance and rotation, agreeing on port access between data centers, and deciding which users can see which input ports. If your team has not run a secured cluster before, an outside review of the design before go-live is a common middle path. Confirm your team's NiFi and PKI experience before choosing between building and buying.
What do we need to provide for a Site-to-Site project?
Plan to provide the following before work starts:
- Named owners for each flow that will send or receive data, who can approve port access and who signs off on test results.
- Certificates or a certificate authority that issues client and server certificates for both clusters, along with the rules for rotation.
- Network access between the clusters, including firewall changes, routing and any reverse proxy or load balancer in the path.
- Test data and a window in which both clusters can be exercised without affecting production traffic.
The scope should state who performs each item. Missing ownership of certificates or network changes is the most common reason a Site-to-Site project waits.
How long does a Site-to-Site implementation take?
Duration follows scope: the number of clusters, the number of ports and flows, how many environments must be built, and how quickly certificates and firewall changes are approved. A single link between two clusters with a small number of flows is much shorter to deliver than a multi-site design with edge sites, load balancers and several environments. No duration is stated here because no researched figure is available for this topic. Ask for a timeline that separates design, certificate and network lead times, build, test and handover, and that names which of these depend on your internal approvals.
What drives Site-to-Site cost up or down?
The main cost drivers are the number of clusters and environments, the number of flows and ports, the security requirements (certificate authority integration, user-level port restrictions, reverse proxy setups), and the amount of testing evidence you need for internal audit. Costs are one-time for design and build and recurring for certificate rotation, monitoring and changes when flows are added. Ask any provider to split the quote into one-time build, testing, handover and any retainer for ongoing support, and to state what changes are priced as new work. Ask which environments are included and whether the quote covers rework after testing.
What changes when moving Site-to-Site from NiFi 1.x to 2.x?
The protocol concepts stay the same, but the platform underneath changes, so the migration needs its own plan. The NiFi migration guidance requires upgrading to NiFi 1.27.0 before moving to 2.0.0 and changing the flow configuration file from flow.xml.gz to flow.json.gz. The NiFi downloads page notes that 1.28.1 reached end of support on December 8, 2024, so a 1.x cluster is a poor place to build new Site-to-Site links. Review your remote process group settings, certificates and reverse proxy configuration against the current Administrator's Guide before cutover, and test both sides of the link on 2.x.
What are the most common Site-to-Site failures?
The most common problems are mismatched certificates or trust chains between clusters, blocked ports or proxy paths, and input ports that are not visible to the sending user because access policies were not set. Other frequent issues include remote cluster membership changes that are not reflected in the sender's configuration, and checksum mismatches that point to network or proxy interference. A good test plan covers each of these: a certificate rotation test, a node-removal test, a proxy path test and a corrupted-payload check that confirms the checksum comparison catches errors. Ask the provider to show the test cases they will run and the logs they will keep.
Does a Site-to-Site design need a compliance review?
If the transferred data falls under a regulation or internal policy, the Site-to-Site design should be reviewed against the controls your auditors test, such as encryption in transit, access control to each port, and logging of transfers. DeployOpen prepares systems, controls and evidence for compliance assessments, but it does not issue audit reports or certifications; independent auditors and certification bodies do that. Ask which control evidence the implementation will produce, such as certificate inventories, access policy exports and test results, and confirm with your auditor which of those they will accept.
Planning Site-to-Site around a NiFi 2.x migration
If a site runs NiFi 1.x, plan the move to 2.x before you expand Site-to-Site traffic between clusters. NiFi 1.x cannot be maintained further because the line depends on Jetty 9.4, Spring Framework 5.3 and AngularJS 1.8, none of which can be upgraded within it. Every remote process group, input port and certificate trust you build on 1.x will need to be revisited during the migration, so new links built now add rework later. The migration guidance requires upgrading to NiFi 1.27.0 before moving to 2.0.0, and it changes the flow configuration file from flow.xml.gz to flow.json.gz. Your Site-to-Site design should be written against the 2.x target from the start.
The NiFi downloads page lists NiFi 2.12.0 as released on September 13, 2026, and the 2.x line is where the project directs new work. Before you cut over, confirm the Site-to-Site protocol behavior, port discovery, and the nifi.remote.* properties against the exact release you will run, using the NiFi System Administrator's Guide for that version. Also confirm how your reverse proxy and certificate chain behave on the new release, since both are part of the remote connection path. Do not assume the 1.x configuration carries over unchanged.
Treat the migration as a checkpoint for the remote link itself. Inventory every remote process group, the ports each one targets, the users allowed on those ports, and the certificates that authenticate them, then decide which links move first. Run the 1.x and 2.x sides in parallel during a test window, and compare sender and receiver checksum results and batch delivery counts before you retire the old path. Ask your implementation partner to document this inventory and the test evidence, so the cutover can be repeated for each site.
How DeployOpen scopes and delivers Site-to-Site work
DeployOpen plans, builds, tests and documents Site-to-Site transfers between NiFi clusters inside your own infrastructure. Each engagement starts from a written scope that names the source and destination clusters, the remote process groups, the ports and the certificates involved.
How we scope
- We inventory the sending and receiving NiFi clusters, their versions and the ports each side will expose for Site-to-Site.
- We record which remote ports each user or group may reach, and which certificate identities authenticate each direction.
- Engagements are fixed-scope, so the written scope lists flows, environments, reverse proxy paths and deliverables before work starts.
How we build
- We configure remote process groups, input and output ports and access policies, then verify port discovery from the remote URL.
- We set up certificate-based encryption and authorization, and test that load distribution follows remote cluster membership changes.
- We tune batching and enable checksum comparison, then run failure tests such as a dropped link or a stopped remote node.
How we hand over
- We deliver the flows, parameter contexts and configuration as documented, versioned artifacts that your team can review and redeploy.
- We provide a runbook covering certificate renewal, port changes, reverse proxy settings and the checks to run after each change.
- We walk your operators through monitoring of transfer status and checksum or delivery errors, and record open items for your sign-off.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer after handover.
- Support can cover planned NiFi version changes that affect Site-to-Site settings, scoped in writing before the work starts.
- Confirm with us during scoping which parts of the Site-to-Site design your own team will operate and which parts a retainer will cover.
