Why teams put MQTT proxying in front of IoT brokers
A fleet of devices connecting directly to one or more MQTT brokers means every device needs a known broker address, every broker must accept traffic from the open internet or from wherever the fleet sits, and any change to the broker layer reaches the devices. The mqtt-proxy plugin on an APISIX stream route puts a controlled edge in front of that layer. It supports MQTT 3.1.x and 5.0, and it routes each client by consistent hash on mqtt_client_id, falling back to the client IP when no client ID is present, so a device's session keeps landing on the same broker node. Teams usually reach for this when they run a clustered broker, need one stable endpoint for devices, or want to change broker capacity without reconfiguring hardware in the field.
The edge is worth its extra moving part when you need to control who reaches the brokers, when you run several broker nodes and must keep sessions sticky, or when you want TLS termination, mTLS and routing decisions in one place. It is less useful for a single broker in a small private network, where direct connections are simpler to operate and debug. Before committing, confirm with your broker vendor how it handles session state across nodes, since hashing keeps a client on one node but does not replicate its session. Also confirm the MQTT versions your devices actually speak, because the plugin's support covers 3.1.x and 5.0 only.
APISIX is licensed under the Apache License 2.0 and is a top-level Apache Software Foundation project. It is built on NGINX and etcd and uses dynamic configuration, so routes and plugins can change without a restart. Its stream proxying is off by default and must be enabled through apisix.proxy_mode before any stream route, including an MQTT one, will serve traffic. Review the APISIX mqtt-proxy plugin documentation against your broker and device firmware before you decide, and plan the stream listener, certificates and firewall rules as part of the same design.
Configuration options and when each one fits
This table compares the routing and TLS options available for MQTT traffic on APISIX stream routes, with the visibility and operational cost of each. Read each row against your device fleet, broker topology and the controls your auditors expect. The main takeaway is that every option that lets the gateway see more of the session also gives the gateway more to operate.
| Option | Best fit | Cost in visibility | Operational complexity |
|---|---|---|---|
| Client ID hashing (mqtt_client_id) | Broker clusters where each device must return to the same node for its persistent session | Gateway reads the CONNECT packet, so the client ID is visible to APISIX | Low to moderate: hash ring changes when broker nodes are added or removed, so confirm session behavior during rebalancing |
| Client IP fallback | Traffic where devices send no usable client ID, or where clients sit behind stable NAT addresses | Gateway sees only the source address, which may be shared by many devices behind one NAT | Low, but many devices behind one address can concentrate load on a single broker node |
| Stream route matching on server_port | Dedicated listener ports for MQTT on a shared gateway | Matches on the listener port without reading the payload | Low: one route per listener is simple to audit and reason about |
| Stream route matching on SNI | Several MQTT tenants or fleets sharing one TLS port with distinct hostnames | Requires TLS handshake metadata, so hostnames must be controlled by you | Moderate: each hostname needs DNS, certificates and a matching route |
| TLS termination with mTLS at APISIX | Fleets where the gateway should verify device certificates and log identities | Full visibility of the MQTT session and the device certificate at the edge | High: certificate issuance, rotation and revocation for every device, plus gateway key storage |
| TLS passthrough by SNI | Fleets where device certificates must reach the broker unchanged, or where the broker performs its own client authentication | The gateway routes on SNI only and cannot inspect MQTT packets or client IDs | Moderate: hash-based routing on client ID is not available, so confirm how the broker cluster balances sessions |
| TLS from APISIX to upstream brokers | Brokers that accept only encrypted connections inside a segmented network | Traffic is decrypted at the gateway, so the gateway handles the plaintext session | Moderate: upstream CA trust, broker certificate rotation and gateway-to-broker identity must be managed |
| Plaintext between gateway and broker | Isolated networks where transport security is provided by the segment itself | Full inspection, but traffic inside the network is unencrypted | Low to set up, but a common finding in assessments unless the segment is documented and restricted |
Choosing traditional, decoupled or standalone mode for IoT traffic
Traditional mode runs the data plane and control plane together on each APISIX node, with etcd holding the configuration. Stream routes for MQTT are created through the Admin API, and each node receives changes from etcd as they are written. This is the most familiar layout for teams already running APISIX, but the Admin API sits on the same node that accepts device connections, so its exposure needs to be planned with care.
- Client traffic port: Devices connect to the gateway on port 9080 in traditional mode, while the Admin API listens on port 9180.
- Stream proxying is off by default: Set
apisix.proxy_modetostreamorhttp&streaminconfig.yamlbefore any MQTT stream route will take traffic. - Route management: Create and update the MQTT stream route through the Admin API on port 9180, with access limited to automation and operators.
- Change propagation: Each node picks up route changes from etcd, so the etcd cluster must be healthy and reachable from every node.
- Network exposure: Keep port 9180 off the device network and restrict it to management addresses.
Before you commit, confirm that your device network can reach port 9080 and that the Admin API is not reachable from it.
Decoupled mode separates the data plane, which handles device connections, from the control plane, which manages configuration. Both planes share one etcd cluster. This keeps the Admin API off the nodes that terminate MQTT connections, which can simplify network segmentation for IoT traffic. The trade-off is an extra component to deploy, monitor and keep in sync with the data plane version.
- Separated roles: Gateway nodes that accept device traffic do not expose the Admin API, while control plane nodes hold the management surface.
- Shared etcd: Both planes depend on the same etcd cluster, so etcd availability and access control affect the entire fleet.
- Route ownership: Stream routes for brokers are written through the control plane and then delivered to data plane nodes.
- Version alignment: Keep control plane and data plane releases compatible, and confirm the supported combination in the deployment modes documentation.
- Segmentation: Place the data plane in the device-facing network zone and the control plane in a management zone with a narrow path between them.
Before choosing this mode, confirm the network paths the documentation requires between the planes and etcd, and confirm the ports your firewall rules must allow.
Standalone mode removes etcd entirely. The gateway reads its configuration from a YAML or JSON file, such as conf/apisix.yaml or conf/apisix.json, or receives updates through the API. For file-driven standalone deployments, APISIX checks the file for changes every second and hot-reloads without a restart. This suits fixed broker layouts that change through reviewed code, but it moves change control from the API to your delivery pipeline.
- No etcd dependency: The gateway has no shared datastore to operate, back up or secure.
- File-driven changes: Stream routes for MQTT live in the configuration file, so changes are reviewed, versioned and deployed like other code.
- Hot reload: Edits to
conf/apisix.yamlorconf/apisix.jsonapply within about a second without a restart, which you should verify in your own environment. - Per-node configuration: Every node needs the same file, so a distribution method that keeps files identical across nodes is required.
- Fleet changes: Adding a broker or device group means editing and redeploying the file rather than calling the Admin API.
Confirm with your team whether file distribution and review will keep every node consistent during broker maintenance.
Setting up an MQTT stream route in APISIX
This sequence turns APISIX into a layer 4 front door for your MQTT brokers, so device connections are accepted over TLS, pinned to a broker node by client ID, and confirmed with real MQTT 3.1.x and 5.0 clients before any device fleet is moved.
Confirm broker and network prerequisites
Your team inventories the MQTT broker nodes, their listening ports, the certificates devices will trust, and the network path from the device network to APISIX. Stream proxying is off by default, so this step also records which APISIX deployment mode and configuration source you will use. Done means you have a written list of broker endpoints, the ports they accept, and a named owner for each certificate. Ask the broker vendor or project to confirm which MQTT versions each node accepts, since the mqtt-proxy plugin's supported versions must match what devices speak.
Enable stream proxying in APISIX
Your platform engineers set the proxy mode in the APISIX configuration to stream, or to http and stream if the same gateway also serves HTTP routes, then restart or reload the gateway according to your deployment mode. This is the setting that makes stream routes exist at all. Check it by listing the loaded configuration on a test node and confirming that stream listeners appear in the startup log without errors. Keep the change in version control so the setting survives redeployment.
Define the stream route and upstream broker nodes
Your team creates a stream route that matches the listening port your devices will use, and an upstream that lists each MQTT broker node with its address and weight. The route is where you decide which server address and port devices connect to and whether remote address or SNI matching applies. Verify by sending a plain TCP connection to the route from a test host and confirming the connection reaches a broker node in the upstream list, using the gateway access log to see which node was selected.
Attach the mqtt-proxy plugin with client ID hashing
Your engineers add the mqtt-proxy plugin to the stream route and set it to balance by MQTT client ID. Each client ID is hashed to a broker node, and when no client ID is present the plugin falls back to the client IP address. This keeps a device's persistent session on one broker. Verify by connecting the same client ID three times from a test host and confirming in the logs that every connection lands on the same broker node. Then connect a second client ID and confirm it may land elsewhere, which shows the hash is working.
Configure TLS for device traffic
Your team decides where TLS ends. With TLS termination, APISIX holds the server certificate devices trust and can optionally re-encrypt to the brokers. With TLS passthrough routed by SNI, the broker keeps the certificate, but APISIX can no longer read the MQTT client ID, so client ID hashing is not available and the plugin must not be relied on for session affinity. Confirm which option you chose, load the certificates, and check with an external TLS client that the presented chain and server name match what devices expect. Ask your security team to review the mTLS requirement before devices are moved.
Verify with MQTT 3.1.x and 5.0 test clients
Your team runs a test client that connects with MQTT 3.1.x and another with MQTT 5.0 through the gateway over the configured TLS setting. Each client publishes to a test topic and subscribes to it, and the session is checked for clean and persistent start behavior. Confirm the publish reaches the subscriber, reconnect with the same client ID and check the session resumes on the same broker node, then kill one broker node in a test cluster and confirm the reconnect behavior your devices will experience. Record the results and the exact plugin and route configuration tested.
Review scaling and operations before fleet rollout
Your team reviews how the broker cluster will grow, because adding or removing a broker node changes the hash target for some client IDs and can force session reconnects. Document the procedure for adding nodes, the monitoring of gateway connections and broker health, and the rollback route you will keep available during migration. Done means a runbook exists, the rollback has been tested on the route, and your operations owner has signed off. Confirm with the APISIX project documentation for your release which stream and plugin behaviors apply to your version.
Securing device traffic and the gateway
Device traffic and the gateway that fronts it need protection at two layers: the MQTT connections that carry device data and the APISIX control surface that decides where those connections go. The cards below cover device identity, certificate lifecycle, the Admin API and etcd, change control for stream routes, and audit logging.
TLS and mTLS for device identity
Terminate TLS on the stream route or pass it through to the broker, and decide where device identity is checked. With mTLS, each device presents a client certificate that identifies it before any MQTT CONNECT packet is read. Confirm which layer enforces the client certificate check, because a passthrough route cannot inspect the payload, and confirm that the broker also validates the identity it is given. Stream routes support TLS termination, mTLS and TLS to upstreams, so test each combination against your broker's own listener settings.
Certificate lifecycle for large fleets
A device fleet multiplies certificate issuance, renewal and revocation work. Decide how devices receive certificates at manufacture or onboarding, how long they are valid, and how a lost or compromised device is revoked and how quickly that revocation takes effect at the gateway. Ask whether your certificate authority supports automated renewal for constrained devices and whether the gateway reloads certificate material without a restart in your deployment mode. Plan a test for expired and revoked certificates before the fleet depends on it.
Protecting the Admin API
The Admin API on port 9180 in traditional mode can change routes, upstreams and plugins, so its exposure is a control plane risk. Keep it on a private management network, bind it to an address that is not reachable from device networks, and require strong credentials for every caller. Confirm the Admin API key handling in your version and rotate it under a documented procedure. Client traffic on port 9080 and the Admin API should never share a reachable interface.
Securing etcd
In traditional and decoupled modes, etcd stores every route and certificate that the gateway uses, so an attacker who writes to etcd can redirect device traffic. Restrict client access to etcd with authentication, encrypt peer and client connections, and limit network reachability to the gateway and control plane nodes. Back up etcd on a schedule and test restoration. If you run standalone mode, the equivalent risk moves to the file or API that holds the configuration.
Limiting who changes stream routes
A stream route change can move a device fleet to a different broker or expose a plain-text listener, so change rights need the same care as production code. Assign separate roles for reading and writing stream routes, keep changes in version-controlled configuration where your mode allows it, and require review before a change reaches production. Record which operator changed which route and when, and verify that a rollback restores the previous client ID hashing and upstream list.
Logging connection metadata for audit
Audit trails for device connections usually need the client ID, the source address, the matched route, the upstream chosen and the connection times, without storing device payloads. Confirm which fields your stream access log and log plugins expose for MQTT traffic in your version, and send them to a log store that retains them under your policy. Check that logs do not capture credentials or certificate private material, and that log access is restricted to the teams that need it.
Scaling broker clusters behind APISIX
The mqtt-proxy plugin pins each device to a broker node by hashing its mqtt_client_id with a consistent hash, falling back to the client IP when no client ID is present. That gives a stable mapping, so a device that reconnects usually lands on the same broker and can resume its session there. The mapping only holds while the set of upstream nodes stays the same. When you add or remove a node, consistent hashing moves only a share of client IDs to a different node, but those devices reconnect to a broker that has no record of their session. Confirm the exact redistribution behavior and the fallback rules in the plugin documentation for your APISIX version before you rely on them in a design review.
Plan for reconnection storms before you change the topology. A node failure or a planned scale-out can drop many devices at once, and every device that reconnects at the same moment creates a burst of TCP handshakes, TLS negotiations and CONNECT packets. Decide how devices back off (with jittered exponential retry in the device firmware), how many connections each broker node can absorb, and whether clean-start or persistent sessions are used for each device class. Session state is the part most often missed: if a broker stores persistent sessions locally and the device lands on a different node, queued messages and subscriptions may not follow it. Ask the broker vendor how session replication or shared storage works in your chosen product.
Broker clustering itself is outside the plugin's scope. APISIX distributes connections, but replication, partitioning of topics, session storage and message redelivery are the broker's responsibility. Run load and failover tests on the target broker product, not on APISIX alone. Test the planned node count at expected device concurrency, simulate loss of one node during a reconnect wave, and measure how long devices take to resume with their subscriptions and queued messages intact. Repeat the test after every change to the node list. Record the results as part of your operational runbook, since they will be the evidence for capacity and recovery decisions.
Limits to confirm in your environment
The mqtt-proxy plugin routes device connections, but it does not decide how your broker behaves behind it. The items below are the points to verify against the APISIX mqtt-proxy documentation, your chosen broker, and your own test results before any device fleet depends on the design.
MQTT feature and QoS pass-through
Confirm which MQTT packet types, QoS levels and retained message behaviors reach the broker unchanged through the proxy. Test each QoS level end to end with a device simulator, including redelivery after a dropped connection. Record which behaviors the proxy handles and which depend entirely on the broker, because the answer determines what your devices can rely on.
Plugin support on layer 4 routes
Stream routes support fewer plugins than HTTP routes, so check the plugin list for stream routes before you plan authentication, rate limits or logging at the edge. Assume that any control you need on device traffic must be verified on the exact APISIX version you deploy. Document the gaps and decide where each control will live instead.
Behavior when a broker node leaves
Test what happens to existing sessions and to new connections when a broker node is removed or fails. Hashing on the client ID keeps a device on one node, but the hash target changes when the node set changes, so a device may reconnect to a node that has no saved session. Confirm the session persistence settings on your broker and how clean-session flags are handled during failover.
MQTT 5.0 properties
Confirm whether MQTT 5.0 properties such as session expiry, topic aliases, user properties and reason codes are preserved through the proxy. Properties that the proxy rewrites or drops can change how your brokers and consumers respond. Run a 5.0 client against the proxy and compare packet captures with a direct connection to the broker.
TLS on device connections
Decide whether TLS terminates at APISIX or passes through to the broker, and confirm how client certificates reach the broker when termination happens at the gateway. Test certificate validation, renewal and the handshake cost for constrained devices. The choice also determines which identity the broker sees for each device.
Scaling and hash distribution
Measure how evenly consistent hashing spreads your real client IDs across broker nodes, since naming schemes can cluster devices onto one node. Confirm how the configuration changes when you add nodes and whether that change can be applied without dropping active sessions. Include this check in your load test plan.
Questions teams ask before proxying MQTT with APISIX
What drives the cost of an MQTT proxying project with APISIX?
The cost follows the size and shape of the broker and device fleet, not the gateway licence, since APISIX is released under the Apache License 2.0. The main drivers are the number of broker nodes behind the proxy, the number of device connections and message patterns to validate, the TLS and certificate work for device traffic, and the testing scope. A quote should split one-time work (design, stream route configuration, TLS setup, load and failover testing, runbooks) from recurring work (upgrades, certificate renewal, monitoring and support). Ask the provider to state which broker versions, device types and failure scenarios are in scope.
How long does MQTT proxying with APISIX take to put in place?
Duration depends mainly on your environment rather than on the plugin itself. The factors that move the timeline are whether stream proxying is already enabled on your APISIX nodes, how many broker nodes and listener ports are involved, how mature your certificate process is for devices, and whether you have a staging environment that resembles production. Teams with an existing APISIX deployment and a test broker cluster usually shorten the work, while greenfield networks, new PKI or firewall changes add time. Ask for a timeline tied to a written scope with named milestones, and confirm which of those depend on your network or security teams.
What do we need to provide before the work starts?
Expect to supply four groups of information before design begins:
- Broker details: the MQTT versions in use (3.1.x or 5.0), the broker software and version, cluster topology, and how sessions are persisted.
- Device identity model: how each device gets its client ID, whether IDs are unique and stable, and how credentials or client certificates are issued and revoked.
- Network design: where APISIX nodes sit, which addresses and ports devices reach, firewall and load balancer paths, and whether devices connect over cellular or other networks with NAT.
- Operations contacts: who owns the broker, the certificate authority and the on-call process.
Missing any of these usually turns into rework during testing.
Can our own team implement MQTT proxying with APISIX internally?
A team with working experience in APISIX, MQTT brokers and TLS can build the basic setup, and the mqtt-proxy plugin runs on stream routes once stream proxying is enabled through the apisix.proxy_mode setting. The harder parts are usually the ones that cross teams: hash-based routing that keeps client sessions on one broker node, failover behavior when a node leaves the pool, certificate lifecycle for thousands of devices, and load testing that reproduces real reconnect storms. Decide whether those skills exist in-house, who will be on call for the gateway, and whether you want a fixed-scope engagement to cover design and testing before your team runs it.
How does TLS for device traffic change the effort?
TLS adds work in design, configuration and operations. Devices may terminate TLS at the gateway, or the gateway may pass traffic through to the broker, and each choice affects how client identity reaches the broker and which checks APISIX can apply. Stream routes support TLS termination, mutual TLS and TLS to upstreams, so the design needs to settle which legs are encrypted and how certificates are issued, rotated and revoked. Devices with limited firmware update paths can make certificate rotation the costliest recurring task, so confirm that during scoping. Ask the provider how certificate renewal is tested and documented.
What should testing cover before MQTT traffic moves to APISIX?
Testing should prove that the gateway does what the device fleet needs under normal and failure conditions. A reasonable scope includes connection and session behavior for each MQTT version in use, client ID hashing across all broker nodes, node addition and removal, broker restarts, TLS handshake and certificate expiry behavior, and a load profile that matches expected connect and publish rates. Confirm with your device vendors which reconnect behavior their firmware uses, since aggressive retries can dominate load. Ask for written test cases, pass criteria and a rollback plan.
Which MQTT behaviors break most often after the proxy goes live?
Problems tend to appear around session continuity and capacity rather than basic connectivity. Watch for sessions landing on a different broker after a node change, clients that reconnect in bursts when a node fails, and TLS certificates that expire on devices that cannot be updated quickly. Hash routing falls back to client IP when no client ID is available, so devices that omit or reuse IDs can be routed unpredictably. Confirm during design how your devices set client IDs, and set up monitoring for connection counts per node and certificate expiry dates before go-live.
Does the work cover ongoing support after go-live?
Implementation is usually a fixed-scope project with agreed deliverables, and ongoing needs are handled separately. DeployOpen offers support, upgrades and continuous testing as retainers, which suit teams that want help with APISIX version upgrades, broker changes, certificate rotation and periodic failover tests. Ask whether a quote includes handover documentation, operator training and a defined support window, and whether recurring support is priced per node, per environment or by hours. The answer determines how much your team must keep in-house after launch.
Rolling out device traffic without breaking the fleet
Move devices to the gateway path in stages, and keep direct broker access working until the gateway has carried real traffic. Start by routing a small, well-understood group of devices through the APISIX stream route while the rest keep connecting straight to the broker. Choose the subset deliberately: include devices from more than one firmware version, more than one site or network, and at least one group with a known history of unstable connections. Before you touch the fleet, confirm that the stream proxy mode is enabled on your APISIX nodes and that the mqtt-proxy plugin is attached to the route you intend to use, since stream proxying is off by default.
Monitor reconnect behavior as the main signal, not only whether devices connect at all. Compare reconnect rates, time to re-establish sessions, and the number of clients that land on a different broker node after a gateway restart or node change. The consistent hash on mqtt_client_id should keep a device's session on one broker, but verify this in your own environment by checking session persistence across broker and gateway restarts, and confirm how your brokers handle persistent sessions and queued messages when a client reconnects through a different path. Set rollback criteria in writing before the pilot begins, and keep a tested way to point the pilot devices back at the broker directly, such as a second hostname or a configuration switch on the device side.
Devices with firmware-fixed broker addresses need a separate plan, because a route change on the gateway does not reach them. Inventory which devices hard-code an IP address and which resolve a hostname, and find out whether each firmware version can be updated over the air. For fixed-address devices, decide whether the gateway can take over the existing IP address, whether DNS can be changed with a low enough TTL to cut over safely, or whether those devices stay on direct broker access for longer. Ask your device vendor to confirm how each firmware version handles DNS changes and certificate validation during the switch, since a failed TLS handshake can look like a fleet-wide outage.
How DeployOpen scopes and delivers MQTT proxying
The sequence turns an IoT broker requirement into a written, fixed scope, then builds and hands over an MQTT proxy layer in your own infrastructure with tests your team can rerun. Each step closes with a named output so both sides know when it is complete.
Confirm device and broker requirements
DeployOpen reviews the device fleet with your team: MQTT protocol versions in use (3.1.x or 5.0), client ID scheme, expected connection counts, message patterns, and which broker software and version you run. Your team confirms the broker topology, including whether brokers are clustered and where they run. The output is a requirements record listing each device class, protocol version and broker endpoint, which becomes the input for the written scope.
Agree the written scope and boundaries
DeployOpen drafts a fixed scope covering stream routes, TLS for device traffic, broker topology changes, test cases, handover material and what is excluded, such as device firmware or broker tuning. Both parties review and sign the scope before build work starts. The output is a signed scope document with deliverables and acceptance criteria, and a quote derived directly from it. Ongoing support can be listed as a separate retainer.
Prepare the gateway and network path
DeployOpen confirms the APISIX deployment mode and host layout in your infrastructure, then enables stream proxying through the proxy mode setting so stream routes are active. Your team opens the required network paths from device networks to the gateway and from the gateway to brokers, and confirms which ports are used for client traffic. The output is a verified connectivity check from a test client to each broker node through the gateway.
Design stream routes and client ID hashing
DeployOpen designs the stream routes that match device listeners, and configures the mqtt-proxy plugin to route on the MQTT client ID with a consistent hash, falling back to client IP when no client ID is present. Your team reviews how client IDs are issued on devices, since changing them moves sessions between nodes. The output is a route inventory and a documented client ID policy agreed with your device team.
Configure TLS for device traffic
DeployOpen sets the TLS model for device connections, choosing between termination at the gateway, mutual TLS, TLS to upstream brokers, or passthrough by SNI, and documents the trade-off for each. Your team supplies certificates and the trust chain, and confirms how device identities are checked. The output is a tested handshake from representative devices, with certificate paths and renewal ownership recorded.
Test sessions, failover and scale
DeployOpen runs test cases that connect clients with both MQTT 3.1.x and 5.0 behavior, confirm that a client ID always reaches the same broker node, simulate a broker node leaving the cluster, and measure how sessions reconnect. Load tests run against the agreed connection targets. The output is a test report listing pass and fail results, observed behavior on node loss, and open issues assigned to an owner.
Hand over runbooks and operating controls
DeployOpen delivers configuration as code, runbooks for adding or removing broker nodes, certificate rotation steps, and the checks your operations team should run after a change. Your team runs one change under supervision to confirm the procedure. The output is an accepted handover package and a closed scope record, with any retainer for upgrades or ongoing testing agreed separately.
