MISP ZeroMQ and Kafka event streaming: when live publishing beats polling
Event streaming lets MISP push events, sightings and tag changes to downstream consumers as they happen, instead of waiting for each consumer to ask for them. It is worth enabling for security operations teams whose detection tools, SOAR playbooks or live dashboards depend on fresh indicators. If those tools poll the MISP API on a schedule, they act on whatever was published at their last successful poll. A shorter interval reduces that lag but adds load on both the MISP instance and the consumer, so polling tends to hit a ceiling that streaming avoids.
MISP can publish these changes over two transports named for this topic: ZeroMQ and Apache Kafka. ZeroMQ is a lightweight messaging library, and it is usually the simpler starting point when one or two consumers subscribe on a network you control. The misp-dashboard, which displays live activity from the instance, relies on the same publishing mechanism, so enabling streaming also affects that view. Kafka is a durable, partitioned log. It fits better when several independent systems need the same stream, when consumers must replay events after an outage, or when your organization already runs Kafka as shared infrastructure.
The choice between the two rests on who operates the consumers and how they recover. Apache Kafka is licensed under the Apache License 2.0 and has no paid edition from the project itself, so its cost is the infrastructure, storage and staff time you commit to running it. Kafka 4.x runs only in KRaft mode and needs Java 17 for brokers, so an older ZooKeeper-based cluster must be migrated first. Kafka also needs deliberate access control. In KRaft clusters, resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true, so define topic permissions for each consumer before connecting it.
Enablement steps, configuration keys and message formats depend on the MISP version and the transport you select, so confirm them against your installed release before planning work. MISP 2.5 is the recommended line for new deployments, and MISP 2.4 is end of life with security fixes only. Ask your MISP administrator or the project documentation to confirm which event types each transport publishes, how sightings and tag changes appear in the payload, and what the consumer must parse. For Kafka, confirm that the cluster version you plan to run is on the supported release list at the time of build.
Before a consumer goes live, test it against a non-production MISP instance and watch for duplicate handling, missed messages after restarts and lag between publication and detection. Decide who owns the consumer, who responds when the stream stops, and how that outage would be detected. Treat the misp-dashboard as an operational view for analysts, not as a record of what was delivered. A streaming design is only as reliable as the monitoring around it, so plan alerts on consumer lag and on publishing failures from the start.
Choosing between ZeroMQ and Kafka for downstream consumers
This table compares ZeroMQ and Kafka on the questions that usually decide the transport: who consumes the feed, whether consumers need replay, how many consumers exist, what your platform team already runs, and who operates the transport. Read each row against your own consumer list, and treat every ZeroMQ behavior marked for confirmation as an open question until the MISP administration documentation answers it. Neither option is better in every case. The right choice depends on your consumers, your outage tolerance and your operating model.
| Decision | ZeroMQ from MISP | Apache Kafka |
|---|---|---|
| Who consumes the feed | Subscribers that connect directly to the MISP publisher, suited to a small set of internal tools | Several teams and services reading a shared topic from a broker cluster you operate |
| Replay after an outage | Confirm in MISP administration docs whether messages queue for a subscriber that is offline | Consumers can re-read stored records within the topic retention period, which you configure |
| Number of consumers | Practical for few subscribers, each connecting to the publisher | Built for many independent consumer groups reading the same data |
| What the platform team already runs | Little extra infrastructure if MISP and its workers are already managed | Fits teams already running KRaft clusters, Java 17 hosts and SASL or TLS |
| Who operates the transport | The team that runs the MISP host, with no separate cluster to maintain | A platform team that runs brokers, KRaft metadata, ACLs, upgrades and monitoring |
| Security controls to confirm | Confirm in MISP administration docs how access is restricted and whether traffic is encrypted | SSL/TLS and SASL are optional, so each listener must be configured on purpose; ACLs apply per resource |
| Upgrade dependencies | Tied to the MISP release in use, so confirm behavior for your MISP 2.5 version | Rolling upgrades from the last three releases; ZooKeeper clusters must move to KRaft before 4.x |
| Live misp-dashboard use | The documented path for the live dashboard, so confirm the exact setup for your release | Dashboard use needs confirmation in MISP documentation before you plan around it |
How MISP publishes events, sightings and tag changes
When ZeroMQ or Kafka publishing is enabled, MISP sends a message to a stream each time a supported change happens, so a consumer learns about new events, sightings and tag changes as they occur. The consumer subscribes to a feed and stays connected, instead of asking MISP for data on a schedule. Before you build anything, confirm in the documentation for the MISP version you deploy which object types and actions are emitted, what the message envelope contains, and how topics are named. These details are defined by the release and the publisher configuration, so do not copy them from an older integration or a blog post.
Streaming differs from the two other ways data leaves MISP. A restSearch call is a pull: the consumer asks for matching attributes or events at a time it chooses, and it must repeat the query to notice changes. Synchronization is a server-to-server relationship in which one instance pushes events to another under distribution rules and sharing agreements. The stream is a notification channel for tools that need to react quickly, including the live misp-dashboard. It does not replace a full copy of the data or the API for fetching context that a message omits, so most designs use a stream for signals and an API call for detail.
Streamed content carries the same distribution and sharing decisions the instance already enforces. A stream is not a way around sharing groups, so a consumer must only receive what its organization is allowed to see. Scope each subscriber to the sharing groups it may access, and test this with an illustrative event marked for a single community to confirm that a subscriber outside that community never receives it. Also confirm how tag changes that alter an event's visibility reach existing subscribers, because a message that is correct at publication can become wrong after an edit.
Plan the transport controls before you connect consumers. Confirm whether your MISP version publishes sightings and tag changes as separate messages, the ordering and delivery behavior you can rely on, and what happens to messages that a consumer misses while it is offline. For Kafka, decide how clients authenticate, using SSL or one of the documented SASL mechanisms, and set topic access with ACLs, because resources without ACLs are limited to super users unless the cluster explicitly allows otherwise. Write these decisions down with the sharing group mapping so the security review can check them together.
Setting up a streaming path step by step
This sequence takes a MISP instance from a confirmed version and install method to a consumer that receives events, sightings and tag changes over ZeroMQ or Kafka. Each step ends with a check, so a fault is found where it was introduced.
Confirm the MISP version and install method
Start by recording the MISP version and how it was installed, such as the provided installer on Ubuntu 24.04, Docker images, Ansible roles, Puppet or RPM packages. MISP 2.5 is the version recommended for new deployments, and 2.4 is end of life with security fixes only, so an existing 2.4 instance should be scheduled for upgrade before streaming is built on it. The platform team owns this step. It is done when the version and install method are written down and the publisher documentation for that exact version has been read, because enabling publishing depends on the release.
Decide the transport and its owners
Choose ZeroMQ or Kafka based on what will consume the stream. ZeroMQ suits a direct subscriber close to the MISP host, while Kafka suits several downstream consumers that need a shared log they can read at their own pace. Kafka is an Apache Software Foundation project under the Apache-2.0 license, with no paid edition from the project itself, but any broker you run still needs sizing, access control and staff to operate it. Record the consumer list, the owner of each side and the network path between them. The step is complete when the transport and owners are agreed in writing.
Enable the publisher and confirm message types
In the MISP administration settings for the deployed version, enable the publisher for the transport chosen in the previous step. Confirm the exact setting names in the MISP documentation for that version before changing anything. Then confirm which message types the publisher emits, covering events, attributes, sightings and tag changes, and record each one as emitted or not. Check: in a test instance, add a sighting and change a tag on an existing event, then confirm each action produces a message in the publisher's output.
Stand up the broker or socket endpoint
For Kafka, deploy a cluster in KRaft mode using a supported release such as 4.3.1, 4.2.2 or 4.1.2, with brokers on Java 17. Enable TLS and one of the documented SASL mechanisms, such as SCRAM-SHA-512, and configure ACLs before any consumer connects, because KRaft clusters deny resources without ACLs to everyone except super users by default. For ZeroMQ, bind the endpoint to an address the consumer can reach and confirm the port in the documentation for your MISP version. Check: from the consumer network, confirm the listener is reachable and the advertised address resolves.
Connect one test consumer
Attach a single consumer from the planned downstream list, using a dedicated credential with read-only access to the topic or socket. Use a client that supports the chosen transport, and for Java clients confirm the runtime meets the minimum version in the Kafka upgrade notes. Start by writing raw messages to a file so the payload can be inspected before any downstream logic depends on it. Check: after a test event is created in MISP, the consumer receives a message. If nothing arrives, check the listener step first, then the publisher settings.
Verify event content in a test instance
Create a test event in a non-production MISP instance with known attributes, tags, a distribution level and a TLP tag. Compare the message the consumer received with the event as shown in MISP, field by field: identifiers, attribute types and values, tags, distribution and timestamps. Confirm the consumer parses the format it expects. Repeat with a sighting and a tag change, and check that each produces a message with the expected content. Record any missing or altered field, and the MISP setting that controls it, before live data is sent.
Record the path and retest after upgrades
Write down the MISP version, transport, listener address, credential owner, emitted message types and test results. Keep the test event and sample messages so later changes can be compared against them. Check: someone outside the build team can trace an event from MISP to the consumer using only this record. Repeat the sighting and tag change checks after each MISP or broker upgrade, since message types and formats can change between releases.
Feeding the live misp-dashboard and other consumers
The misp-dashboard is a visual consumer that shows activity as it happens, including new events, attributes, sightings and tag changes. It needs a steady feed of recent activity and does not need the full event history, so its requirements center on freshness and staying subscribed. Confirm which message types and format the dashboard version you run expects, because compatibility depends on the installed MISP version.
- Subscribe only to the message types the dashboard displays.
- Place the dashboard on a network path with predictable latency to the MISP publisher.
- Expect gaps in the display after an outage, since it shows what arrives while it is running.
- Review the dashboard's access so it reads only the events your viewers are cleared to see.
Check the dashboard's supported MISP version and message format against your installed release before you enable the feed.
Security tools use streamed indicators and sightings for detection and automated response, so they need complete, well-described data. They depend on context such as tags, TLP markings and distribution levels to decide what to act on. A missed indicator can mean a missed detection, so message loss has more operational weight here than on the dashboard.
- Filter by tag, TLP marking or event type so only actionable indicators reach the tool.
- Confirm how the consumer acknowledges processing, because that decides what is redelivered after a crash.
- Plan a replay path through the MISP API for periods when the consumer was offline.
- Test that enrichment steps and playbooks handle duplicate messages without creating duplicate tickets or blocks.
Confirm with the SIEM or SOAR vendor which MISP message format and version their connector supports before you build the integration.
Internal services need the most explicit contracts because nothing outside your team defines their behavior. They should version their message schema, handle errors deliberately and process messages idempotently. When the stream is carried by Kafka, the consumer controls its own position, so processing logic and offset handling must be designed together.
- Use consumer groups so several instances share the work and each group processes each message once.
- Commit offsets only after processing succeeds so a crash does not silently skip messages.
- Make handlers idempotent, because redelivery and duplicates are normal in streaming systems.
- Grant read access explicitly. In KRaft clusters, resources without ACLs are limited to super users unless allow.everyone.if.no.acl.found is set to true, so leave that setting off and grant each service only the topics it needs.
Check retention, consumer group and authorization settings on the Kafka side before any custom service goes into production.
The two transports behave differently when a consumer stops. ZeroMQ publish and subscribe does not hold messages for absent subscribers, so anything published during an outage never reaches that consumer. Kafka keeps messages for its configured retention period, so a restarted consumer can resume from its last committed position if topics and groups are set up correctly.
- For ZMQ, plan reconciliation. Consumers that reconnect still miss messages sent during the gap, so schedule a comparison against the MISP API for the outage window.
- For Kafka, confirm offset behavior. Restarts resume from committed offsets, and auto-commit settings determine how far that position can run ahead of processing.
- Monitor consumer lag so you know when a consumer has fallen behind.
- Record each outage window so you can backfill the affected period from MISP.
Test a deliberate stop and restart of each consumer in a non-production environment, then record what was missed and how it was recovered.
Securing the streaming path and consumer access
Event streams carry indicator data that MISP may share only under specific distribution rules, so the security review covers three things at once: the link between MISP and the broker, the authorization model inside the Kafka cluster, and the access each consumer receives. The cards below follow that path from the publisher to the consumer and then to day-two operations.
Encryption and authentication to the broker
The link between MISP and the broker carries event data, so confirm how the publisher is encrypted and authenticated in your MISP version before go-live. If the ZMQ publisher offers no transport security of its own in your setup, plan a private network path or tunnel and document it. For Kafka, the producer should authenticate with its own identity, never a shared account used by other systems. Test the connection with a non-sensitive event first.
Kafka SSL and SASL options
Kafka authenticates clients and brokers with SSL or SASL and encrypts traffic with TLS, but security is optional, so a cluster can run an unsecured listener beside a secured one. Confirm that the MISP producer and every consumer connect only to secured listeners. The documented SASL mechanisms are GSSAPI (Kerberos), PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER. Choose the one your identity team can operate, and use PLAIN only over TLS.
KRaft ACLs and default deny
In a KRaft cluster, the StandardAuthorizer stores ACLs in the cluster metadata log. Resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true, so keep that setting at its restrictive value and grant each MISP producer and consumer group explicit rights. Limit super user membership to administrators. After each ACL change, verify access with a test principal, and confirm the exact operations each role needs against the Kafka authorization documentation for your release.
Who may subscribe to which topics
Scope read access by topic and consumer group, so each downstream system sees only the MISP topics it needs. Map each consumer to a dedicated principal, and give the live dashboard, SIEM connectors and custom tools separate groups so one consumer cannot read another's topics. Keep write rights with the MISP producer alone. Review subscriptions whenever a consumer changes owner or is retired, because unused principals stay valid until someone removes them.
Distribution controls carried into the stream
MISP distribution levels and sharing groups decide who may receive an event, and the stream must respect that model. Confirm which events, attributes and tag changes the publisher emits, and whether those items are already permitted to leave your instance. Events restricted to a community should reach only topics or brokers that community can access. Verify the mapping with test events at each distribution level before enabling the feed for consumers.
Credential rotation, segmentation and access logging
Rotate SCRAM passwords, OAuth client secrets and TLS certificates on a schedule your team can run without downtime, and rehearse one rotation in a test cluster first. Place brokers in a network segment reachable only from the MISP host, approved consumers and administrators. Log consumer connections and authorization denials at the broker, and store those logs where consumer owners cannot edit them. Review the logs against the list of approved principals on a fixed cadence.
Running Kafka versions and prerequisites you must plan for
The first decision is which Kafka release line your cluster will run, and that choice comes before any MISP configuration. Apache Kafka is licensed under Apache-2.0, and the project offers no paid edition, so the cost question concerns your own staff, infrastructure and support arrangements. The Kafka downloads page lists 4.3.1, 4.2.2 and 4.1.2 as the supported releases, and earlier versions are archived. Confirm which of these your organization will standardize on, and confirm that choice in your own environment, because the release sets the Java runtime, the metadata mode and the upgrade path for everything that connects to the broker.
Kafka 4.x runs only in KRaft mode. ZooKeeper mode has been removed, so any existing cluster that still depends on ZooKeeper must migrate to KRaft before you upgrade it to 4.x. The Kafka upgrade notes also recommend that KRaft clusters older than 3.3.x move to 3.9.x before they move to 4.x. Ask the team that operates the cluster which metadata mode and minor version it runs today. Schedule any migration as its own change with a rollback point, and do not let the MISP publishing project set the timeline for it.
Java requirements land on different hosts, so inventory them early. Since Kafka 4.0, brokers, Kafka Connect and tools require Java 17. Clients and Kafka Streams applications need at least Java 11. Downstream consumers of the MISP event stream are the workloads most likely to run older runtimes, so list every consumer, its owner and its JVM version before the broker moves. Any consumer that cannot reach the required runtime should appear in the risk register with an owner and a date for resolution, because a broker upgrade will not wait for it.
Kafka supports rolling upgrades from the last three releases to the latest version, so a cluster that sits outside that window needs an intermediate step. Plan the upgrade window with the people who run the MISP publishing side, since broker restarts can interrupt publishing. Before cutover, confirm the exact steps for your target release in the upgrade notes. Also confirm that the principal used by MISP has explicit access control entries, because in KRaft clusters resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true.
Limits to confirm before you commit to streaming
Streaming changes what your downstream systems depend on, so several behaviors need confirmation before you build on them. The six items below cover MISP's message types, deployment support, volume handling, delivery semantics, dashboard compatibility and the Kafka release you run. Check each one against your own MISP version and environment, not only the documentation.
Message types in your version
Confirm which event, sighting and tag change message types your installed MISP version publishes over ZeroMQ and Kafka. Message names, fields and the actions that trigger a message can differ between releases, and consumers written against one version may fail or drop data after an upgrade. Ask the team that owns your consumers to list the message types they parse, then compare that list against a test instance running the same MISP release and record any gaps before go-live.
Publishing support by deployment method
ZeroMQ and Kafka publishing depend on how MISP was installed. Confirm that your method, whether the official installer, Docker images, Ansible roles, Puppet modules or RPM packages, exposes the publishing settings and any client libraries you need. Installer-based and container-based deployments can differ in where settings live and how services restart, so verify the enabled options in the running instance and do not assume the defaults match the documentation for your chosen method.
Behavior under high event volume
Find out how the publishing feed behaves when many events, sightings and tag changes arrive in a short window. Ask for observed throughput of your MISP release on hardware comparable to yours, how messages are buffered when a consumer is slow or offline, and what happens to MISP processes when the broker is unavailable. Run a load test that mirrors your sync and automation patterns, because bulk imports and feed pulls generate very different traffic from analyst edits.
Replay and delivery guarantees
Confirm what delivery guarantees apply to messages from MISP and whether consumers can replay missed messages. Publish-and-subscribe feeds are not always designed to retain history, so a consumer that restarts or loses its connection may miss updates unless you add a reconciliation step. Ask the project, your vendor or your own testing which messages persist, for how long, and how to rebuild state from the MISP API after an outage, then document that recovery procedure before depending on the stream.
misp-dashboard version mapping
The misp-dashboard reads the live feed and displays it in a separate application, so its version must match the MISP release that publishes the messages. Confirm which misp-dashboard release is compatible with your installed MISP version, and check that its Redis and ZeroMQ connection settings match your deployment. Pin both versions in your change records so an upgrade of one component does not silently break the live view for analysts.
Kafka release and security behavior
Kafka's security and authorization behavior is described in the documentation for the specific Kafka release, so check the release notes for the version you run. Confirm which authentication mechanisms and TLS settings that release supports, and how ACLs apply in KRaft mode. Under the documented default, resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true, so test that producers and consumers can connect before you open access to them.
What does MISP event streaming cost and take to set up?
Is MISP event streaming free to set up?
The software is free to use, but setting up streaming still costs money. MISP is licensed under the GNU AGPL version 3, and Apache Kafka is licensed under Apache-2.0 with no paid edition from the project itself. The spending comes from infrastructure (servers or cloud resources for MISP, the broker and consumers), broker operations (monitoring, upgrades, capacity and backups), consumer development, testing, and ongoing support. Infrastructure and operations recur every month, while design, build and testing are usually one-time work. The MISP project offers paid support, and a number of third-party providers offer commercial support. Ask any quote to list each component separately so you can see who pays for what.
How long does setting up MISP streaming usually take?
The duration depends on four things: the transport you choose, the number of consumers, whether your team already has Kafka or messaging skills, and whether security controls such as encryption and access control must be built during the project. A single consumer on an existing messaging platform will take far less time than several consumers feeding a new broker cluster with its own authentication and authorization. No fixed timeline applies to every environment. Ask for a timeline in the written scope, and ask the provider to state which of the four factors it assumed.
Should we use ZeroMQ or Kafka for MISP event streaming?
Choose based on the consumers you need to serve and the skills your team already has. ZeroMQ is a lighter option with fewer components, but each consumer must be written to handle the messages it receives. Kafka adds a broker cluster you operate, along with its own sizing, retention, access control and upgrade work, in return for a shared, durable log that several consumers can read. If your team already runs Kafka, the added operating load is smaller. Confirm in the MISP documentation which events, sightings and tag changes each transport publishes and how the live misp-dashboard receives them before you commit to either option.
What does running a Kafka cluster for MISP involve?
Start with the version and mode. Apache Kafka 4.0 supports only KRaft mode, and ZooKeeper mode has been removed, so a ZooKeeper-based cluster must migrate to KRaft before you upgrade to 4.x. The upgrade notes recommend that KRaft clusters older than 3.3.x move to 3.9.x first. Brokers, Kafka Connect and tools require Java 17 from Kafka 4.0 onward. Rolling upgrades are supported from each release in the past year. Plan for these steps in the design, because they determine when you can take the cluster to a supported version and how much testing each upgrade needs.
Can our own team set up MISP event streaming without outside help?
Yes, if your team has MISP administration experience and messaging experience, particularly with the transport you choose. Your team would handle the MISP configuration, the broker or ZeroMQ setup, the consumer code, security controls and testing. The main risk is underestimating the security and operations work, such as encrypted listeners, authentication, access control lists and monitoring of consumer lag. Before you decide, have your team list the tasks in a written plan, then compare that plan with the scope an outside provider would propose. An outside engagement is set up as fixed scope, with deliverables and timelines agreed in writing before work starts.
Which security controls must we build for Kafka-based MISP streaming?
Several controls are optional in Kafka by default, so you must decide to enable them. Kafka supports SSL/TLS to encrypt traffic between clients, brokers and tools, and clusters can run unsecured or mix secured and unsecured listeners. Kafka authenticates clients with SSL or SASL, and the documented SASL mechanisms are GSSAPI (Kerberos), PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER. In KRaft clusters, the StandardAuthorizer stores ACLs in the metadata log, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Also confirm with your own policy whether the events sent to consumers must respect the sharing and distribution settings that apply inside MISP.
What should a quote for MISP streaming include?
A quote should follow from a written scope that names the work, not a generic package. Ask for the transport and the reason for it, the number and type of consumers, where the MISP and broker components will run, which security controls are included, how testing will be done, what documentation and handover you will receive, and what happens after go-live. DeployOpen does not publish a price list. Most engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts. Ongoing support, upgrades and continuous testing are available as retainers, so ask separately what recurring support would cost after the build.
What usually goes wrong with MISP streaming after go-live?
The most common problems appear after the first build is finished. Consumers fall behind the event stream, so you need lag monitoring and an agreed response. Broker certificates expire, and TLS settings drift during upgrades. ACL changes block a consumer that worked before, especially in KRaft clusters where unlisted resources are restricted by default. A MISP version upgrade can change how events are published, so retest every consumer after each upgrade. Also confirm that the live misp-dashboard still receives the messages it needs after any transport change. Ask the provider who owns each of these checks after handover.
How DeployOpen scopes and delivers MISP streaming
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure. For MISP ZeroMQ and Kafka event streaming, that means agreeing which events, sightings and tag changes leave the instance, who consumes them, and where the broker runs before any configuration changes are made.
How we scope
- The written scope lists which MISP events, sightings and tag changes publish, and each ZMQ or Kafka consumer.
- Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts.
- Ongoing support for the streaming setup can be arranged as a retainer after handover.
How we build
- Confirm the MISP settings required for ZeroMQ or Kafka publishing, and record each enabled setting in the build notes.
- Deploy Kafka in KRaft mode, since Kafka 4.0 and later supports only KRaft and ZooKeeper mode has been removed.
- Configure TLS listeners and SASL authentication, with ACLs for the MISP producer and each downstream consumer group.
How we hand over
- Documentation covers enabled MISP settings, topics, consumer groups, ACLs, Kafka version and upgrade path.
- Test evidence shows that events, sightings and tag changes reach each consumer in the expected format.
- Runbooks cover restarting the publisher, rotating credentials and recovering consumers after a broker outage.
How we support
- Kafka upgrades follow the project's rolling upgrade path, which supports moving from the last three releases to the latest.
- MISP upgrades are planned separately, since MISP 2.4 is end of life and MISP 2.5 is recommended for new deployments.
- Retainers can cover ongoing monitoring of the publisher and broker, with alert handling agreed in the written scope.
