MISP threat intelligence automation with Shuffle

Choose the delivery path before you build the first workflow, because it sets how fast indicators reach Shuffle, how much load lands on MISP, and what happens after an outage. Three options exist. The ZMQ forwarder extension pushes event and attribute updates as MISP publishes them, which suits watchlist pushes that should follow an analyst's publication within moments. Scheduled polling calls the restSearch endpoint from the OpenAPI-described API on a timer, either with direct HTTP requests or through the bundled PyMISP library, and pulls whatever changed inside a time window. MISP's own synchronization copies events between MISP instances, so Shuffle reads from a second MISP node that receives the data.

Latency and load pull in different directions. Streamed updates give the lowest delay, and each message is small, but the forwarder stays connected for as long as Shuffle depends on it. Polling delay is roughly the interval you set, and load grows with query frequency, result size and how wide the time window is. Narrow windows, tag filters and a sensible interval keep the query small. Synchronization adds the most operational weight, since you run and patch a second MISP instance, and its timing follows the sync configuration and distribution rules rather than your Shuffle schedule. Ask your MISP administrator which sharing groups and distribution levels the sync will carry before you count on it for coverage.

The extension dependency and missed-message behavior differ more than teams expect. A streamed design depends on a forwarder that must track MISP upgrades, and any message sent while the forwarder is down is lost unless something replays it. Polling depends only on the API that MISP already exposes. If Shuffle stores a timestamp cursor and polls with a small overlap, a restarted workflow can backfill the gap and catch edited attributes. Synchronization recovers from outages through MISP's own sync mechanism, but you must check sync status and failed pulls, because a broken link can go unnoticed until someone searches for an indicator that never arrived.

Shuffle's licensing and runtime add a cost that is easy to miss. Licensing counts app runs, which are executed actions, and triggers and skipped actions do not count. A scheduled poll that starts a workflow every cycle, even when nothing changed, therefore consumes runs for no benefit. Put a check for new or changed data at the start of the workflow so that only real changes reach the enrichment and push steps. Confirm with the vendor documentation how the ZMQ message formats are structured, whether the forwarder has any event or rate limits, and how the open-source edition limits apply to your version. Test each path against a MISP instance running the version you intend to operate.

Place the forwarder and the workflows that act on its messages on runtime nodes you can afford to lose, and choose that placement from the Shuffle architecture you already run. A single Docker Compose server keeps the backend, the runtime and the forwarder on one host, which is simple to operate but makes that host the failure domain for the whole threat intelligence path. Docker Swarm spreads the services across several servers. The hybrid option, with Shuffle Cloud and a local Orborus, needs no inbound firewall ports because the local runtime reaches out, which often suits a MISP network with tight ingress rules. Kubernetes with Helm charts fits teams that already operate clusters. In every case, confirm that the runtime location can reach the MISP ZMQ publisher and the MISP API, because a placement that cannot see MISP is not a placement at all.

For high availability, Shuffle's configuration guidance starts with an OpenSearch cluster rather than a single OpenSearch node, since workflow execution records and saved app credentials are stored there. Plan the node count, storage and snapshot policy with the current Shuffle configuration documentation in hand. Memcached is optional and only useful when you want a shared cache across backends. When several backends run, NFS holds the shared files, which makes the NFS export a dependency you must protect with its own availability plan, permissions and backups. Scale the backend and frontend across all servers, and run more than one Orborus node so that workflows keep executing if one runtime fails. Set SHUFFLE_ENCRYPTION_MODIFIER to the same value on every backend, because changing it breaks all saved app authentications.

A single forwarder consumer is a point of loss. If the process, its host or its network path stops, indicator updates published during the gap can be missed, and the watchlists on firewalls, EDR and the SIEM silently fall behind. Before you rely on it, confirm with the extension's maintainers whether it buffers messages, resumes from a checkpoint after a restart, or replays anything at all. Then cover the gap yourself: alert when the consumer has been silent longer than your expected publishing rate allows, and write a runbook that re-pulls changed events and attributes through the MISP REST search API for the outage window and feeds them through the same workflow. Test that runbook before an incident, not during one.

Recovery and failover both produce duplicates. A restarted consumer, a failover to a second consumer, or a handover between Orborus nodes can deliver the same update more than once, so the workflow design must treat repeated messages as normal. Key every action on the MISP event and attribute identifiers together with the update timestamp, check whether the indicator already exists before writing it, and make each watchlist write replace the entry with the same key. Test this by replaying one message twice and confirming that the firewall, EDR and SIEM lists end in the same state. Duplicates also cost money under Shuffle's licensing, which meters app runs, so repeated executions add to the metric. Ask any partner quoting this work how they measure and cap duplicate runs.

Test the indicator flow in stages, and promote it to production only after each stage has produced evidence you can file. Start in a non-production MISP instance that matches production as closely as you can manage, using the same MISP line you run in production. MISP 2.5 is the recommended version for new deployments, so confirm which branch your production instance runs before you build the test. Publish a dedicated test event that contains only the attribute types your workflow handles, using reserved or clearly fictitious values such as a documentation-range IP address and a test domain. Give the event a name that no analyst could mistake for live intelligence, restrict its distribution so it cannot leave the test instance, and record its UUID before you publish.

Next, confirm that Shuffle receives and parses the event correctly. Check that the ZMQ forwarder delivers the publish message to the Shuffle workflow, and that the parsing step extracts each attribute with its type, value and to_ids flag. Compare the parsed output with the JSON export of the same event from MISP, and confirm that unexpected attributes are dropped rather than passed to targets. Then run each target in monitor-only mode. Each firewall, EDR and SIEM watchlist action should log the indicators it would have pushed, and no live list should change. Confirm that each target receives only the attribute types it is meant to handle, and that Orborus started the app containers at the versions you expect.

Verify removal and expiry before any target can enforce anything. Unpublish or delete a test attribute, or let an expiry condition pass if your design uses one, and confirm that the workflow removes the indicator from every monitor list and logs the removal. Edit a test value as well, and confirm that the new value replaces the old entry so that both versions cannot remain on a list. Next, test failover. Stop one Orborus node while a test event is publishing, then confirm that the remaining node completes the workflow executions. Check that the surviving node is the one picking up the jobs, because a standby that never runs can pass a simple success check. Confirm that no indicator is lost or duplicated during the switch.

Keep a change record that an engineer or auditor can follow without asking you questions. Include the test event UUID and its JSON export, the MISP and Shuffle versions and the Docker tags you deployed, since the Shuffle documentation recommends the latest tag for stable releases, so pin and record the tag you actually tested. Add the workflow revision, an export of the workflow definition with credentials excluded, the execution IDs and timestamps for each stage, and the target-side logs showing monitor-only entries and removals. Record the failover timeline with the stopped node identifier, and the approver names and dates. Do not place the SHUFFLE_ENCRYPTION_MODIFIER value or app credentials in the record. Note where they are stored instead. This evidence supports readiness, and it is not an audit. Ask your assessor which artifacts they expect before you start testing.