Shuffle scaling with Docker Swarm and Kubernetes

The scaling question for Shuffle starts with one chain of components. The frontend and the Go backend API, which listens on port 5001, handle users and workflow definitions, and OpenSearch stores the documents behind them. Orborus is the runtime piece. It polls the backend for jobs, and for each workflow execution it starts a worker container. The worker then starts the app containers that perform the individual actions in the workflow. Load therefore moves in stages: the backend and OpenSearch carry the control plane and the bookkeeping, while the runtime hosts carry the containers that actually execute work. Sizing the backend alone will not tell you how much the runtime hosts need.

For planning, the useful question is where each container lands. In Docker Swarm mode and in Kubernetes mode, the scheduler decides placement, so the number of workers and app containers that a given host receives depends on the orchestrator, the resource requests set for those containers, and any placement constraints you define. Ask your implementation partner or the Shuffle team to show you, in your own environment, which hosts receive worker containers and which receive app containers, and how a failed host is handled. Do not assume that a second Orborus node automatically spreads load evenly. Confirm how each Orborus node claims jobs and how that interacts with your chosen orchestrator.

Many short app runs and a few long ones stress the runtime differently. Short runs create many container starts, which means frequent image pulls if images are not cached on the host, heavier churn in the Docker daemon or Kubernetes node, and bursts of scheduling activity. Long runs hold memory and CPU for longer and reduce the number of containers that fit on a host at once, but they generate fewer starts. A workflow that calls one API hundreds of times per hour and a workflow that runs a single long enrichment job can need very different host capacity even when the total number of app runs looks similar. Pull a sample of your expected run profile before deciding host sizes and registry placement.

Concurrency and placement settings are where assumptions cause the most trouble. The configuration documentation describes the environment variables and options that control how many workflows and apps can run in parallel, how Orborus behaves across nodes, and how images are pulled and handled in the runtime. Those names, defaults and limits must be read from the current configuration docs for the release you deploy, not copied from a blog post or an older version. Record the values you set, the release tag you run, and the test result that shows the setting behaves as you expect under load.

Before go-live, run a controlled load test that mirrors your real mix of short and long runs, watch image pull times, host memory and CPU, and failed or queued executions, and then repeat the test after any change to concurrency, placement or registry configuration. Keep the frontend, backend and worker images on matching releases, because a mismatch between components can show up as failures that look like scaling problems. Plan for the case where a runtime host or Orborus node stops during a busy period, and confirm in writing how queued workflows resume.

Every Shuffle app runs as a versioned Docker container, so the speed at which a scaled run starts depends on how quickly each runtime node can pull the app image it needs. When Orborus starts a worker for a workflow execution and the worker starts app containers, any node that lacks a required image must fetch it before the step can run. On a long-lived host the image is usually already cached. On a new host added to a Docker Swarm or Kubernetes cluster to absorb load, the first run pays the full pull cost. That makes the registry path a scaling decision, not just a packaging detail, and it should be designed before you add nodes.

The first decision is where the registry lives. Every runtime node must be able to reach it over the network and authenticate if the registry is private. Confirm this from each node, not only from the server that builds images, because Swarm workers and Kubernetes nodes often sit in different subnets or security zones than the Shuffle backend. For hybrid setups where Orborus runs inside a private network, the registry should be inside that network too, so app pulls never depend on an outbound path to a public registry. Ask your team which registry will serve each runtime location, who owns it, and how it is backed up.

For private networks, mirror the images your workflows depend on into a registry you control. A mirror removes dependence on external availability during a scale-out event and keeps pulls on fast internal links. Decide which images are mirrored, how the mirror is refreshed when Shuffle or an app is updated, and who approves a new image before it reaches production nodes. Pin app image versions explicitly so every node runs the same build of an app. A floating tag can resolve to different digests on different hosts, so one worker may execute a different app build than another for the same workflow. Record the version of each app in your change process and verify it on every node after a rollout.

The Shuffle documentation recommends using the latest Docker tag for stable releases and the nightly tag only for experimental changes. Apply that split to your own environment: production runtimes should track stable tags that you have tested, and nightly images should stay in a test location where failures cost little. Do not use nightly images in a runtime that serves live security automation. Confirm the exact tag names, registry configuration, credential handling and any image caching options in the configuration documentation for the release you deploy, because these settings change between versions. Check how registry credentials are supplied to Swarm services or Kubernetes pull secrets, and keep those credentials out of workflow definitions.

Before you rely on this design at scale, run a controlled test. Add a new node, schedule a workflow that uses several apps, and time the first execution against a warm node. Repeat the test after each registry change or app upgrade. Also simulate loss of the registry and confirm which runs fail and how the team is alerted. A registry that works for a single server may still be a bottleneck when many workers pull at once, so measure concurrent pulls as well as single pulls. Ask whoever plans your implementation to include this test in the scope, since it reveals problems that a single-node check will miss.

A scaled Shuffle deployment is proven only when a written test shows that workflows keep completing after a runtime node stops and that every run can be traced to the host that executed it. Start with a fixed set of workflows, agreed with the business owners, run at the target concurrency from your scope document. Use the same set before and after the failure so the results are comparable. For each run, record when it was queued, when it started, when it finished and whether it failed. Queueing is the first signal. If runs wait longer after the failure and do not return to the pre-failure pattern, the remaining capacity is too small for the load, and the sizing needs to change before go-live.

Before any failure is injected, confirm placement. In Docker Swarm mode, list the tasks for the backend, Orborus and worker services and map each worker and app container to its host. In Kubernetes mode, map each worker and app pod to its node. Check that the second Orborus node actually receives jobs during the run. An idle Orborus node will look healthy in a dashboard but provides no failover. Then test scale-out: add a new node, schedule a run that needs each app image, and confirm the image pulls cleanly from your registry on that node. Registry authentication and network reachability from new nodes are common failure points, so record pull success, the image tags used and any pull errors.

Next, stop one Orborus node during a steady run. The Shuffle HA guidance calls for more than one Orborus node so workflows keep running if one fails, which means a single-Orborus setup cannot pass this test. Watch for runs stuck in a running state, and confirm that queued jobs are picked up by the surviving node. Repeat the exercise by stopping one Swarm worker or one Kubernetes worker node that hosts app containers. Run this in an approved change window with a documented rollback, and record the exact time of each stop and restart. Note which runs were in flight on the stopped node and what state they ended in, because those runs are the ones most likely to be lost.

Once Shuffle runs across more than one server, the first thing to watch is whether workflow execution is keeping up. Track queue depth and run failures per node, so that a single Orborus instance or worker host falling behind is visible before users notice delayed playbooks. Compare failure rates across nodes rather than only in aggregate, because a node with a stale image or a full disk often fails in a way the others do not. Record a baseline during the first weeks after go-live, when load is still close to what the concurrency settings were tested against, and keep that baseline with the handover pack so later changes have something to be measured against.

Watch host CPU, memory and disk on every server that runs worker and app containers, not only on the backend hosts. Worker containers are short-lived, so disk pressure often comes from unpruned images and stopped containers rather than from data. Image pull errors deserve their own alert, since a registry outage or a missing tag on one node causes runs to fail only where that node is scheduled. Confirm that each Orborus node shows as registered with the backend, and check that a node that drops out of the list is detected and reported. Ask your platform team which Docker Swarm or Kubernetes events and node conditions they already collect, and extend those rather than building a parallel system.

Back up OpenSearch data and the SHUFFLE_ENCRYPTION_MODIFIER secret as separate items, stored in different places with separate access controls. The secret is required to decrypt saved app credentials and API keys, so a restored OpenSearch snapshot without the matching secret leaves integrations unusable, and a secret kept only beside the database defeats its purpose. Confirm the backup scope with the vendor documentation for your version, and test a restore into a non-production environment. When you upgrade, align the frontend, backend and runtime images to the same Shuffle release. The docs recommend the latest Docker tag for stable releases and the nightly tag only for experimental changes, so record which tag each environment runs. The current release at the time of writing is v2.3.0, published on September 28, 2026. Confirm the edition before scaling too. Paid tiers differ in scaling limits, and since v2.1.1 those limits are enforced, so a cluster sized on the open-source assumption may stop executing runs above the limit. Confirm the limit with Shuffle before adding nodes.

The handover pack should let a new engineer rebuild the runtime without asking the original team. Include the node inventory with hostnames, roles and Orborus registration identifiers, the placement standard that states which workloads may run where, and the registry layout with image names, tags and the cache or mirror location. Add the concurrency settings with the values chosen, the reason for each and the test that justified them, and the results of load and failover tests with the dates and Shuffle version tested. Note any Kubernetes Helm values or Swarm stack files used.

Write a runbook for adding and removing a node. Adding a node means checking the edition limit, pulling the aligned images, registering Orborus, confirming it appears in the node list, running a test workflow pinned to that node, and then updating the inventory. Removing a node means draining it, waiting for running workflows to finish or be rescheduled, deregistering Orborus, confirming no pending runs reference it, and removing its entry from the inventory and monitoring. Assign an owner for each step and review the runbook after every Shuffle upgrade, because placement behavior and image requirements can change between releases.