SOAR RFP questions

A good partner starts with the alert sources your team already handles most often and the decisions analysts repeat by hand, not with a catalog of playbook templates. In discovery, you should expect a review of alert volumes, the tools that raise them, the enrichment lookups analysts run, and the response steps that need an approval. Each first-wave playbook should name one alert source, the enrichment and response actions it uses, and the outcome it is expected to improve. Ask for that list in writing before build starts, and ask which candidates were rejected and why.

Measure each playbook against a baseline taken before it goes live. Common measures are time from alert to triage decision, share of alerts closed automatically with a documented reason, analyst minutes saved per alert type, and the rate of escalations that turn out to be real incidents. Set the targets and the measurement method in the statement of work, including where the data comes from, such as Shuffle execution history or the case system. Avoid measures that reward closing alerts quickly, because they can hide missed detections.

False positives should be tracked by rule, by alert source and by playbook, with a named owner who reviews them on a fixed cadence. Early versions of a playbook should auto-close only the cases with a narrow, well-understood pattern, and route everything else to an analyst. Ask the partner how they record each automated closure, how they sample closed alerts to check them, and how they tune enrichment thresholds when a source changes. Shuffle execution records can support this review, but the review process itself has to be agreed with your SOC.

Playbooks that isolate hosts, disable accounts, block addresses or change firewall rules should run behind explicit approval gates and be written so a human can see what will happen before it does. Ask the partner which actions are read-only, which need one approver, and which need two, and who holds those approvals outside business hours. Production-facing steps should be tested in a non-production environment with the same connectors and permissions, and each change should go through your change control process. Confirm rollback steps for every destructive action before go-live.

A partner that commits to a fixed playbook count and delivery date before discovery is asking you to accept unknown scope. The right order is discovery first, then a written scope that lists each first-wave playbook, its alert source, its actions and its measure, with the timeline attached to that list. The number and timing depend on how many alert sources are connected, how many systems each playbook touches, how many approval paths you need, and how clean your existing data is. Ask the partner to show how the scope would change if a playbook turns out to need a new connector.

Start with the sources that produce the highest volume of alerts that follow a known pattern, such as endpoint detections, identity provider sign-in alerts, email security alerts and network intrusion alerts. If you run Wazuh, Suricata or Zeek, the playbooks often begin from those detections because they already carry the fields that enrichment needs. For threat intelligence lookups, MISP is a common enrichment source, but confirm that the connector you need exists or can be written as an app. Ask the partner which sources they propose, what each one would be used for, and what they would not connect in the first wave.

Ownership should be assigned per playbook before go-live, with a named internal owner for the business decision and a named partner or internal engineer for maintenance. Playbooks break when connected tools change their APIs, credentials expire, or alert formats shift, so ask how the partner monitors for those changes and how much of that work is in the initial fixed scope versus a retainer. DeployOpen offers ongoing support and upgrades as retainers, and its work is delivered inside the client's own infrastructure, so the owner and the change log should live with the client.

Each playbook should come with a one-page specification, a workflow export from Shuffle, a list of the apps and credentials it uses, the approval rules, the test cases and results, the success measure and its baseline, and the owner. Ask for version control of the workflow and app definitions, and for a record of which changes were approved and by whom. These documents are also the evidence that an assessor or incident reviewer will ask for later, so agree on their format and storage location before the first playbook is built.

Start from your alert volumes and the playbook mix, then count the actions that will actually execute. Shuffle prices on app runs, which are actions that execute in a workflow. Triggers and skipped actions do not count, and users, apps and workflows are unlimited. A capacity model lists each playbook, the alert types that start it, how many actions run on the common path and on the escalation path, and the expected daily volume for each. Ask the partner to show that model in writing and to confirm how the enterprise limits apply to your edition. Since v2.1.1, the limits between the open-source and enterprise editions are actively enforced, so confirm the figures for your version before you commit to a volume.

Plan more than one Orborus node so workflows keep running if one fails. Orborus polls the Shuffle backend for jobs and starts a worker for each workflow execution, so a single node is a single point of failure for execution even when the backend is redundant. The count depends on peak concurrent executions, the resources each worker container needs, and how much headroom you want during a burst of alerts. Ask the partner to show the node count, where each node runs, how a failed node's jobs are picked up by the others, and how they test that failover before go-live. If the proposal shows one Orborus node described as highly available, treat that as a red flag.

Shuffle's configuration guidance for high availability calls for an OpenSearch cluster, optional Memcached for a shared cache, and NFS for shared files when several backends run. Sizing depends on stored workflow executions, retention settings, the number of backend and frontend instances, and the disk and IOPS the shared file store can sustain. Ask the partner for a sizing sheet that states node counts, storage per node, retention periods, backup method and the expected behavior if NFS or one OpenSearch node becomes unavailable. Also ask who owns the NFS platform and how its performance is monitored, because a shared file store that slows down can stall several backends at once.

Shuffle documents four architectures: a single server with Docker Compose, distributed Docker Swarm, a hybrid with Shuffle Cloud and a local Orborus, and Kubernetes with Helm charts. A single server suits a small pilot with limited alert volume. Swarm or Kubernetes suit larger volumes, multiple backends and several worker nodes. The hybrid model can reduce inbound firewall rules because the local Orborus does not need inbound ports, but it places part of the execution path outside your network, which your data handling rules must allow. Ask the partner to justify the choice against your alert rate, data residency rules and the number of teams running playbooks.

A credible partner monitors queue depth, worker count, execution duration, failure rate and resource use on each Orborus node, OpenSearch and the backend. Ask which metrics they collect, where dashboards live, which thresholds trigger alerts, and who responds at night. Ask how they detect a playbook that loops or runs far more actions than the model predicted, since that directly consumes app runs. Also ask how monitoring data is retained, because it becomes evidence when you need to show that automated response ran as designed.

Test with replayed or synthetic alert volumes that match the peak your capacity model predicts, plus a margin. Run the test against a non-production environment with the same node counts, then record execution latency, backlog and app-run counts. Include a node-failure test that stops one Orborus node during load and confirms that queued jobs complete. Ask the partner to provide the test plan, the pass criteria, and the report they will hand over. Any synthetic alerts should avoid triggering destructive actions in connected tools.

The written agreement should state the expected app-run volume by playbook, the enterprise limits for your edition confirmed by the vendor in writing, the Orborus and backend node counts, the OpenSearch, Memcached and NFS sizing, and the thresholds that trigger a capacity review. It should also say who pays for added workers or storage, how growth in alert volume is handled, and how often the model is refreshed. Ask for a named owner on each side. Because limits and pricing can change between versions, confirm the figures for the version you deploy, not the version in a proposal written earlier.

Your organization typically pays for the compute, storage, network and any Shuffle license tier, and the partner charges for planning, configuration, load testing and ongoing tuning. Costs recur as volume grows, so the model should show which components scale with app runs and which scale with nodes or storage. Ask for a quote that separates one-time build work from recurring infrastructure and license costs, and ask what happens when a playbook is added that multiplies action counts. Do not accept a quote that cannot show how the cost moves with volume.

Effort depends mostly on how many alert sources, tools and playbooks are in scope, and on how many of those tools already have a usable Shuffle app. Shuffle builds apps from OpenAPI specifications in the App Creator or from Python code using the app SDK, so a tool with a clean API and a published specification is far quicker to connect than one that needs custom authentication or a hand-written app. Effort also rises with the number of approval gates, the change control your organization requires, and the number of environments (lab, staging, production) that each playbook must pass through. Ask any partner to split the estimate by alert source, playbook and integration so you can see where the work sits.

The timeline depends on the deployment architecture you choose, the readiness of your identity and network access, and how many playbooks reach production in the first phase. Shuffle documents four architectures, from a single server with Docker Compose to Kubernetes with Helm charts, and a hybrid option where a local Orborus needs no inbound firewall ports. Choosing a larger architecture adds platform work before the first playbook runs. Approval from your security, change and operations teams often sets the pace more than engineering does. Ask the partner to tie each milestone to a dependency you own, such as account access, a test tenant for each connected tool, or a named approver for destructive actions.

Your team needs to supply access and decisions, not only tools. Expect to provide administrative accounts or service identities for each connected system, test or sandbox instances where automated actions can be run safely, a list of alert sources with sample events, and the named owners who approve destructive actions such as isolating a host or disabling an account. You also need to decide where the Shuffle server and its workers will run, which identity provider will handle single sign-on, and where the encryption secret used for saved app credentials will be held. Missing any of these usually delays testing more than it delays building.

A capable internal team can build and run playbooks, but the work splits into skills you may or may not already have. Writing a playbook in Shuffle requires familiarity with its workflow editor, the apps for each tool, and the security logic of triage and response. Keeping connectors working is a separate, recurring job, because each connected tool changes its API and authentication over time. Shuffle's open-source edition includes its features, so the software cost is not the main question. The real question is whether your team can staff the build, the testing, and the upkeep of every app after go-live. Ask a partner to show which parts they would hand over and which they would keep.

App-run licensing is predictable when you can estimate how many actions will actually execute each month. Shuffle prices on app runs, meaning actions that execute in a workflow. Triggers and skipped actions are not counted, and users, apps and workflows are unlimited. Your monthly volume therefore depends on alert volume, how many enrichment lookups each alert triggers, and how many playbooks run automatically. Noisy detection rules that trigger many enrichment steps can raise the count quickly, so tuning early matters. Ask the partner to model run volume from your real alert data, and confirm with Shuffle which edition and limits apply to your deployment, since limits between the open-source and enterprise editions are actively enforced since v2.1.1.

After handover, the playbooks keep working only if someone maintains the apps that connect them to your tools. Each connected system can change its API, authentication method or data format, and a broken app causes a failed playbook. Shuffle apps are versioned Docker containers, so updates can be tested before they replace a working version. Upkeep also includes platform upgrades. Shuffle's configuration docs warn that losing or changing SHUFFLE_ENCRYPTION_MODIFIER breaks all saved app authentications, so key custody needs its own procedure. Ask any partner what support, upgrades and testing they offer under a retainer, what is covered by a fixed-scope engagement, and how they document each playbook so your team can take over.

A safe test process uses separate environments, test accounts, and recorded runs before any playbook can take action on production systems. Ask the partner to show how they run each playbook against sample alerts, how they confirm that destructive actions only fire after approval, and how they verify that saved credentials and encryption settings match between environments. Testing should also cover failure modes, such as what happens when a connected tool is unreachable or when a worker node stops. Request written test evidence for each playbook as part of the handover package.