Security Onion high availability and scaling

When the manager node is down, search nodes stop indexing events. The Security Onion architecture documentation gives the cause: search nodes cannot reach Redis while the manager is unavailable, so they cannot take in new data. The grid does not stop receiving traffic at that point, and this is where receiver nodes matter. Receiver nodes add pipeline redundancy and load balancing, so incoming events can keep moving through the pipeline while search indexing is paused. In a distributed design, the outage is a search and indexing problem first, and it becomes an ingest problem only if no receiver remains in service. Plan for that distinction before you draw your failure domains and decide how many receivers you need.

Analysts should expect a narrower console during the outage. Confirm in your release's documentation which searches still return results from data already indexed on the search nodes, and which console functions depend on the manager, such as detection management, grid-wide status views and any action that writes configuration. Events that arrived during the outage are unlikely to be searchable until indexing resumes, so hunts and investigations that depend on the most recent minutes should be paused or cross-checked against another source. Write these limits into the SOC runbook so shift leads do not assume a complete view of current activity while the manager is down.

Sensors are expected to keep capturing locally, and your architecture review should confirm this for each sensor model you deploy. Suricata and Zeek run on the sensor and continue to observe traffic on the monitoring interfaces even when the manager cannot be reached. That makes sensor disk sizing part of outage resilience, not only a capacity question. Ask how much local storage each sensor has for logs and packet capture, how long that storage lasts at your traffic rate, and what happens to sensor-side capture if the manager stays down longer than expected. The answers determine how much data you can recover later, so record them per sensor role.

Buffering and backfill are where release differences matter most. The architecture documentation describes receiver nodes as adding redundancy, but it does not settle how long receivers hold events, what happens when their queues fill, or whether events held during the outage are replayed into search once the manager returns. Before you rely on either behavior, confirm in the documentation for your exact version: buffer capacity and how it is measured, the order in which queued data drains, whether backfill is automatic or manual, and how duplicate or out-of-order events are handled. Get those answers in writing from the vendor or your implementation partner before the design review.

Recovery behavior depends on the release, so treat any statement about it as unverified until you have tested it. Build a test grid that matches production in version, distribution, node roles and receiver count, then stop the manager during a scripted window while representative traffic replays. Record which nodes stop indexing, what the receivers hold, how fast the queue grows, how the backlog drains after the manager returns, and whether any events are lost or duplicated. Repeat the test after each upgrade. The results should set your receiver storage, your outage runbook and the list of documented blind spots for analysts, and each test run should be kept as evidence for your audit file.

Retention, data volume, search node count and replica settings have to be decided together. If you size storage for search nodes first and set retention later, you can end up with a grid that holds the data you wanted to keep but cannot keep it searchable, or one that is searchable today and runs out of disk within a few months of growth. In a Security Onion distributed deployment, the search nodes hold the indexed data that analysts query in the Security Onion Console, so every retention decision is also a storage and node-count decision.

Start with three questions. First, how long must hunting data stay searchable? Hunting, incident response and any internal or external reporting obligations may call for different windows, and the answer should come from your own retention policy and legal review rather than from a default. Second, how much usable storage does each search node have after the operating system, indexes and headroom are accounted for? Confirm the storage figures in the hardware documentation for your version, and remember that minimum hardware is not the same as production sizing. Third, what happens to indexing if a node is lost? Ask whether the remaining nodes can absorb the shards that were on the lost node, and how long recovery takes.

Replica settings determine how much of the indexed data survives a node failure, but they also multiply the storage needed for every day of retention. A higher replica count improves resilience and read capacity at the cost of disk, so the replica decision and the retention window must be checked against the same storage budget. Keep in mind that the architecture documentation notes that search nodes stop indexing when the manager is down because they cannot reach Redis. Replicas do not protect indexing from a manager outage, so that risk needs its own plan, covered in the receiver node section.

Before you set production retention, confirm the replica, shard and retention settings in the Security Onion documentation for the version you will run, since defaults and configuration paths change between major releases. Then test a full search node loss on a test grid that matches your production node count, storage and data volume. Record how long the cluster takes to return to a healthy state, whether queries still return complete results during recovery, how much data is at risk if a second node fails during that window, and whether indexing backs up. Use those results to adjust retention and node count before production data reaches the limit.

Ask any implementation partner for a written sizing that shows the retention window, expected daily data volume, replica count, usable storage per search node and the number of nodes that can fail without loss of searchable data. Ask them to show the node-loss test results from a matching test grid. Treat a sizing that lacks these numbers, or that cannot explain the effect of a node loss on indexing, as incomplete.

The goal of this test is to prove two things: the grid keeps collecting and holding data when the manager is down, and it returns to normal indexing without manual data repair once the manager comes back. Run it on a test grid that matches production in node roles, versions, receiver configuration and network paths. Do not run it against production. The architecture documentation states that when the manager goes down, search nodes stop indexing because they cannot reach Redis, so the central question is what the receiver nodes do with events in that window and whether they hold them safely.

Start with a baseline before you break anything. Confirm that every manager, search, sensor and receiver node shows as healthy in the Security Onion Console and that each node has registered with the manager. Check that Elastic Fleet agents on the monitored hosts report as connected. Then prove end-to-end flow: generate known test traffic at a sensor, such as a set of uniquely tagged DNS queries or HTTP requests, and search for those markers in the Console. Record the time each marker was generated and the time it first appeared in search. This baseline is your reference for every later comparison, so do not skip it.

Next, stop the manager service on the test grid and hold it down for a defined period that you record in the test plan. During that window, keep generating the same kind of tagged traffic at the sensors. Observe which components keep running, which queues grow and where new events appear or stop. Confirm that receiver nodes continue accepting events and that search nodes stop indexing, as documented. Check the Console for alerts or status changes and note exactly what an analyst would see. The purpose is not to pass or fail on a number, but to learn which behaviors are expected and which are surprises that need a design change or a runbook entry.

Restore the manager and watch the backlog drain. Confirm that search nodes reconnect, that indexing resumes, and that the tagged events generated during the outage become searchable. Compare the set of markers sent with the set found. Any gap is a dropped or unrecovered event and must be traced to a specific component. Repeat the test with a search node stopped and then restored, and with a receiver node stopped and then restored. For a search node, confirm that the remaining search nodes absorb the load and that the stopped node rejoins cleanly. For a receiver node, confirm that traffic continues through the surviving receiver and that the sensors' configured destinations behave as designed.

Record evidence for each step in a single log: UTC timestamps for the stop and start of each service, the time each tagged marker was generated and first searchable, counts of markers sent versus found, any dropped or duplicated events, screenshots of node health, and the time from manager restart to indexing resumed. Ask your vendor or implementer to state in writing which measures they will report and how they will count dropped events. Avoid setting pass thresholds that the documentation does not give you. Instead, agree with your security and operations leaders what counts as acceptable for your monitoring requirements before the test starts, and keep the signed-off results with your architecture records.