Security Onion syslog from firewalls and network devices

Use a supported Elastic Agent integration wherever Security Onion ships one for the device family. Reserve a custom ingest pipeline for formats that no supported integration parses. Decide per device family and per log type, since one vendor often sends several message types, such as traffic, threat, system and authentication logs, and coverage can differ between them. Before you choose, check the vendor's log format documentation for each message type you plan to forward, then compare it with the integration's documented fields for the Security Onion version you will run. If the documented fields do not include what your detections need, record that gap before the build starts. Supported integrations change between releases, so confirm coverage against the release notes and documentation for your version.

The maintenance difference shows up after go-live. A supported integration receives updates through the Security Onion release cycle, and your team mostly validates those updates. A custom pipeline changes only when someone edits it, which looks stable but means it can fall behind the rest of the platform. When Security Onion or Elasticsearch changes how fields are mapped or named, a custom pipeline may keep writing fields that dashboards, detections and hunting queries no longer read. Field naming is a likely place for silent failures. Integration fields follow the documented schema, so analysts see consistent names across sources. A custom pipeline uses the names you write, so you must map them to the fields that Console searches and detections expect, and document that mapping for the next analyst.

Custom pipelines move parsing ownership to your team. Someone must write the grok or dissect patterns, test them against real samples, handle timestamp and time zone parsing, and update them when a firewall firmware upgrade changes the message layout. Plan for a sample corpus per log type, taken from the device itself, and a regression check that runs the corpus through the pipeline before each change is deployed. Record which team owns each pipeline, who approves changes and how a rollback works. Ask any implementation partner to name these owners in the written scope, because a pipeline without a named owner usually breaks unnoticed. Integrations still need owners for version pinning and upgrade testing, but the parsing logic is maintained by the integration's developers.

Before any device is onboarded, produce a decision table with one row per device family and log type. Useful columns include vendor and model, firmware version, log formats, integration status for your Security Onion version, documented fields checked, decision and owner. Use the table to list which fields your detections depend on and to confirm those fields in the first test data. Mixed outcomes are normal. Keep field names consistent across both paths so that a search for a source address returns the same field whether the event arrived through an integration or a pipeline. Confirm with the vendor that the syslog transport (UDP, TCP or TLS), message framing and time format match what your chosen path expects.

Adding firewall and appliance syslog to Security Onion increases the number of events written to Elasticsearch, and that growth lands on the nodes that store and search them. Before you enable a new source, plan for more disk on the search nodes that hold indexed data and on the manager, which runs the console and the coordination services. The growth is not spread evenly. A perimeter firewall that logs every allowed and denied session can produce more events than all of your other syslog sources combined, so one device is often the first thing to exhaust storage. Treat firewall sources as a separate sizing line rather than folding them into a general estimate.

Four drivers set the size of that growth. The first is the event rate per device, which depends on traffic volume, rule count and how many sessions each rule matches. The second is verbosity on the device itself, since logging every session, traffic logging on implicit denies or debug-level appliance output can multiply the rate without any change in business traffic. The third is the retention period your team sets for each data class, and the fourth is the index and field mapping choices made in the Elastic Agent integration or custom ingest pipeline. Fields that are parsed into structured values usually consume more space than the raw message, and unparsed fields that land in generic mappings can make searches slower and storage less predictable. Each driver is controlled by a different team, so name an owner for each one before go-live.

Measure rather than assume. Pick a representative period, such as a normal business week that includes your heaviest change window, and count events per second for each device from the firewall or from the receiving pipeline. Compare that rate with the same device's traffic logs to confirm that verbosity matches what your detection content and investigations need. Then multiply the sustained rate by average stored event size after parsing, which you can read from a test index loaded with the same sources. Check that result against the documented minimum hardware for your node roles, which set the floor for CPU, memory and storage, and remember that those minimums describe a baseline, not a sized production system. Your own retention requirements, set by regulation, contract or internal policy, determine how many days of that volume you must keep online.

Ask for these items in writing from whoever will operate the deployment: the measured event rate per device with the measurement window, the verbosity settings each device will run, the retention period for each index class, and the node counts and disk layout that result. Confirm with your Security Onion deployment plan whether search nodes are sized for the combined load, and whether the manager's storage can absorb its share of growth. If you expect to add sensors or search nodes later, record the trigger, such as a sustained rate or a disk threshold, so the expansion is planned instead of urgent.

Plan for the next firewall or appliance you add, not only the first one. Each new source changes the total rate, and a single verbosity change on a firewall can shift retention needs more than a new small device would. Schedule a review of event rate and disk use after the first full retention cycle, and repeat it whenever a major new source, rule change or retention policy update takes effect. Keep the measurements and the sizing decision together so that an auditor or a future engineer can see why the storage was sized the way it was.

Before a firewall or switch feed is treated as usable, run a fixed test plan for each source and record what Security Onion actually stored. The goal is to prove three things for every device: the events arrive from the expected sender, the timestamp and fields are populated the way your searches and detections assume, and the content that depends on those fields still fires. Assign one owner per source and keep the results in the same change record as the port and pipeline work, so a later reader can see what was tested, against which Security Onion version, and by whom.

Start by generating known events on the device itself. Use three cases that map to different parsing paths: one allowed connection, one denied connection, and one configuration change made by a named administrator account in a lab or maintenance window. Note the exact time each event is generated and the source address of the device. In the Security Onion Console, search for those events by source address and a narrow time window. Confirm the sender matches the device management or data address you expect, that the timestamp reflects the device's clock (or the timezone correction you configured), and that the fields a parser should extract, such as action, source and destination addresses, ports, protocol and user, are present with the values you generated. A raw message that appears with empty fields is a parsing failure, even if the message is stored.

Next, run a search that uses those extracted fields, not free text. For example, filter on the action field for denied traffic from the test source and confirm the single event you generated is returned. Then check the dashboards and Sigma or NIDS detections your team relies on for that device category. Confirm each one still matches the test events it should match, and that no unrelated noise appears in the results. Detection content and parsers are updated independently, so a detection that worked last month can silently stop matching after a change to field names. Record each result as pass or fail with the query text, so the test can be repeated exactly.

Plan the rollback before the first test run. If a feed is parsed incorrectly after a change to a custom ingest pipeline or device configuration, you need to know how to stop the source, restore the previous pipeline version and remove any bad events from the search path if your retention policy allows it. Store the pipeline definition and the device configuration in version control so the previous state can be redeployed. Confirm with your Security Onion administrators how the detections and dashboards are versioned, and whether a rollback of one source affects other feeds.

Repeat the full test after every Security Onion upgrade and after any change to the device firmware or log format, because parsing content can change between releases. Check the release notes for the target version, then rerun the same three events and the same searches against a test node if one exists. Ask your vendor or implementation partner to confirm which parsing content, pipelines and detection rules changed in that release. Keep the results from each run so that a regression can be traced to a specific upgrade or device change.