Graylog stream data routing: why teams design routes before they add sources
Stream routing in Graylog Open decides which messages land in which place, so that each log type gets the right search index, retention and access. Incoming messages from inputs such as syslog, GELF or Beats are matched against *stream rules*, which are field conditions that place a message into one or more *streams*. Each stream is then tied to an *index set*, which controls the indices the messages are written to and how those indices rotate and expire. *Pipelines* sit alongside this: they run processing rules on messages, and a pipeline can be attached to a stream so that only the matching traffic is parsed or enriched. When every source lands in the default stream and the default index set, every firewall heartbeat and debug line competes for the same storage and retention as your authentication and endpoint events. That is the point where a deliberate routing design pays off.
A team needs a routing design before onboarding a second or third major source when any of these hold: log types have different retention obligations, some sources are high volume but rarely searched, some need different access for analysts, or search performance is degrading because one index set holds everything. The design work is mostly decisions, not configuration. List the log types, assign each an owner, a retention period and an index set, and decide which messages should be dropped or sent elsewhere before they are stored. Write down the stream rule order, because a message matched by an earlier stream may or may not also reach a later one, depending on how each stream is configured. Confirm these behaviors against the Graylog version you run, since the routing documentation and the stream settings are the authoritative reference for your release.
Plan the change in a test environment with representative traffic before you touch production streams. Route a sample of each source, check the message counts per stream and index set, and confirm that searches and alerts still return what analysts expect. Pay attention to existing rules: streams created from an Illuminate content pack may already carry processing rules that run before any pipeline rules you add later, so review those first. The Graylog data routing documentation explains how the intake step displays the stream rules and pipeline rules that send messages into a stream, which is the first place to check when a message appears where you did not expect.
Routing options and when each fits
This table compares the Graylog mechanisms that decide where log data goes and how long it is kept, along with processing rules that Illuminate content packs bring. Read each row for what the option changes, where it fits and what it cannot do. The main takeaway is that routing and storage are separate decisions, so a message can be routed correctly and still fill expensive search storage.
| Option | What it changes | When it fits | What it cannot do |
|---|---|---|---|
| Stream rules at intake | Decide which messages enter a stream as they arrive, based on field conditions such as source or facility. | Separating log types by origin so each stream can get its own index set and alerts. | Cannot transform content or reduce volume on its own. Matching is by field conditions only. |
| Pipeline rules attached to streams | Run parsing, enrichment or filtering logic on messages routed to a stream. | Normalizing fields or filtering low-value events for one source without changing its routing. | Does not choose the index set directly. Storage follows the stream the message lands in. |
| Separate index sets | Store a stream's messages in their own indices with their own rotation and retention settings. | Log types with different retention needs, such as audit logs kept longer than debug output. | Does not reduce ingest volume. Messages still count toward storage and must be sized for. |
| Dropping noisy data | Discards matching messages in a pipeline before they are stored, so they are never searchable. | Health checks or repeated debug lines with no investigation value. | Dropped messages cannot be searched or restored later. Confirm retention requirements first. |
| Diverting to a separate index set | Sends matching messages to their own stream and index set, where shorter retention limits how long they stay searchable. | Bulky logs that are occasionally needed, such as verbose network flow records. | Still uses Graylog search storage in a standard setup. Confirm which archiving or tiering options your edition includes. |
| Processing rules from Illuminate packs | Streams created by an Illuminate content pack can already carry processing rules that run before pipeline rules you add later. | Adopting a content pack and then tuning routing around its existing rules. | Added rules may not run in the order you expect. Review the pack's existing rules before adding routing logic. |
How intake order and Illuminate content pack rules affect routing
Before you add a new route, check what already routes messages into the target stream. The intake view in Graylog lists the stream rules and pipeline rules from other streams that send messages into the stream you are about to change. Those inherited conditions decide what the stream actually receives, so a rule you add may match far more or far fewer messages than you expect. Record the existing rules, note which ones come from other streams, and only then decide whether a new stream rule, a pipeline connection or a change to an existing rule is the right tool.
Streams created by an Illuminate content pack can arrive with processing rules already in place. According to the Graylog data routing documentation, those rules may run before any pipeline rules you add later, which means a message can be rewritten, tagged or moved before your own logic sees it. Pipeline order and the stream's processing sequence therefore matter as much as the conditions you write. Confirm in your own environment which pack-created rules are active, what fields they set, and whether they change the stream a message ends up in, using a sample of recent messages rather than assumptions about how a pack should behave.
Document the pack behavior before you change routing, and keep that record with your stream design. When you upgrade Graylog or update a content pack, repeat the intake check, because a new pack version or a changed rule order can shift which messages reach each stream. The Graylog data routing documentation is the reference to check against your installed version. Where the documented behavior and your observed results differ, treat the observed results as the working truth and raise the difference with Graylog support before building further routes on top of it.
How to set up stream routing in Graylog step by step
This sequence takes you from an inventory of log sources to a tested routing design in which each log type lands in a stream with the right pipelines and index set, so noisy data stays out of costly search storage.
Inventory log sources and daily volumes
Before touching Graylog, list every source that will send logs, including hosts, network devices, applications and cloud services, and note the inputs each one will use, such as syslog, GELF or Beats. For each source, estimate daily ingest and identify the data class it belongs to, such as security authentication events, application debug output or network flow records. Ask the owners of each source to confirm volumes from their own systems, because estimates from memory tend to be wrong. The step is done when you have a table mapping every source to an input, a data class and a daily volume that the source owner has confirmed.
Define index sets and retention per data class
Create one index set for each data class that needs different retention or search behavior, rather than sending everything to a single default index set. Graylog manages storage through index rotation by message count, size or time and deletes the oldest indices under the retention setting, so each index set's rotation and retention must match the business and compliance requirement for that class. Confirm the retention requirement with your compliance owner for each class, and size storage with the Graylog guidance of daily ingest times retention days times 120 percent. The step is done when each data class has a named index set with documented rotation and retention values.
Create streams with stream rules
Create one stream per data class, assign each stream its index set, and add stream rules that match the fields your inputs produce, such as the source, the facility or a field set by an input or extractor. Decide whether a message may match several streams or only one, and enable the option to remove matches from the default stream only where you have confirmed that is intended, because a message that matches several streams can be stored more than once. Check that the default stream is not quietly receiving the traffic you intended for a specific stream. The step is done when test messages from each source land in the expected stream and the default stream holds only what you planned.
Attach processing pipelines to streams
Connect pipelines to the streams that need parsing, enrichment or filtering, so that pipeline rules run only on the traffic they were written for. Keep drop or discard logic in the stream or pipeline that handles the noisy data class, and document which pipeline runs on which stream and in what order. Before adding your own rules, check the intake view for processing rules that an Illuminate content pack may already have attached to a stream, because those rules can run before any pipeline you add later and change what your rules see. The step is done when each stream lists its pipelines in a documented order and no unexpected pipeline is attached.
Check the intake view for each stream
Open the intake view for each stream and confirm the rules that route messages into it, including stream rules and pipeline rules from other streams that send messages into it. This view shows the full path a message takes before it reaches the stream, so it is where you find routing loops, duplicated routing and processing rules that run earlier than expected. Fix any surprise by adjusting the stream rules or the pipeline order, then recheck. The step is done when the intake view for every stream matches the design from the earlier steps.
Test routing with sample messages
Send representative sample messages from each source, including edge cases such as malformed lines, very long messages and messages that match several rules, and confirm where each one is stored and how it was parsed. Use the message details and the stream counts to verify that the noisy class goes only to its low-cost index set and that security-relevant messages are never dropped by a filter. Keep the sample set and the expected result for each sample so the same test can be repeated after changes or upgrades. The step is done when every sample lands in the expected stream, index set and field layout, and the results are saved.
Review search and storage use
After traffic has run for a representative period, review index sizes, rotation activity and search performance for each index set, and compare them with the sizing estimates from the first steps. Look for data classes that grow faster than planned, streams that still hold data they should not, and searches that scan large index sets when a narrower stream would answer the question. Adjust rules, retention or index set assignments through a change record, and rerun the sample tests. The step is done when actual storage and search behavior match the design, and the findings and changes are recorded for the next review.
Security and operational risks in routing design
Routing decisions decide which messages get searched, how long they stay, and who can see them. The cards below pair each common failure in a Graylog routing design with the control that keeps it visible, so review happens before a change reaches production data.
Security events in short-retention indices
A stream rule that matches too broadly, or too narrowly, can send authentication and firewall events into an index set built for debug logs. Those events then age out on the shorter retention schedule before an investigation starts. Before go-live, trace a sample of known security events from intake to their index set and confirm the index set's retention meets the period your investigations and reviews require.
Pipeline changes that drop fields
A pipeline rule that renames, removes or overwrites a field can break searches, alerts and dashboards built on that field, and the failure often shows up as silent empty results rather than an error. Test each change against sample messages from every source on the stream and compare field sets before and after. Keep a list of fields that downstream alerts depend on, and check it on every pipeline change.
Stream and index set access by role
Routing decides which users can see which messages, so a permissive stream or index set exposes data to roles that never needed it. Review which roles can read each stream and index set, and separate administrative rights to edit routing from rights to search the data. Confirm the role model with a test account for each role before granting access to production streams.
Unreviewed routing changes
Stream rules and pipelines are often edited directly in the web interface, so no one else sees the change and no earlier version can be restored with confidence. Require a written request for each routing change, a second reviewer, and an exported copy of the pipeline and stream configuration before and after the change. Store those exports in version control so each change has a traceable history.
Illuminate rules running ahead of your own
Streams created by an Illuminate content pack may already carry processing rules that run before any pipeline rule you add later. A new pipeline can then see messages in a different shape than you expected, or a pack rule can route a message away from the stream you planned. Inventory the processing rules on every pack-created stream and document the order they run in before adding your own rules.
Storage growth from noisy sources
Routing a high-volume, low-value source into the same index set as security data uses up storage that the important events need. Set the volume each stream is expected to add per day, and check it against the storage sizing guidance before onboarding the source. Route noisy sources to a separate index set with its own retention, or drop them at intake if no one searches them.
Questions teams ask before routing logs in Graylog
How much does Graylog log routing work cost?
Routing work is priced from the written scope, so cost depends on how many log sources, streams and pipelines you need. DeployOpen does not publish a price list, and a quote follows from the agreed scope. Graylog Open is free with no volume limits, but the Graylog pricing page lists Enterprise from 15,000 US dollars per year and Security from 18,000 US dollars per year. Both carry a minimum of 10 GB per day or 100 Graylog Compute Units and are licensed on processed data, with no per-user or per-device fees. Ask the quote to separate one-time design and build work from any recurring support, and to state which license tier it assumes.
What drives the effort in a Graylog routing design?
The main drivers are the number of log types, the number of destinations each type needs, and how much parsing or processing already exists. Each log type needs a stream rule at intake, a decision on which pipelines attach to that stream, and a test with real sample messages. Effort rises when sources send mixed formats, when one message should land in several streams, or when noisy data must be filtered before it reaches expensive search storage. Existing Illuminate content packs add work, because their streams can carry processing rules that run before any pipeline you add later, so the order must be checked. Ask for an inventory of sources and their message volumes before agreeing scope.
How long does it take to validate a routing design?
No validated duration is available for this topic, so the timeline is set in the written scope after the source inventory is known. Validation time depends on how many log types need sample testing and whether a test environment matches production inputs. A useful validation checks each stream's intake behavior, the pipeline results on representative messages, and the storage effect of filtered and unfiltered data. Plan extra time if sources send inconsistent formats or if sizing must be confirmed first. The Graylog system architecture guidance advises sizing storage as daily ingest times retention days times 120 percent, with the message journal at 3 to 5 times daily ingest on SSDs. Ask the proposal to list validation steps and exit criteria.
What must our team provide before routing work starts?
Your team supplies the context that routing decisions depend on. Expect to provide a list of log sources with named owners, sample messages for each type including field names and sensitive values, expected daily volume, and the retention period for each category. You also need a running Graylog test environment, administrative access to create streams, pipelines and index sets, and a named approver for any routing change that touches production. Security and compliance staff should state which logs must be kept in full and which may be filtered. Without these inputs, the design rests on assumptions that surface at cutover. Confirm access and approvals at kickoff, because they often decide whether validation starts on schedule.
Can our own team design Graylog stream routing in house?
Yes, a team with Graylog administration experience can design routing in house, since the streams, rules and pipelines it uses are documented. The questions to answer honestly are whether the team has time to test with real messages, whether it understands the order in which processing rules run across streams, and whether it can maintain the design after go-live. Routing mistakes usually show up as missing messages in a stream or as costly storage filled with low-value data, so the test cycle matters as much as the first build. A partner helps when you need an outside review, a fixed scope, or a repeatable pattern across many sources. Decide by comparing your source list with the skills and hours available.
How does Graylog Open routing differ from Enterprise?
Graylog Open includes the stream and pipeline routing this topic covers, so a basic routing design does not depend on a paid license. The differences that affect routing sit in the higher tiers. The Graylog pricing page says Enterprise adds a data lake with archive restore and data tiering, while Security adds SIEM features such as investigations and prebuilt threat detection content. Data tiering matters most for noisy data, because it can hold lower-value logs at a cost that suits them. Some Illuminate content packs also depend on licensing, so check which packs your routing relies on. Graylog Open is licensed under SSPL-1.0, which is not an OSI-approved open source license, so confirm the license terms fit your organization. Ask the partner to map each planned destination to the edition that supports it.
How do Illuminate content pack streams affect routing order?
A content pack can create streams that already contain processing rules, and those rules run before any pipeline rules you add later. This affects where messages end up and how many copies reach storage, so review the intake sequence before changing routing. Open each stream's rules and attached pipelines, and identify which rules the pack created. Then confirm your new pipeline does not duplicate or undo the pack's work. Record the order in your design so that a future pack update or upgrade does not change routing without review. The Graylog data routing documentation describes the intake view, which lists the stream rules and pipeline rules from other streams that route messages into a stream.
How DeployOpen scopes a routing design
A routing engagement starts with a source inventory and a written routing map, not with stream rules. Before anyone creates a stream, DeployOpen and your team list every log source that will reach Graylog, the input type each one uses (syslog, GELF, Beats or another input you run), the fields and volume each produces, and who owns it. The routing map then records, for each source, which stream it lands in, which pipelines run on that stream, which search index set stores it, and what happens to data that matches no rule. Your team reviews and signs off the map in writing before configuration begins, so the scope is fixed on paper and changes after that point go through a defined change process.
Delivery covers the configured streams, stream rules, pipeline rules, index set assignments and any Illuminate content pack streams your edition includes, along with a test plan. The test plan sends sample messages from each source through the intended path and checks where they land, whether pipeline rules alter the fields you expect, and whether the destination index set receives only the data it should. It also checks the rules that Illuminate content packs bring in, because streams created by those packs can carry processing rules that run before any pipeline you add later. Ask for the test results and the final routing map as deliverables, and ask which items the test plan does not cover, such as high-volume load behavior, which needs its own sizing exercise.
Routing rules change as new sources appear and noisy sources are tuned, so ongoing work is often handled under a retainer. Under a retainer, DeployOpen can review proposed stream or pipeline changes against the routing map, update the map, retest affected paths and keep a record of each change. Confirm with DeployOpen how change requests are logged, how urgent changes are handled, and which systems the work covers, since retainer terms are agreed in writing like the rest of the engagement. Nothing in this section sets a price, duration or outcome for your environment, and the written scope is the document that does.
How DeployOpen delivers Graylog routing work
DeployOpen plans, implements, documents and supports self-hosted open-source platforms inside your own infrastructure, including Graylog stream data routing. We start from a written scope that maps each log type to its stream, pipeline and destination before any rule is changed on a running cluster.
How we scope
- We inventory log sources, their volumes and the searches and alerts that depend on each stream today
- The written scope lists the streams, stream rules, pipelines and destinations to be built or changed
- Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts
- We confirm which routing features your Graylog edition supports, including any Illuminate content pack rules already in place
How we build
- We design stream rules at intake and sequence pipeline rules so earlier processing does not conflict with later changes
- We review existing content pack processing rules on imported streams before adding pipelines on top of them
- We set up test inputs and sample messages to confirm each log type reaches the intended stream and index set
- We check that noisy data is routed to lower-cost destinations without losing messages needed for investigations
How we hand over
- We document the routing map, stream rule logic, pipeline order and destination for each log type
- We record the rollback steps for each routing change so your team can reverse a change on its own
- We walk your operators through how to trace a message from input to stream to index during an investigation
- Handover covers the configuration exported from your environment, which stays under your control
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer after handover
- We can review routing changes when you add new log sources or move to a newer Graylog release
- Retainer scope, response expectations and service terms are agreed in writing and confirmed before work starts
- Ask your provider which support terms apply to your Graylog edition and deployment model
