Why Graylog processing pipelines and parsing decide whether logs are usable
Logs arrive from firewalls, servers, applications and cloud services, each with its own format, timestamp style and field names. Until those messages are parsed into consistent fields, the team cannot reliably search for a user across sources, alert on a failed login pattern or route payment events to a separate retention policy. In Graylog, pipeline rules and parsing are how raw messages become that structure. Inputs receive the data, but the fields your searches, dashboards and event definitions depend on are usually created or cleaned up in processing pipelines. Treat this layer as the core of the logging design, not a finishing step.
The difficulty is that pipelines behave differently from a set of independent extraction rules. Pipelines run in stages, and rules within a stage execute in an order that determines what later rules see. A rule that normalizes a hostname in stage one can change what a stage-two enrichment matches on. Before go-live, map which rules touch which fields, decide which rules must stop further processing once they match, and document the order. Reviewing this order is one of the first checks to run when a field appears empty in search results.
Grok patterns are the most common source of fragile parsing. A pattern written against today's syslog or application format will quietly stop matching when a vendor adds a field, changes a delimiter or alters a timestamp. The messages still arrive, so the failure is silent: the fields are missing and alerts stop firing. Lookup tables carry a different risk. They enrich messages with values such as asset owners, threat indicators or user departments, and their results are only as current as the data source behind them. Decide who owns each lookup table, how often it refreshes and what happens to messages when the lookup has no match or the source is stale.
Routing adds a consequence that teams often overlook. Streams send each message to an index set, and index sets carry their own rotation and retention settings in Graylog Open. A rule that sends a message to the wrong stream can move sensitive events into a short-retention index, or leave high-volume noise in an index kept for a long period. Routing decisions therefore belong in the retention plan, not only in the parsing design. Confirm with your compliance owner which data types need separate index sets before the rules are written, because changing them later means reprocessing or accepting a gap.
Graylog Open includes streams and processing pipelines, so the functionality needed for this work is available without a commercial edition. The effort is in design and testing rather than in learning the rule syntax. Build a sample set of messages for each source, including malformed and edge-case lines, and run new rules against them before they touch production traffic. Keep a rule inventory with owners and a change log, and check parsing output after every vendor or software update. Ask any implementation partner to show how they test, version and roll back rules, since that record determines whether logs stay usable over time.
Which Graylog components the parsing layer touches
Parsing in Graylog is not one feature. It is a chain of inputs, fields, pipelines, streams, lookup tables and index sets that each shape what the next component receives. Knowing which component reads and writes what is the first step in changing parsing without breaking searches, alerts or retention.
Inputs that receive raw traffic
Inputs are where syslog, GELF and Beats traffic enters Graylog, and each one sets the initial message fields before any rule runs. Check which input type and port each source uses, whether the source sends structured GELF or plain syslog text, and which input carries which fields. A change to an input's listener or protocol affects every rule downstream, so confirm sender configuration before editing extractors or pipelines.
Message fields read and written
Extractors and pipeline rules both read message fields such as the raw message and fields added by the input, then write new or changed fields back. Inventory which fields each parser creates, their names and types, because searches, dashboards and alert conditions reference those names directly. Renaming a field in one rule can silently empty a dashboard panel or event definition that depends on it.
Processing pipelines, stages and rules
Pipelines hold ordered stages, and each stage holds rules that run against messages matching their conditions. Stage numbers control order, and rules within a stage run in a defined sequence, so a later rule may see fields that an earlier one created or removed. Document the stage plan for each pipeline and test any change that reorders stages, since a single misplaced rule can change what every later rule matches.
Streams that route messages
Streams select which messages a pipeline sees, and they can also send messages to index sets and alert conditions. Stream rules match on fields, so the order in which parsing creates those fields decides whether a message is routed at all. Confirm which streams each pipeline connects to, and whether a message can match several streams, which would cause it to be processed more than once.
Lookup tables used for enrichment
Lookup tables let rules add fields such as owner, asset group or threat context by looking up a value from a message. A rule that calls a lookup table depends on its data source, cache settings and key format, so a stale or missing table quietly produces empty enrichment rather than an error. Decide who maintains each table, how changes are reviewed and how missing keys are handled in the rule logic.
Index sets that store the output
Index sets decide where parsed messages are stored, how they rotate and how long they are kept. A message routed to the wrong index set may be searchable in one place and invisible in another, and retention settings apply to the whole set. Map each parsed data type to a named index set before going live, and confirm that the fields written by the pipeline match the index mappings the team expects.
Message journal and storage beneath the pipeline
The message journal buffers accepted messages on disk before processing and indexing, and the index layer stores the results. If the journal fills or the search backend slows, parsing stops being the bottleneck and ingestion backs up. Size the journal from expected daily ingest and check its usage during rule testing, since heavy regex or lookup work can slow processing enough to push pressure down to the journal.
Decisions to make before writing the first rule
This table compares the choices that shape Graylog parsing before any pipeline rule is written: where parsing runs, how fields are extracted, how pipelines are divided, how messages are enriched and how they are routed. Read each row as one option, its operational cost and the conditions under which it fits. The main takeaway is that most ongoing effort goes into keeping parsing and reference data current, so choose the options your team can maintain.
| Decision | Option | Operational cost | When it fits |
|---|---|---|---|
| Where parsing runs | Parse at the shipper | Shipper configs and agent versions drift across hosts; each change is a rollout across the fleet | Few sources, stable formats, or hosts that must send already structured data |
| Where parsing runs | Parse in Graylog | Rules live in one place and change without touching hosts; parsing adds load on Graylog nodes at high volume | Many sources, frequent format changes, or shippers that are hard to update |
| Extraction method | Grok patterns | Patterns are hard to read and can fail silently when a format changes; each source needs its own tested pattern | Plain-text syslog or application lines with no structured output |
| Extraction method | JSON extraction | Reliable and cheap when the source emits JSON; nested or inconsistent fields still need mapping rules | Applications or agents that already write structured JSON events |
| Pipeline layout | One large pipeline | One place to read the full flow, but every message is checked against rules it does not need, and edits affect all sources | A small number of sources with similar handling and one owner |
| Pipeline layout | Several pipelines per source family | Clear ownership and testing per family, but more pipelines to connect, order and keep consistent | Different teams own sources, or source formats differ widely |
| Enrichment | Lookup tables | Tables must be refreshed as assets, owners or threat lists change; stale data adds wrong context without raising errors | Stable reference data, such as asset ownership or site codes, with a named refresh owner |
| Enrichment | Data written by the source | No lookup upkeep, but the source team must add the fields and keep their values accurate | The source system is the system of record for the value |
| Routing | Stream rules or pipeline rules that set the stream | Stream rules match before parsing; pipeline routing depends on stage order, so misordered stages send messages to the wrong stream | Stream rules for fields known at ingest; pipeline routing when parsed content decides the stream |
How to build pipeline rules from sample messages to production
This sequence turns raw log samples into tested Graylog pipeline rules that parse, enrich and route each log type. Each step ends with a check the team can run, so rules reach production only after the fields, defaults and index destinations are confirmed.
Inventory sources and collect sample messages
List every log source that will send data to Graylog, including the input each one uses (syslog, GELF or Beats), its format and a named owner. For each input, capture a representative set of raw messages that covers normal events, error cases and known variants, and store them in a replayable sample file. The step is complete when every source has an owner, an assigned input and at least one sample set that someone outside the build team has reviewed.
Map the fields each consumer needs
Ask each consumer of the logs, such as a security team, an operations dashboard or an event definition, which fields it relies on and in what form. Record the mapping as a table with the source field, target field, data type, whether the field is required and what should happen when it is missing. Agree the table before writing any rule, because later field renames affect searches, dashboards and alerts. The check is that every field referenced by an existing dashboard or alert appears in the table.
Write extractors and parsing for each format
Use extractors for simple patterns on an input, grok patterns for delimited or semi-structured text, and JSON parsing for structured payloads. Run each parser against the sample set in the pipeline rule simulator, then compare the output with the field table. Confirm in your Graylog version how a parse failure is recorded and whether the message keeps its raw text. The check is that a sample message produces the expected fields with the expected types, and that a malformed message is flagged while the remaining messages keep parsing.
Group rules into pipelines and stages
Place rules into pipelines connected to the relevant streams, and order them in stages with explicit priorities. A typical order runs parsing first, then normalization, enrichment and finally tagging or dropping. Give each rule one purpose and a clear name so reviewers can read the pipeline top to bottom. Confirm in your version how an error in one rule affects later stages. The check is that a message which fails one rule still passes through the remaining stages, and that the pipeline view matches the design.
Add lookup tables with an owner and refresh
Add lookup tables for enrichment, such as mapping host addresses to a site or asset owner. Each table needs a named owner, a data source, a refresh method (manual upload, scheduled import or API push) and a record of when it last changed. Define the default value returned on a miss so that unknown keys do not leave fields blank or break downstream searches. The check is that a lookup miss produces the defined default and that a test message shows the enriched field.
Route messages to the right index set
Use stream rules to send matching messages to a stream, and connect each stream to an index set whose rotation and retention suit that log type. Pipeline rules can also route messages to a stream after parsing, which is useful when the match depends on parsed fields. Decide which messages fall through to the default stream and who reviews them. The check is that test messages land in the expected stream and index set, and that unmatched messages appear in the default stream for review.
Promote rules through a test environment
Export the streams, pipelines, rules and lookup configuration from the development instance, and import them into a test Graylog that uses the same input types. Replay the sample set and a sanitized copy of recent traffic, then compare the fields, defaults and volumes against the mapping table. Obtain sign-off from each consumer, record the change with its rollback steps, and then import the approved content into production. The check is that the replay matches the mapping table and that the production change record names the rollback procedure.
How to test parsing rules before they reach production traffic
Test every parsing rule against a copy of real traffic before it runs on production streams. The most reliable method is to capture raw messages from each input, replay them into a non-production Graylog instance that has the same inputs, streams, index sets and pipeline connections, and compare the stored results with what you expected. Capture raw bodies from the source side where you can, because a message that has already been parsed cannot show you whether a rule will handle its original form correctly. Keep the captured set small enough to rerun in minutes and broad enough to include every log type the rule touches.
Use the pipeline simulator, where your Graylog release provides it, to step a sample message through each stage and see which rules fired and which fields changed. Then check the outputs that matter downstream: field names, field types and the exact values that searches, dashboards and event definitions depend on. A field that is a number in one message and a string in another will break numeric comparisons without raising an error, so assert types as well as presence. Confirm in the vendor documentation which simulator features your release includes, since the tooling available for testing has changed between versions.
Build a deliberate set of edge cases. Include messages with missing fields, unexpected or mixed character encodings, very long lines near any size limit your inputs enforce, and empty values. Test stage ordering explicitly: if a rule in a later stage depends on a field set earlier, send a message that triggers only the later rule and confirm that the dependency is handled. Finally, verify routing. For each sample, confirm that the message reaches the intended stream and therefore the intended index set, and that it does not also match a stream it should not belong to.
Plan for three failure cases before go-live. Messages that match no rule continue through the pipeline unchanged and are routed only by stream rules, so they can land in the wrong place without any visible error. Messages that fail parsing, for example a grok pattern that does not match, may be stored with a missing field or a failure marker rather than being rejected, depending on how the rule is written. A lookup miss returns no value unless the lookup table or rule defines a default. For each case, confirm with the vendor and in your own test environment exactly what is recorded, then create a monitoring check, such as an event definition that counts unmatched or failed messages per source, so the behavior is visible in production.
Keep a set of golden samples under version control, with each file paired with the expected parsed output in a known format. Store the rule exports beside them so a change to either one shows up in review. Run the full replay before every rule change and before each Graylog upgrade, and treat any difference from the expected output as a failed test. Because testing features and simulator behavior can change between releases, confirm with the vendor which testing capabilities each Graylog version supports before you rely on them in your process, and record that answer alongside the test suite.
How pipeline design changes across deployment models
On a single Graylog server, one node runs the inputs, the processing pipelines and the web interface, so every message follows the same rule set. The design is simpler to reason about, but the impact of a bad rule covers the whole ingest path. There is no peer node to absorb a failure or a restart.
- Fewer moving parts: one server means one place to check pipeline order, stage behavior and stream connections.
- Single processing path: a rule that drops, misroutes or errors on a field affects all incoming traffic at the same time.
- Test in a copy first: export the pipelines, load them into a separate test instance and replay representative sample messages before editing production.
- Document stage order: keep a written record of which pipelines connect to which streams so a rollback target is clear.
- Plan restarts: confirm with your operations team how inputs behave during a restart, since a single node has no peer to take over.
Before a production change, confirm that the test instance runs the same Graylog version as production and that you can revert the last rule change within your normal change process.
The Graylog multi-node guide for medium to large deployments recommends three Graylog and MongoDB nodes, with MongoDB running as a replica set, three Data Nodes on their own cluster and a load balancer in front of the Graylog nodes. Pipeline configuration is shared through the cluster, so all Graylog nodes work from the same rules. The guide also sets is_leader to false on follower Graylog nodes, which means the design must account for which node handles which duties.
- Leader setting: set is_leader to false on each follower node in its own configuration, and confirm that exactly one node is configured as the leader.
- Shared configuration: pipeline rules and their connections are stored centrally, so a change should appear on every node. Verify this in your version rather than assuming it.
- Consistency check: after each rule change, send the same sample message through the load balancer and confirm it produces identical fields and stream assignments on each node.
- Load balancer testing: route test traffic so that messages reach different nodes during verification, not only the node where the rule was edited.
- Failure test: during a planned test window, stop one node and confirm that ingestion continues and that pipeline output does not change.
Ask your implementation team for a written verification procedure that covers every node and records the results of each rule change.
Package installs on Ubuntu, Red Hat, SUSE and Debian, and Docker Compose deployments both keep server settings in files that teams can version and review. Pipeline rules are usually created in the web interface, so the design has to include exporting them and storing the exports with the deployment definition. Lookup tables often depend on source files, such as CSV data or exported reference lists, that must travel with the rules.
- Export and version rules: keep rule and pipeline exports in the same repository as the Compose file or configuration management code.
- Keep lookup sources with rules: store the CSV or data files that feed lookup tables next to the rules that reference them, so a change to one is reviewed with the other.
- Pin versions: record the Graylog, MongoDB and Data Node versions used in testing and use the same versions in production.
- One definition, two environments: use the same deployment definition for test and production with environment-specific variables, so differences in results come from configuration you can see.
- Review exports: require a code review of exported rule files before any import, the same way application code is reviewed.
Confirm how your chosen method imports rules and whether an import replaces existing pipelines or adds to them.
In an air-gapped environment, Graylog cannot pull lookup updates, sample messages or package updates from outside sources. Every input to pipeline design has to be carried in through approved media or an internal transfer path. Plan this transfer before the first rule is written, because it sets how quickly lookup data can change and how test messages reach the team.
- Inventory data inputs: list each lookup table, its source system, its update cadence and the person who approves each transfer.
- Select sample messages: choose representative logs, remove sensitive values according to your data handling policy and move them as a controlled set.
- Confirm the transfer path: ask your security team which media, checksums and approval steps apply to each transfer.
- Build an offline test copy: deploy a test instance inside the enclave so rules are verified before they are imported to production.
- Use an approved package source: obtain installation packages through your internal mirror and confirm that versions and checksums match the test environment.
Ask your team to confirm the update schedule for each lookup table, since a stale table can change enrichment results without any visible error.
Access control and data handling for parsing and enrichment
Pipeline rules and lookup tables sit upstream of every search, alert and retention decision in Graylog, so changes to them affect all teams at once. The cards below cover who can change that logic, what lookup data may contain, how sample messages are handled, and how classification of data types shapes parsing and routing.
Restrict pipeline edit rights to a named group
Pipeline rules, stages and their connections to streams can alter or drop log data for every team that depends on the stream. Limit edit permissions in Graylog to a small, named group with an owner who answers for changes. Confirm in your installed release which permissions cover pipelines, lookup tables and stream connections, since these may be granted separately. Read-only access for everyone else keeps investigators from accidentally changing shared parsing.
Review every rule change before it ships
Treat a pipeline rule change like a code change: a second person reads the diff, tests it against representative messages, and approves it before it is applied to production traffic. Keep a written record of who requested each change and why. A rule that drops messages or rewrites a field can break alerts and dashboards without any error being raised, so the review should check what leaves the pipeline as well as what enters it.
Keep secrets out of lookup sources
Lookup tables often come from asset inventories, identity directories or spreadsheets, and their contents are visible to anyone who can view the table and can be copied into enriched messages. Exclude passwords, API tokens, private keys and session identifiers from every lookup source. Where an external lookup needs credentials, confirm how the connection is stored and who can read it in your installed release, and limit the account to the minimum read access it needs.
Treat sample messages as sensitive
Raw logs routinely contain personal data such as usernames, email addresses, IP addresses and customer identifiers, and sometimes credentials pasted into error text. Sample messages used to build and test grok patterns or rules inherit that sensitivity. Pull samples from a restricted source, mask or truncate fields that are not needed for the pattern, and remove test copies when the work is finished. Document where samples are kept.
Enrichment can add personal data
A lookup that joins a user identity, employee record or customer account to an event brings personal data into messages that may not have held it before. That changes who should see the stream and how long the data should be kept. Map each enrichment source to a data owner and decide, before building it, whether the enriched fields belong in the same index set as the raw events or a more restricted one.
Let data classification drive routing and retention
Stream routing decides which index set stores a message, and each index set has its own rotation and retention settings. Classify the data types each parser produces, such as security events, personal data or operational logs, before writing rules. The classification then determines which stream and index set a message reaches, how long it is kept and who may search it. Parsing designed without that mapping often has to be rebuilt later.
Confirm the configuration audit trail
Before relying on change history for review or investigation, confirm in the installed Graylog release whether changes to pipelines, rules, lookup tables and stream connections are recorded, which user made each change, and whether the record can be exported or retained independently. Test the trail by making a harmless change in a non-production environment and checking what is logged. Where the release does not record enough detail, define a manual change log as a fallback.
What to monitor, back up and document after the rules go live
Once pipeline rules handle live traffic, the risk moves from building parsing to noticing when it quietly stops working. Start with three signals that are easy to build and worth checking on a fixed schedule. First, look for messages that match no rule at all. In Graylog, messages that pass through a stage without matching any rule are a sign that a source has changed its format, a new application is sending logs that nobody mapped, or a rule condition is too narrow. Second, track parsing failure rates per source. A grok pattern or JSON extraction that fails on a fraction of messages will not raise an error in the interface, so you need a search or dashboard that counts failed or unparsed events by source input. Third, check that lookup tables are still refreshing. A data adapter or cache that stops updating leaves enrichment fields stale, and the pipeline still reports success. Confirm the refresh interval and the last successful update for each table, and decide who is alerted when it stops. Sudden changes in field population deserve their own check. After an application or agent update, a field that was present on most messages may disappear, or a new field may appear with unexpected values. A simple per-source count of how often each important field is populated, compared over time, will show these shifts before analysts notice gaps in searches. Pair this with event definitions for the conditions that matter most, such as a critical source going silent. Keep the thresholds in a written runbook so the team knows what a normal baseline looks like for each source. Pipeline rules, stream definitions and lookup tables are configuration, and they should be backed up as such. Graylog keeps this configuration in MongoDB, so a backup of the database is not enough on its own to rebuild rules in a readable form. Export pipeline rules, pipeline connections, streams with their routing rules, and lookup table definitions on a schedule, using the API or the content export that your Graylog version provides. Confirm the exact export method and what it includes for your release, and test a restore into a non-production instance at least once. Lookup table data may be large or sourced from external files or APIs, so record where that data comes from and how it is recovered. The handover documentation should let an engineer who did not build the rules change them safely. Include a rule inventory listing each rule, its owner, its purpose and the sources it applies to. Keep a sample message for every source, with the fields it should produce. Record the stage order and why it matters, because a rule in an earlier stage can change the fields a later stage depends on. Add a routing map showing which streams send messages to which index sets, and the test cases used to verify each rule. Finally, plan for upgrades. Graylog releases can change how rules and functions behave, so retest the full test suite after every release you deploy, and treat the test results as part of the change record.
What drives the effort and cost of pipeline work
This table lists the drivers that change how much labor pipeline work takes and what it costs to run afterward. Read each row as a question to answer during discovery: how many formats, how messy, how much enrichment, and who signs off on fields. Much of the cost comes from decisions made before the first rule is written, and parsing choices also affect licensing because Graylog Enterprise and Graylog Security are priced on processed volume.
| Driver | Why it changes effort or cost | How to keep it down |
|---|---|---|
| Number of log formats | Each distinct format needs its own parsing rules, sample messages, and tests, so effort grows with the count. | Group sources by format, retire unused formats, and parse shared fields once where formats overlap. |
| Inconsistency within a source | A source that changes field names or layout between versions forces branching rules and more test cases. | Standardize output at the source, such as one log schema per application version, before building rules. |
| Enrichment sources and change rate | Each lookup table needs a loader, an owner, and refresh logic, and frequent data changes add recurring work. | Limit enrichment to fields that searches, dashboards, or alerts use, and agree a refresh schedule per table. |
| Required field schema | SOC analysts, dashboards, and alerts depend on field names and types, so late changes force rework across rules. | Agree field names, types, and mandatory fields with consumers in writing before rule writing begins. |
| Volume per pipeline | Heavy rules on high-volume streams raise processing load and slow tests that replay large sample sets. | Filter noise early, route high-volume data to dedicated streams, and keep expensive regex out of hot paths. |
| Environments needed for testing | Rules need a non-production Graylog and replayed sample traffic, so each added environment adds setup and sync work. | Keep one test environment with replayed sample logs, and promote rules through version control. |
| Compliance field requirements | Rules that keep, mask, or drop fields must match retention and privacy obligations, and reviews add time. | Document which fields must be kept or masked, and confirm each requirement with your compliance owner before parsing. |
| Licensing exposure | Graylog Enterprise and Graylog Security are licensed on processed data, so changes in volume affect subscription cost. | Drop or reduce noise during parsing where policy allows, and confirm current volume terms on the Graylog pricing page. |
Questions teams ask before writing Graylog pipelines
Can our own team build Graylog processing pipelines without a partner?
Yes, if the team can read sample logs, write patterns or JSON field mappings, and test changes safely. Processing pipelines are part of Graylog Open, so licensing does not decide who builds them. The deciding factors are how many log formats need parsing, how well the team knows streams and index sets, and who will maintain the rules after go-live. Teams with a few stable formats often build rules in-house by working from the documentation for their version. Teams with many sources, shared lookup data or compliance-driven field requirements usually spend more time on design and review than on writing rules. Before deciding, ask who will fix a rule when a vendor changes a log format, and whether that person has time for it.
How long does a Graylog pipeline project take, and what sets the duration?
Duration follows scope, so the answer depends on the number of sources and formats involved. The main drivers are how many log sources need parsing, how many distinct formats each source emits, how many enrichment lookups are required, how many stages and routing rules the design needs, and how many review cycles the business expects. Work on logs that already reach Graylog is usually shorter than work that starts with new inputs. Each format needs sample data, a parsing approach, tests and sign-off from the owner of that log. DeployOpen agrees timelines in writing as part of a fixed-scope project, so ask for a schedule tied to named sources and deliverables, and check which assumptions about access and sample data it depends on.
What must we provide before a pipeline consultant starts work?
Sample logs, field requirements and named owners are the inputs that most affect the schedule. Provide representative samples for each source, including error messages and unusual edge cases, so parsing is tested against real variation. Document which fields each consumer needs, along with field names, data types and timestamp formats. Name one owner per log source who approves parsing decisions. For lookup data such as asset lists, user directories or threat feeds, name the owner, the update frequency and who maintains the source. Also confirm read access to the Graylog environment, a non-production instance for testing, a change window and a rollback approach. Missing items usually show up later as rework.
Does Graylog Open include processing pipelines, or do we need Enterprise?
Graylog Open includes processing pipelines. The Graylog Open product page describes routing logs through streams and processing pipelines, and the Graylog pricing page lists Open as free with no volume limits and community support. Enterprise starts at 15,000 US dollars per year, according to the Graylog pricing page, and adds a data lake with archive restore, data tiering, a correlation engine and 24/7 support. Security starts at 18,000 US dollars per year and adds SIEM features such as risk scoring and prebuilt threat detection content. Both subscriptions are licensed on processed data with a minimum of 10 GB per day or 100 Graylog Compute Units. Choose an edition based on retention, support and detection needs, and confirm current terms on the pricing page or in a written quote.
How is grok parsing different from JSON parsing in practice?
Grok parses text by matching patterns, while JSON parsing reads fields the sender has already labeled. Grok suits syslog lines, web access logs and application messages with fixed but varied layouts. In practice, grok rules need sample lines for every message variant, and a small change in spacing or wording can stop a rule from matching. JSON rules need fewer patterns, but they depend on the sender producing valid JSON with stable key names and value types, so the risk shifts to the source team. Mixed estates often use JSON where the application controls output and grok for legacy text. Test both approaches with samples that include malformed lines, because unmatched messages still need a defined route.
How do we test pipeline rules without affecting live log traffic?
Test in a non-production Graylog instance fed with copies of production sample logs, and leave live rules unchanged until results are reviewed. Confirm in the documentation for your version whether the platform offers a way to run rules against sample messages before you rely on it. Other safe patterns include a separate test input that receives duplicated traffic, a dedicated test stream that only receives sample messages, and a staging index set with short retention. Check parsed fields, timestamps, stream routing, lookup hits and dropped messages. Also measure processing cost, because regex-heavy rules take more time at high volume. Record the expected output for each sample so later changes can be compared against it.
What does an outside partner add to a Graylog pipeline project?
An outside partner adds design and implementation capacity inside your environment, along with written deliverables your team can keep. Typical work includes a log source inventory, parsing design for each format, pipeline and stage layout, lookup table setup, routing and index set alignment, test cases and handover documentation. DeployOpen plans, implements, customizes, documents and supports self-hosted open-source platforms inside the client's own infrastructure. Most engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts, and ongoing support is available as a retainer. The partner does not replace the owners of log data or the people who decide retention. Ask for the deliverables list, the acceptance criteria and a plan for who maintains the rules afterward.
What changes in our pipelines after a Graylog upgrade?
Upgrades can change the platform your rules run on, so check pipelines against each release before moving production. The current Graylog documentation covers version 7.2, so confirm which minor version your cluster will move to. The compatibility matrix sets the MongoDB and OpenSearch or Data Node versions that each release supports, so include it in upgrade planning. Rules that depend on specific field names, lookup configuration or documented behavior should be retested with the same sample set used at build time. OpenSearch 1.x use has been deprecated since Graylog 7.0 and will be removed in Graylog 8.0, so any parsing work tied to that backend belongs in the plan. Keep rules in version control and store expected outputs with them.
How DeployOpen scopes, builds and hands over Graylog parsing work
DeployOpen plans, builds, documents and supports Graylog processing pipelines and parsing inside your own infrastructure, whether in your cloud accounts, data centers or air-gapped networks. Work starts from a written scope that names the log sources, pipeline stages and rules, lookup tables and streams covered. Each rule set is tested against representative messages before it runs on production traffic.
How we scope
- The written scope lists each log source, its format and the fields the pipelines must extract.
- Scoping maps which streams route each data type and which rules apply to each stream.
- Engagements are fixed-scope, so the scope, deliverables and timeline are agreed in writing before work starts.
How we build
- Grok patterns and JSON parsing rules are written against sample messages captured from each source.
- Pipeline rules are organized into stages, with lookup tables and stream routing kept as separate, documented components.
- Each rule set is tested against replayed sample messages before it is deployed to production traffic.
How we hand over
- Handover includes the pipeline rule source, lookup table definitions, stream and routing configuration, and test samples.
- Documentation explains which rule runs at each stage, what each extracted field means and how to add a source.
- Your team receives a runbook for testing changes and rolling back a pipeline rule set.
How we support
- A retainer can cover rule changes as new log sources or message formats appear.
- Retainer work can include reviewing rules when Graylog ships a new version that changes pipeline behavior.
- Ask which support hours and response terms apply before signing, since these are set in each agreement.

