When Graylog lookup tables for log enrichment earn their place
Raw logs rarely say who owns the host, which team a user belongs to, or whether an IP address appears on a threat list. Analysts then pivot through several systems to answer questions the log should have answered. Lookup tables attach that context to each message at processing time, so the enrichment is stored with the event and available to searches, dashboards and alerts that follow.
Graylog's lookup tables are built from four parts, as described in the Graylog lookup tables documentation. A *data adapter* performs the lookup against a source such as a CSV file, a MongoDB collection or a geo-IP database, and the docs note that adapters are pluggable. A *cache* stores results so repeated keys do not hit the source again. The *lookup table* connects the two, and each lookup returns either a single value or multiple values. Lookup tables are then called from converters, decorators and pipeline rules, so the choice of where to apply enrichment shapes both performance and what gets stored.
Graylog Open is licensed under SSPL-1.0, which is not an OSI-approved open source license, and lookup tables are documented as part of the platform. Confirm with Graylog which features your planned design uses, particularly if you rely on Enterprise or Security content, and confirm the current edition requirements for each adapter and pipeline function before you commit to a design. Plan for the data behind each table as well: decide who owns the source file or collection, how often it changes, and what happens to enrichment when a source is stale or unreachable.
Which data adapter and cache setup fits each enrichment need
This table compares the CSV File, MongoDB and geo-ip data adapters, plus custom adapters, across source of truth, update method and typical use. Read the cache row alongside each adapter, because any adapter that calls an external service needs a cache sized to protect that service.
| Option | Source of truth | Update method and typical use | Cache behavior and what to confirm |
|---|---|---|---|
| CSV File adapter | A CSV file on the Graylog server or nodes, managed by your team | Replace or edit the file and reload the adapter. Suits small, slowly changing reference lists such as asset owners or known service accounts. | Confirm each node holds the same file version. Caching is configured on the lookup table, so set a TTL that matches how often the file changes. |
| MongoDB adapter | Key-value records stored in MongoDB, the same database that holds Graylog configuration | Records are changed through the Graylog interface or API. Suits editable mappings that analysts maintain, such as site codes or ticket-to-owner links. | Confirm your MongoDB backup covers these collections. Lookups hit the database, so a cache reduces load during heavy searches. |
| geo-ip adapter | A GeoIP database file supplied and refreshed by your team | Load a new database file when the vendor or your licence provides an update. Suits adding country or network context to source IP fields. | Confirm which database edition and licence terms apply to your use. Lookups are local, so caching mainly saves processing time on high-volume streams. |
| Custom or pluggable adapter | Whatever the adapter queries, such as an internal CMDB or a threat intelligence API | Built and maintained by your team or a partner. Suits sources that have no built-in adapter. | Required when the source is a rate-limited or paid API. Set cache size and expiry to limit outbound calls, and confirm how failed lookups are handled. |
| Cache layer (any adapter) | Stores recent lookup results in front of the adapter | Configured per lookup table. Needed when many messages share the same keys or when the adapter calls a remote service. | Confirm the expiry time against how quickly the source changes. Long expiry hides updates. Short expiry sends more calls to the source. |
| Single or multi value results | Determined by the adapter response, not by the cache | Single values suit one answer per key. Multi value results suit several answers per key, such as a list of tags. | Confirm the pipeline rule or decorator handles empty results, since a missing key can return no value. |
| Where lookups are used | Converters, decorators, pipeline rules and dashboard or search parameters | Choose the use based on whether the value must exist at ingest or only at search time. | Confirm which use each lookup table serves before changing its cache, because the same table can feed several places. |
Where lookups run: converters, decorators, and pipeline rules
Pipeline rules are the main path for enrichment. A rule calls a lookup table with a value from the message, such as a source IP, and writes the result into a new field before the message is indexed. Because the result is stored, every later search, alert and dashboard sees the same enriched value without repeating the lookup. Pipelines attach to streams, so you control which traffic is enriched and which is left alone.
- Single value results: A lookup returns one value, such as a country code or an asset owner, which the rule assigns to a field.
- Multi value results: A lookup can return several values from one key. Decide in the rule which values to store, and name the fields so searches stay predictable.
- Missing keys: Write an explicit branch for a lookup that finds nothing, such as a default value or a marker like unknown. Otherwise the field is absent, and searches for that field silently miss those messages.
- Stream scope: Attach enrichment rules only to the streams that need them to avoid lookup calls on high-volume data that does not benefit.
- Ordering: Run enrichment stages after parsing has created the source field the lookup depends on.
Confirm in your Graylog version how a rule should handle a lookup error versus a lookup miss, and test both cases on sample messages before you roll the rule out.
Converters change the value of a field during extraction, such as when an extractor reads a field from raw text. A lookup-based step can sit in that path, so the converted value is what gets stored. Converters suit narrow cases where the result is needed as part of field extraction rather than as a separate enrichment field. They run on messages at ingest, so changes apply only to new data.
- Field replacement: The converted value replaces the extracted value, so the original raw content may be lost unless you also keep it in a separate field.
- Ingest-only effect: Changing a lookup table does not rewrite messages already indexed, so history keeps the old result.
- Limited branching: Converter logic is simpler than a pipeline rule, so complex missing-key handling usually belongs in a rule.
- Traceability: Store the original value in its own field if an analyst may need to compare it with the converted result.
- Testing: Verify converter output with the message simulator or sample messages before it reaches production streams.
Check which converter types your Graylog version exposes for lookups and whether the version you run documents the same behavior.
Decorators change how a stored field is displayed in search results without changing the stored message. They are useful when an analyst needs a readable label next to a raw value, such as an owner name beside an asset ID. Because nothing is written back to the index, decorator changes take effect immediately for all viewers and do not require reprocessing old data.
- Display only: The underlying message keeps its original value, so searches and alerts still match the stored field, not the decorated label.
- Immediate effect: Updating the lookup table changes what viewers see right away, including for historical messages.
- Search mismatch risk: Analysts who search for the displayed label will not find messages, so document which value to search on.
- Cost control: Decorators run at display time, so keep lookup tables small and cached to avoid slow result pages.
- Access: Confirm which roles can view the decorated output, since displayed data can reveal enrichment content you may want to restrict.
Ask your Graylog contact or check the version documentation which decorator types are available in the edition you license, since some decorator functions depend on the features in your deployment.
Most enrichment needs point to one usage. Choose the usage based on whether the enriched value must be stored, searched and alerted on, or only read by an analyst. Mixing usages for the same field creates confusion, so pick one primary path per field and document it in your data dictionary.
- Searchable and alertable data: Use a pipeline rule, because the stored field can drive event definitions and dashboards.
- Analyst convenience only: Use a decorator when the label is for reading and the raw value stays the search key.
- Extraction-time transformation: Use a converter only for a narrow change that must happen while the field is extracted.
- Historical accuracy: Pipeline results are frozen at ingest, so if an auditor needs the value as it was then, rules preserve that.
- Single source of truth: Avoid enriching the same field through both a rule and a decorator.
Before finalizing, test each choice with a representative set of messages and confirm the chosen approach against your alerting and retention requirements.
Lookup behavior should be verified before production. Build a test set that covers a matched key, an unmatched key, a key with several values, and a message where the source field is absent. Run each message through the intended usage point and record the stored fields and displayed output. This makes missing-key defects visible before they affect detections.
- Matched and unmatched keys: Confirm the rule writes the expected value and the fallback for each case.
- Multi value handling: Check that every value you need is stored and that field names stay consistent.
- Absent source fields: Confirm the rule does not fail the pipeline or write misleading values.
- Change propagation: Update a lookup entry and confirm whether new messages change while old ones remain the same.
- Performance: Watch pipeline processing on representative volume, since lookup calls add work per message.
Record the test results and the Graylog version tested, then repeat the checks after upgrades because pipeline and lookup behavior can change between releases.
How to set up a lookup table in Graylog step by step
This sequence takes a reference dataset from a named source to a lookup table that pipeline rules can query, with a check at each stage so enrichment is proven before it touches production traffic.
Confirm the data source and owner
Before opening the Graylog interface, identify the dataset you want to enrich with, such as an asset inventory, a user directory export or a list of known-bad addresses. Record who owns the file or API, how often it changes, which field in your logs will serve as the lookup key, and which fields the lookup should return. The data owner and the platform team should agree on this in writing. Done well, you finish with a one-page note naming the key field, the returned fields and the update owner. Check that a sample export contains the key field in the same format your log messages use, for example the same hostname casing or IP notation.
Create the data adapter
In the Graylog web interface, open the System menu, choose Lookup Tables, and create a data adapter. For a static reference set, select the CSV File adapter and point it at the file on the Graylog server nodes, confirming the path, the separator, the key column and the value columns. For a database-backed source, the MongoDB adapter is the documented alternative. Graylog also documents a geo-ip adapter for IP enrichment. Save the adapter and use the adapter's test or status view to confirm it reads the file without errors. The check is that the adapter reports a successful load and shows the expected number of rows or columns. Each Graylog server node must be able to read the file, so confirm this in a multi-node deployment.
Create the cache
Create a cache that sits between the lookup table and the data adapter. For a local file that changes rarely, a generous cache lifetime keeps lookups fast and avoids repeated reads. For an external API, the cache is what protects the upstream service from one request per log message, so size and expiry should reflect the vendor's rate limits and how quickly the data changes. Record the cache type and its settings alongside the adapter so a reviewer can see both. The check is that the cache appears in the list with the correct type and that its settings match the decision recorded in the first step. Confirm the API vendor's terms before you set an aggressive lifetime or a high request rate.
Connect the adapter and cache in a lookup table
Create the lookup table and select the data adapter and the cache you created. Name the table so pipeline authors can recognize its purpose, for example a name that reflects the dataset and its owner. Save the table. Graylog uses the lookup table to connect the two components, and the table is the object that pipeline rules and converters reference later. The check is that the table shows as active or available, and that a test lookup against a value you know is in the source returns the expected result. If the table is not active, return to the adapter and cache to confirm both saved without errors.
Test a lookup with a known key
Use the lookup table tester in the Graylog interface to query a key that you know exists in the source file, and then query a key that you know does not exist. Compare the returned values with the source row for the first key, and confirm that the second key returns no value rather than an error. Test one multi-value result as well if your table returns several fields. The check is that both results match what the source file says. Keep the test keys and results in your change record, since they become the baseline for the verification in the final step.
Reference the table in a pipeline rule
Open the processing pipeline that handles the relevant log source and add a rule that calls the lookup table with the message field you chose as the key. Store the returned values in new message fields with a clear prefix, so the original fields stay untouched and analysts can tell enriched data from raw data. Include a condition so the rule only runs on messages from the intended source, and handle the case where the lookup returns nothing. Save the rule and attach it to the pipeline stage it belongs in. The check is that the pipeline shows the rule as connected and that the simulator, where available, shows the new fields for a sample message.
Verify enriched fields on sample messages
Send or select a set of real sample messages from the source, including at least one whose key matches the table and one whose key does not. Open them in search and confirm the enriched fields appear with the expected values, and that messages with no match carry no misleading values. Check the Graylog server logs for lookup errors during the test window. Record the search query and the results. The check is that matched messages show the correct enrichment, unmatched messages show none, and no errors appear. After that, schedule a periodic review of the source file or API so the enrichment stays current, and assign the owner you recorded in the first step.
Keeping enrichment data current without breaking searches
Stale enrichment fails silently. A CSV file that nobody re-exported, a MongoDB collection that a finance or asset team stopped updating, or an external API that changed its response shape will keep producing log messages with fields that look valid but point to the wrong owner, location or reputation. Graylog will not raise an error for this, so the first sign is often a search result or dashboard panel that no one trusts. Before building any lookup table, assign a named owner to each source: the team that maintains the CSV, the system of record behind the MongoDB collection, and the team that holds the API key and watches for vendor changes. Record who is accountable for correctness, not only for access.
The refresh cadence is a team decision. Graylog does not impose a refresh schedule on lookup data, so the interval should follow how quickly the source changes and how much a wrong value costs your investigations. For CSV and MongoDB sources, confirm how the export or sync job is scheduled and what happens when it fails. For API-backed adapters, the cache expiry setting in the cache configuration is the trade-off point: a short expiry gives fresher values but sends more requests to the external service, and a long expiry protects the API but keeps old answers in circulation. Confirm the provider's rate limits and terms before you set the value, and test how the adapter behaves when the upstream service is slow or unavailable.
Changed values alter history, so test before you rely on it. Enrichment is applied when a message is processed, so a changed lookup value does not rewrite the fields already stored on older messages. Searches and dashboards that filter on an enriched field will therefore return different results for old and new time ranges, which can distort trend charts and saved searches that prompt for lookup values. Before you roll out a changed table, run representative searches for a known period, compare results before and after the change, and document the cutover date. Keep the previous CSV version or export so you can explain a difference later, and review any parameter or dashboard that depends on the table.
Security and operational risks of enrichment data
Lookup tables sit inside the log pipeline, so anything that changes or leaks enrichment data changes what analysts see and what other teams can read. The cards below cover who can change lookup data, how external API keys and cached values are handled, how rule changes spread across streams, and how backups and failure monitoring keep enrichment reliable.
Write access to CSV and MongoDB sources
A CSV file or MongoDB collection behind a data adapter is a trusted input: whoever can edit it can change how every matching message is enriched, labeled or prioritized. Confirm which accounts and service identities can write to the source files, the MongoDB database and the Graylog admin screens that edit tables. Separate the people who curate indicators from the people who administer the platform, and record who changed what. Treat the MongoDB credentials for the adapter as privileged and rotate them on the same schedule as other backend secrets.
API keys stored for external adapters
Adapters that call external services need credentials, and those keys are stored in Graylog configuration. Anyone with access to the configuration, the database or a backup can read them, so the key may reach people who never needed the external service. Use a key with the narrowest quota and permissions the vendor allows, confirm where Graylog stores it, and plan rotation with a named owner. Check whether your edition and backup process protect these values at rest before you rely on them.
Sensitive lookup contents in logs and caches
A lookup result can contain personal data, internal hostnames or customer identifiers, and cached results keep those values in memory for the cache lifetime. Errors and debug output from an adapter may also write the looked-up value into server logs, where it spreads to other readers. Decide which fields may be enriched at all, review lookup output in the server logs during testing, and confirm the cache size and expiry settings with your data classification rules.
Pipeline rule changes affecting all streams
A pipeline rule that calls a lookup can be attached to several streams, so one edit changes enrichment for every message that passes through those streams. A rule that drops, renames or overwrites a field can break searches, dashboards and alerts that depend on it. Test changes against sample messages from each attached stream, stage them in a non-production Graylog, and keep rule versions in source control so a bad change can be reverted quickly.
Backup and restore of lookup data
Lookup tables depend on three things that must restore together: the CSV files or MongoDB collections, the data adapter and cache configuration, and the pipeline rules that reference them. Restoring the Graylog configuration without the matching source data leaves lookups returning empty results, which can look like a quiet period rather than a failure. Include all three in backup scope, test a restore into a separate environment, and document the order in which components must come back.
Monitoring for failed lookups
A failed or unreachable adapter usually does not stop log ingestion, so messages keep arriving without the fields that alerts and dashboards expect. Without monitoring, the first sign is often a missed detection or an empty dashboard panel. Define checks for adapter errors, unexpected drops in enriched message counts, and stale source files or external API timeouts. Confirm with your Graylog edition which health signals and notifications are available, and route them to the team that owns enrichment.
Questions teams ask before building lookup enrichment
What drives the effort of a Graylog lookup project?
The effort is driven by how many lookup sources you need, how often their data changes, and how much cleanup the source data needs before Graylog can use it. Each source needs a data adapter, a cache, and a lookup table that ties them together, and each one has to be wired into the pipeline rules or converters that use it. A static CSV file with a stable schema is a small job. A feed from an external API with rate limits, authentication, and changing field names is a larger one. Ask for the work to be counted per source, with freshness rules and error handling listed separately, so the cost of each decision is visible.
Can a team build CSV-based Graylog enrichment on its own?
Yes, a team with Graylog administration skills can build CSV-based lookups in-house, because Graylog documents CSV File as a built-in data adapter. The harder parts are usually outside the Graylog console: deciding which columns to keep, agreeing who owns the file, and making sure the file is updated on schedule without breaking running pipelines. Teams that own the data source, the pipeline logic and the review process can often manage this internally. A partner becomes more useful when the team lacks time, when several sources must be combined, or when the enrichment must be tested against realistic message volumes before it goes into production.
What does the work need from our side?
The project needs named owners for each data source, read access to the systems that produce or hold the enrichment data, and a set of representative test messages. Data owners confirm field meanings, update frequency and which values are authoritative. Source access covers files, databases or APIs, including credentials that can be stored safely in your environment. Test messages should include normal events, edge cases and malformed entries, taken from your own log sources with sensitive values removed where policy requires it. Without these inputs, lookups tend to be built on assumptions that fail the first time real traffic arrives.
How long does a first lookup enrichment version take?
The duration depends mainly on the number of sources and on their freshness rules. One static CSV file with a clear owner is usually the shortest path, while several sources with different refresh schedules, an API with caching, and a requirement to keep the last good data during outages take longer. No duration is set in advance here, so ask for a timeline in the written scope, with milestones for data review, adapter setup, cache tuning, pipeline testing and handover. Ask the provider to state which of those steps depend on your team's availability, since waiting on data owners is a common cause of delay.
When should we bring in a partner for lookup enrichment?
Bring in a partner when enrichment affects detection or investigation work and the internal team cannot test it thoroughly. Common triggers include external threat intelligence feeds that need caching to protect API quotas, enrichment that feeds alerts or dashboards used in incident response, and multiple sources that must be kept consistent. A partner also helps when lookups must run in a restricted network or meet documented change control. If the work is a small, owned CSV file that changes rarely, internal staff are usually enough. In either case, the decision should rest on who will maintain the lookups after launch.
What should a fixed-scope quote for lookup work list?
A fixed-scope quote should list each lookup source with its data adapter type, expected update method, and owner. It should state which caches are configured, what time-to-live values are proposed, and how failures from external services are handled. The quote should name the pipeline rules and converters that will use the lookups, the test messages used for verification, and the documentation and handover items delivered. Ask which items are one-time, such as adapter setup and rule testing, and which recur, such as data refresh reviews or feed changes. Confirm what your team must provide before work starts, and how changes to scope are agreed in writing.
Does enrichment data need extra security controls?
Yes, because lookup data often contains internal asset ownership, user directories or threat intelligence that should not be exposed beyond its intended audience. Protect the source files, database credentials and API keys with restricted access and documented rotation. Consider whether enrichment results will appear in searches visible to wider groups, and whether external API calls leave your network under an approved route. Confirm these questions with your security team before the first lookup goes live, since they affect both the data design and the Graylog roles that will read the results.
Which Graylog edition do we need for lookup tables?
Lookup tables, including the CSV File and MongoDB data adapters and the geo-ip adapter, are documented as part of Graylog's lookup table features, and they are used in pipeline rules, converters and decorators. Confirm the exact feature availability for your release in the Graylog documentation and your license terms before design starts, because some related features, such as parameters that use lookup tables, depend on an Enterprise license. Graylog Open is licensed under SSPL-1.0, which is not an OSI-approved open source license, so review that license with your legal team as well.
How DeployOpen scopes lookup enrichment work
Scoping starts with a list of sources and the fields each one must add to a message. DeployOpen is a Chainscore Labs company that plans, implements, integrates and supports self-hosted open-source platforms inside the client's own infrastructure. For lookup enrichment, that means working through each proposed data adapter (a CSV file, a MongoDB collection, a GeoIP database or an external API), confirming who owns the source data, how often it changes, and which message fields it will read from and write to. The output is a written field map: the input field, the lookup table, the cache settings, the target field and what happens when the lookup returns nothing. Teams that skip this step usually discover later that two pipelines write the same field name or that an external API is called for every message in a high-volume stream.
Build work then follows that map. DeployOpen configures the data adapters, caches and lookup tables, writes the pipeline rules that call them and attaches those rules to the right streams, so enrichment runs only where it is needed. Each rule is tested against sample messages captured from the real inputs, including malformed messages, messages with missing keys and lookups that should fail closed. Ownership of the enrichment data is agreed during scoping as well, with a named person for each source, a refresh method (manual upload, scheduled file replacement or a database sync) and a check that confirms the data is still current. Where an enrichment source is a threat intelligence platform such as MISP or OpenCTI, the scope also covers which indicator types are imported and how they are kept in step with the source.
The engagement ends with handover material that the operations team can use without the original builders. This includes a runbook for each adapter and cache, the refresh procedure with its owner, the test messages and expected results, and the steps to diagnose a lookup that returns stale or empty values. Most engagements are fixed-scope, so the sources, fields, deliverables and timelines are agreed in writing before work starts, and any change to the field map goes through the same written process. Ongoing work such as adding new sources, reviewing cache behavior after volume changes or keeping adapters working through Graylog upgrades can be covered by a retainer. Before requesting a quote, confirm your Graylog edition and which adapter types your license includes, because the scope depends on it.
