Pulling REST API logs into Graylog with the JSON path input
When a SaaS tool or internal API holds audit or event data but has no dedicated Graylog input, the JSON Path from HTTP API pull input is the built-in way to poll that endpoint and extract fields with JSON paths. On each poll, the input sends an HTTP request to the configured URL, reads the JSON response, and turns the values you select into Graylog messages. Because Graylog initiates every request, this is a pull input: the source does not push to Graylog, and the input is only as current as its polling interval. Graylog's inputs documentation lists this input as one that does not support TLS settings in the inputs table, so the transport security of the connection has to be planned separately before the input is configured.
That limitation shapes the design. Confirm in your Graylog version exactly how the input handles certificate validation, custom CA trust and mutual TLS, and whether the endpoint can be reached only through a proxy you control. If the API requires a bearer token or other header-based authentication, decide where that secret is stored and who can read it, since it will be entered into the input configuration. Also confirm the vendor's rate limits and how the API pages results, because a poll that misses a page or repeats a window will silently create gaps or duplicates in your audit trail. Test the endpoint from the Graylog node itself with the same headers, not from an administrator workstation.
Graylog Open is licensed under SSPL-1.0, which is not an OSI-approved open source license, so confirm that this license fits your procurement and redistribution policy before you build a pipeline around it. For APIs with heavy paging, strict rate limits or complex authentication flows, a small script or a collector that writes to a Graylog input may be more reliable and easier to monitor than this input. Decide that trade-off early, because it determines who owns the polling code after go-live.
What you need before creating the input
The JSON path from HTTP API input is only as dependable as the access and approvals behind it. Confirm the Graylog version, the backend, the API contract and the network path before anyone builds the input, because most delays come from these checks rather than from Graylog configuration.
Confirm the Graylog version
Check which Graylog release your deployment runs and read the matching documentation before you build the input. The current documentation covers version 7.2, but your installed release decides which input options and UI steps you will see. Note the version in the change record so later upgrades can be checked against the same input settings. If you run an older 4.x or 5.x release, confirm the input exists there before planning around it.
Running nodes and a healthy backend
Graylog needs at least one running server node connected to a reachable Data Node or OpenSearch cluster before an input can store anything. Confirm the MongoDB configuration database is also up, since input definitions live there. Check cluster health and free disk space first, because an input that polls successfully still loses value if the search backend is rejecting writes.
API documentation and endpoint details
Collect the endpoint URLs, the HTTP method, the request parameters, and the JSON structure the API returns, including where the event list sits in the response. The JSON path setting depends on that structure, so a sample response saved from a test call is more useful than a description. Record the pagination method, because a poll that reads only the first page will silently miss events.
A read-only test account
Request a service account or API client with read-only scope limited to the audit or log resources you need. Ask the API owner which credential type is issued, such as a bearer token, API key or OAuth client, and how it expires or rotates. Confirm the auth method and the rate limit policy with the vendor or internal API owner before you configure polling, because the polling interval must stay inside those limits.
Network egress and proxy approval
Every Graylog server node that polls the API must reach the API host over HTTPS, so firewall rules must allow egress from each node, not only from a single admin workstation. Check whether outbound traffic goes through a proxy, and whether the API owner or your network team must allow-list the Graylog node addresses. Graylog's JSON path input does not expose TLS settings in its inputs table, so confirm how the API certificate chain is trusted in your environment before testing.
Configure the JSON path from HTTP API input step by step
This sequence takes a REST endpoint from a test credential to a verified Graylog input that polls on a set interval, extracts fields from each record, and lands messages in the stream and index set you chose. Each step ends with a check so the team knows whether the input is working before it goes live.
Create a read-only test credential
Ask the API owner or the SaaS administrator for a dedicated test credential with read-only access to the data you need. Store it in your secrets manager, not in a notebook or chat thread, and confirm the credential's scope, token lifetime and rotation process with the vendor's documentation. Record the API's stated rate limits and any pagination behavior while you have the owner's attention. You are done when you can call the endpoint with that credential from a workstation and receive a valid JSON response with the records you expected.
Add the JSON Path from HTTP API input
In the Graylog web interface, open System, then Inputs, and launch the JSON Path from HTTP API input on a Graylog node that can reach the endpoint through your egress rules. Give the input a clear title that names the source system and environment. Note that this input does not offer TLS settings in the inputs table, so confirm with the vendor and your network team how the connection is protected before you enter any credential. The input appears in the list as running once it starts without configuration errors.
Set the endpoint, method and headers
Enter the full endpoint URL, the HTTP method the API expects, and the authentication header that carries your test credential. Use the header format the vendor documents, and keep query parameters for time windows or filters in the URL so each poll requests the same shape of data. Check that the endpoint returns HTTP 200 from the Graylog node itself, not only from your workstation, because firewall and proxy paths often differ. You are done when a manual test request from the node returns the same record count you saw in step one.
Define the JSON path and field extraction
Set the JSON path that points to the array holding the records, then map the fields you want to keep from each record. Keep the mapping narrow at first, such as event time, user, action and source IP, so the index does not fill with fields you will never search. Take a saved sample response and confirm the path resolves to an array of objects, not a single nested object, because the structure changes what each message contains. You are done when a test run shows one message per record with the fields named as you intended.
Set the polling interval within the API limits
Choose a polling interval that stays inside the API's stated rate limits, counting every other client that uses the same credential or quota. Confirm with the vendor whether the limit is per token, per user or per organization, and whether it applies to the Graylog node's outbound address. Start with a conservative interval and watch for throttling responses before you tighten it. You are done when the interval is written down with the limit it respects, and the first several polls return without rate-limit errors in the input diagnostics.
Route messages to a stream and index set
Assign the input's output to a dedicated stream and an index set that matches the retention and access rules for this source. Separate streams for each API make it easier to apply permissions, alerts and retention later. Confirm the index set's rotation and retention settings against your storage plan before the first poll writes data. You are done when the new messages appear in the chosen stream and not in the default stream.
Verify one poll and the extracted fields
Run one poll, then open Graylog search with the stream filter applied and compare the message count against the number of records in the saved sample response. Check that each extracted field is populated and that timestamps parse correctly, since wrong time zones distort every later investigation. If counts differ, check the JSON path, pagination and whether the API returns the same records on repeated calls. You are done when one poll returns the expected message count, the fields are searchable, and the confirmed interval is recorded alongside the API's stated limits.
Choosing polling, authentication and rate settings by API type
Many SaaS APIs issue a long-lived bearer token that you create once in the vendor console and send on every request. This is the simplest case for the JSON path from HTTP API input, because the input can carry the token as a fixed header value without a refresh step. The main risks are token lifetime and storage: a token that never expires becomes a standing credential, so it must be scoped to read-only permissions and stored in a secret store rather than typed into input configuration by hand.
- Header format: Confirm whether the vendor expects
Authorization: Bearer <token>or a custom header name, and enter it exactly as documented. - Polling interval: Choose an interval that stays well under the vendor's documented rate limit, and widen it after the first week of observed volume.
- Duplicate records: Store a stable record ID or timestamp watermark and check whether the source returns overlapping windows on each poll.
- Rotation: Plan a manual rotation procedure, since the input will fail silently or with authentication errors when the vendor revokes the token.
- Scope: Request read-only audit or log scopes only, so a leaked token cannot change source data.
Before go-live, confirm in the running Graylog version how the input reports a 401 response and whether it retries.
APIs that require OAuth 2.0 client credentials or tokens that expire within hours change the design. A static header will stop working when the token expires, so you need either an input that refreshes tokens itself or a small intermediary that obtains a token and forwards records. Graylog's JSON path input does not document TLS settings for this source type, so credentials and payloads travel over whatever transport the input uses, and you should verify how the token request is made before relying on it.
- Token endpoint: Record the token URL, grant type, scopes and client secret owner before configuring anything.
- Refresh behavior: Confirm in the running version whether the input requests a new token on expiry or only when it starts.
- Secret handling: Keep client secrets in a managed secret store and restrict who can read the input configuration.
- Polling interval: Align the interval with token lifetime so that a poll does not land in a refresh failure window.
- Fallback: If the input cannot refresh tokens reliably, a scheduled script or collector that writes to a GELF or syslog input is often the more dependable choice.
Test a forced expiry in a non-production environment and watch what the input logs.
Most APIs return large result sets in pages, using either a cursor (a token pointing to the next page) or a time window (a start and end timestamp). Each style affects how you avoid duplicates and gaps. Cursors are tied to the source's ordering and can expire, while time windows depend on clock alignment and on how the source handles late-arriving events. Whether the JSON path input follows next-page links or cursors on its own must be verified in the running version, along with how it persists its position across restarts.
- Cursor pagination: Confirm the cursor field name and whether it is returned in the body or a header, and check how long cursors remain valid.
- Time windows: Overlap windows slightly and deduplicate on a record ID, because events written late can fall outside a strict boundary.
- Interval and page size: Match page size and interval so one poll finishes before the next one starts.
- Gaps after restarts: Test a restart mid-pagination and compare record counts against the source.
- Ordering: Confirm whether results are sorted by event time or by ingest time, since that decides which watermark you can trust.
Compare Graylog's counts with the source for a known time range before you rely on the pipeline.
Internal APIs usually sit behind an API gateway or reverse proxy that handles authentication, TLS and rate limiting. This moves some decisions out of Graylog and into the gateway. The input must present whatever the gateway expects, which may be a service account token, a mutual TLS client certificate or a network-level allowlist. Because the gateway often sets the limits, you should ask the platform team for the documented quota for the service account before choosing a polling interval, and confirm the network path from the Graylog server to the gateway.
- Service account: Create a dedicated read-only account for Graylog, not a personal account, so access can be revoked independently.
- Gateway limits: Ask for the per-client request quota and any burst limits, and set polling below them.
- Network path: Confirm firewall rules from every Graylog server node, since the input may run on any node.
- Certificates: Find out whether the gateway requires client certificates, which the JSON path input may not support directly.
- Ownership: Agree who gets alerted when the gateway changes a route or version.
If the gateway enforces mutual TLS, a collector running close to the API is usually easier to operate than forcing the input to match.
Mapping JSON fields to Graylog message fields
This table pairs each part of a typical REST API event record with the Graylog field it should populate and the reason for that choice. Read it as a starting schema to confirm against your vendor's sample responses, not as a fixed mapping. The main takeaway is that timestamp, unique identifier and outcome fields decide whether searches, de-duplication and streams work at all, so they need the most testing.
| Source element | Example JSON path (illustrative) | Graylog target and purpose |
|---|---|---|
| Record timestamp | $.events[*].published | Message time. Confirm the vendor's format and time zone with a sample, then test parsing before go-live. |
| Event identifier | $.events[*].id | A unique field used to detect duplicates when overlapping poll windows return the same record. |
| Actor (user) | $.events[*].actor.email | A user field for searches, dashboards and stream rules about who acted. |
| Target (resource) | $.events[*].target.id | A resource field so investigations can pivot from the actor to the object acted on. |
| Outcome or severity | $.events[*].outcome.result | A standard field used in stream rules and alert conditions. Map vendor values to a short, agreed set of terms. |
| Event type | $.events[*].eventType | A classification field for grouping and filtering. Keep the vendor's original value in a raw field for reference. |
| Nested context | $.events[*].client.ipAddress | Optional fields for source IP and client details. Confirm which nested keys are present on every record. |
| Field naming | Team schema | Names should follow your organization's logging schema so content, streams and searches stay consistent across sources. |
Securing credentials, transport and the polled data
The JSON path from HTTP API input does not offer TLS settings in the inputs table, so transport, credentials and stored data have to be handled as separate decisions. Each card below covers one of them and names what to verify before the input goes live.
Confirm how transport is protected
The JSON path from HTTP API input does not expose TLS settings in the inputs table, so the team must decide how traffic to the API is protected before the first poll. Options include an HTTPS endpoint the input can reach with the certificate chain it trusts, a local TLS-terminating relay or script that fetches from the API and forwards to Graylog, or a private network path that never leaves your controlled segment. Verify the choice by capturing traffic on the path and confirming that no token appears in cleartext.
Store tokens as Graylog-managed secrets
API tokens and authentication headers entered into an input should be treated as secrets, not as values pasted into a shared configuration file or a ticket. Check how your Graylog version stores and displays the value after saving, and confirm that it is not returned in plain text to users who only need to view the input. Record which token belongs to which input so that rotation and revocation do not depend on memory.
Limit each token to read-only scope
Create a dedicated service account or API token for each polled source, with read-only permissions limited to the endpoints the input actually calls. Ask the SaaS vendor or internal API owner which scopes exist, because scope names differ between products. Test the token with a direct request before it is placed in Graylog, and confirm that write or administrative calls are refused.
Plan token rotation
Tokens that never expire turn a single leak into long-term access to the source system. Agree a rotation interval with the owner of each API, and plan the change window so that the new value is entered into the input and the old one is revoked after the first successful poll. Confirm whether the source supports overlapping tokens, since that determines whether rotation causes a gap in ingestion.
Restrict who can edit inputs
Anyone who can edit a polling input can usually change its URL, headers or target, which can redirect the stored credential to a server you do not control. Limit input editing to a small administrator group, separate it from search and dashboard roles, and review changes to input configuration. Confirm in your Graylog version which role grants input management and test that a search-only user cannot open the input settings.
Treat polled data as sensitive
Audit trails, customer records and identity events pulled from an API are written to Graylog indices like any other message. They therefore inherit the retention, access and backup decisions made for the cluster, and they may contain personal data that your privacy rules cover. Classify each polled source before ingestion, restrict the streams and index sets that hold it, and confirm that its retention matches the policy for the data itself.
Why REST API pulls fail and how to fix them
This table pairs each common failure in a Graylog JSON path from HTTP API pull with its likely cause and the fix, and points to the setup step where you verify it. Read it when a pull stops returning messages, returns errors, or returns duplicates, and work down the causes in the order the setup sequence checks them. The main takeaway is that most failures trace back to credentials, request rate, or how the time window and pagination are defined.
| Symptom | Likely cause | Fix and where to check it |
|---|---|---|
| 401 or 403 responses | Token expired, rotated, or issued without the scope the endpoint needs | Confirm the token's validity period and scopes with the API vendor, then replace it in the input's authentication header and test one request by hand before saving the input |
| 429 or other rate limit responses | Polling interval shorter than the API's quota allows, or several inputs sharing one key | Lengthen the polling interval, give each input its own credential where the vendor allows it, and confirm the quota in the vendor's documentation during interval planning |
| Duplicate messages after a restart | Overlapping time windows or a cursor that resets when the input restarts | Use a stored cursor or a non-overlapping window, then check message counts for the first two intervals after a restart in the input's diagnostics |
| Missing records | Pagination not followed, so only the first page of results is ingested | Confirm how the API returns next-page links or cursors, configure the input to follow them, and compare a sample count from the API with the count in Graylog for the same window |
| Wrong or shifted timestamps | API returns timestamps in a time zone or format Graylog does not parse as expected | Identify the timestamp field and its format in a raw response, set the JSON path and time format to match, and check that search results sort by the expected event time |
| TLS handshake or certificate errors | Endpoint certificate chain not trusted by the Graylog server, and the JSON path input does not offer TLS settings | Confirm the chain is trusted on the Graylog server's operating system, or place a trusted reverse proxy or script in front of the endpoint if the input cannot validate it |
| Connection timeouts or no response | Firewall, proxy, or egress rule blocking outbound HTTPS from the Graylog server or its network | Test connectivity from the Graylog server itself with the same URL and headers, then request the outbound rule from your network team using the vendor's documented host names |
When a script or collector is more reliable than a pull input
Use the built-in JSON Path from HTTP API input only when the endpoint is simple, authentication is a static header, polling volume is modest and the transport requirements can be met by the input as configured. In practice that means one GET request returning a JSON document, a fixed Authorization header or API key, a predictable response shape that a JSON path can address, and no requirement for TLS settings the input does not expose. The topic facts note that this input does not support TLS settings in the inputs table, so confirm the endpoint's certificate chain and any mutual TLS or custom CA needs before committing to it. Verify these points against the current inputs documentation for your Graylog version, since input options change between releases.
Choose a scripted poller instead when the API needs a token refresh, paginates in ways a single request cannot follow, requires retries with checkpoints, or needs a local buffer so that events are not lost during an outage. A poller that runs on a schedule, stores its own cursor or last-seen timestamp, and forwards events to a GELF input, a Beats input or a syslog input gives you control over TLS, retry behavior and backoff. The same approach helps when the team wants the collector to keep running and queuing while Graylog restarts, because the Graylog input only pulls on its own schedule and does not hold a checkpoint you can inspect. Plan the checkpoint store and the failure alerts as part of the build, not as an afterthought.
Whichever path you pick, test it against the API's documented rate limits before going live. Measure how many requests one poll cycle makes at your intended interval, how the provider signals throttling (commonly HTTP 429 responses with a retry hint, though you should confirm the exact behavior with the vendor), and how long a full backfill takes at that pace. Other Graylog input types, such as syslog, Beats and cloud-specific inputs, are covered in the sibling topics on this page, so compare them there if the source can also push events. A reliable choice is the one that still delivers complete data when the API throttles you, the token expires or Graylog is restarted.
Questions teams ask before pulling REST API logs
How much effort does pulling logs from a REST API into Graylog take?
The effort depends mostly on how many endpoints you pull, how the API authenticates, and how it paginates results. A single endpoint with a static token and a flat JSON response is a small job. Each additional endpoint adds a configuration, a field mapping and a test pass. Effort also grows with OAuth refresh flows, cursor or date-based pagination, nested JSON that needs extraction, and volume that forces you to tune polling intervals. Ask for the work to be counted per endpoint and per authentication pattern, with testing listed as its own line, so the quote shows where the hours go.
Can our own team build a REST API input without outside help?
A skilled Graylog administrator can usually handle a simple API that uses a static token, returns flat JSON and needs no pagination. The work becomes harder when the API uses OAuth with token refresh, returns large result sets, enforces strict rate limits, or needs duplicate handling across polls. Those cases need careful testing against the vendor's behavior, and mistakes show up as gaps or repeated messages in search results. Assess your team against those specific patterns before deciding, and confirm the authentication and rate limit details in the vendor's own API documentation.
What do we need to provide before a REST API log integration starts?
Most of the delay in these projects comes from access, not configuration. Plan to provide:
- Owner approval from the team that runs the API, along with the scopes or roles it will grant.
- A dedicated test account or API client with read-only access to the relevant logs.
- Network approval for the Graylog nodes to reach the API endpoint over HTTPS.
- Sample responses from each endpoint, including error responses and an empty result.
- A written sign-off on how long the logs are retained in Graylog and who may search them.
Does the JSON path from HTTP API input require a Graylog Enterprise license?
Confirm this against the Graylog inputs documentation before you plan the budget, because licensing for this specific input is not settled in the material we rely on here. Graylog Open is licensed under SSPL-1.0, and some inputs require an Enterprise license, so the input you choose determines whether you need one. Ask your Graylog contact or check the inputs table to confirm whether the JSON path from HTTP API input is available in your edition. Also note from the same documentation that this input does not support TLS settings, which affects how you protect the connection.
How do we handle API rate limits and duplicate log messages?
Set the polling interval so that each poll stays under the vendor's published rate limit, and back off when the API returns throttling responses. Duplicates usually appear when a polling window overlaps the previous one or when the API returns the same record across pages. Use a stable identifier from each record, such as an event ID, and filter on a time cursor that moves forward only after a successful pull. Test both behaviors with a window that spans a known set of events, and confirm the limit values with the vendor rather than assuming them.
When is a script or collector more reliable than a Graylog HTTP input?
A script or collector is often the better choice when the API needs multi-step authentication, complex pagination, request signing, or transformation before the data is useful. It also fits when you need local buffering to survive Graylog outages, or when the API's rate limits require shared state across runs. In those cases a small script can fetch records, handle retries and forward them to a GELF or syslog input. Weigh that against the extra component you will have to monitor, patch and document.
What does a fixed-scope REST API integration from DeployOpen include?
A fixed-scope engagement covers the work agreed in writing before the start: the endpoints in scope, the authentication method, the Graylog input or collector configuration, field mapping, test cases, and handover documentation. Your quote should list each endpoint, each access request you need to make, and the verification steps. DeployOpen does not publish a price list, so pricing comes only from a written quote that follows from that scope. Ask for the scope items and the exclusions to be stated explicitly so you can compare offers on equal terms.
How DeployOpen scopes and delivers REST API log pulls
DeployOpen plans REST API log pulls in Graylog as a written, fixed-scope piece of work. We confirm the source API, the input behavior and the rate limits first, then build, test and document the pull inside your own Graylog environment.
How we scope
- We list each API endpoint, the fields you need and the data volume you expect per poll.
- We confirm the authentication method, token lifetime and the permissions the service account requires.
- We record the polling interval, the vendor's rate limits and the fallback path if the API throttles us.
How we build
- We configure the JSON Path from HTTP API input in your Graylog and map the response fields into your streams.
- We check each pull against a test endpoint and confirm TLS behavior, since this input does not expose TLS settings.
- We decide with you whether a script or a collector such as Graylog Sidecar is the more reliable path for a given source.
How we hand over
- We deliver a written record of the endpoints, headers, intervals, extractor settings and known limits for each pull.
- We document the steps to rotate tokens, change polling intervals and add a new endpoint without breaking existing streams.
- We walk your operations team through failure checks, including how to spot empty polls, errors and stalled inputs.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer after handover.
- We can review pull behavior after vendor API changes or Graylog upgrades, based on the agreed scope.
- Ask any provider to confirm in writing how API version changes and credential rotation are covered.
