How Graylog collects CloudTrail logs through S3 and SQS
A team that needs AWS API activity in Graylog for investigations and retention should use the AWS CloudTrail input. This input polls an SQS queue for notices of new CloudTrail log files written to S3, then retrieves those files and passes their events into Graylog. The pattern is an event-driven pull: AWS announces each new file, and Graylog fetches it, so the Graylog node never needs to list or scan the whole bucket.
The work splits cleanly between two owners. The AWS account owner creates the organization trail, the S3 bucket that receives the logs, the SQS queue, the bucket notification that writes to that queue, and an IAM role with sqs:ReceiveMessage, sqs:DeleteMessage and s3:GetObject permissions. The Graylog operator then configures the input with the queue and bucket details, the region and the authentication method, and checks that events arrive in the expected streams. Agree on who owns each resource before anyone clicks, because a mismatch in the queue name, the bucket region or the role trust policy is the most common reason the input stays silent. The full input configuration is documented in the Graylog AWS CloudTrail input docs.
Confirm three points with your AWS team before you start: whether the trail is single-account or organization-wide, which account hosts the bucket and queue, and whether the Graylog nodes can reach AWS endpoints in that region from your network. Graylog 7.1.0 added a CloudTrail input setup wizard and support for automatic authentication, so check which of these your running version provides and record the decision in your runbook.
What you need in place before creating the input
The CloudTrail input depends on four AWS pieces working together: the trail, the bucket, the queue and the role. Graylog also needs a supported version and cluster behind it. Check each layer before you open the input setup, because a gap in any one of them usually shows up as an input that starts but never ingests.
Graylog version and cluster
Start from a Graylog release that includes the CloudTrail setup wizard, which arrived in Graylog 7.1.0. The current documentation covers Graylog 7.2, so plan against that line for new builds. Confirm that your Graylog server nodes, MongoDB and Data Node or OpenSearch versions match the compatibility matrix for the release you choose, since Graylog 7.0 to 7.2 have specific MongoDB and OpenSearch ranges.
Existing CloudTrail trail writing to S3
The input reads CloudTrail log files from an S3 bucket, so a trail must already deliver to that bucket before you configure anything in Graylog. Decide whether the trail is organization-wide or single-account, and which region or regions it covers. Confirm the bucket's default encryption and any KMS key policy requirements against AWS documentation for your account, because Graylog's role must be able to read objects that are encrypted with that key.
SQS queue receiving S3 notifications
The input does not list the bucket. It polls an SQS queue that receives a notification each time CloudTrail writes a new log file. Create the queue, allow S3 to send messages to it, and configure the bucket's event notifications to target that queue. Keep the queue dedicated to this input, because another consumer that deletes messages will cause Graylog to miss log files.
IAM role with the required actions
Create an IAM role for the Graylog nodes that grants sqs:ReceiveMessage and sqs:DeleteMessage on the queue, and s3:GetObject on the CloudTrail bucket prefix. Scope these to the specific queue and bucket ARNs rather than wildcards. If you use an instance profile, attach the role to every Graylog node that will run the input, and confirm which node runs the input before you assign it.
Network egress to AWS endpoints
Each Graylog node that runs the input must reach the SQS and S3 endpoints for your AWS region over HTTPS. In private subnets, this usually means a NAT gateway or VPC endpoints for SQS and S3. Test egress from the node itself, not from your workstation, since the input runs on the Graylog server. Also confirm that any outbound proxy or firewall rule allows these hosts and that the node's clock is synchronized, because AWS request signing depends on it.
Authentication method and change control
Decide how Graylog will authenticate to AWS before you build the role. The 7.1.0 changelog notes support for automatic authentication, but confirm which method your environment allows, such as an instance profile or static keys, with your security team. Record the trail name, bucket, queue and role in your change ticket so the input can be reviewed, rotated and rebuilt later without guessing which resources it depends on.
Configure the CloudTrail input in sequence
This sequence takes you from an existing or new CloudTrail trail to a running Graylog input that delivers AWS API events into a stream and index set you can search. Each step ends with a check so you can confirm the path works before moving on.
Confirm or create the CloudTrail trail
An AWS administrator confirms that a trail exists and writes its log files to an S3 bucket in the account or region you plan to collect from. If no trail exists, create one that delivers to a dedicated bucket and record the bucket name, the prefix and the account ID. Decide whether the trail covers one account or an organization, because that choice changes which accounts send notifications to your queue. Done means the trail status is logging and the bucket already holds recent CloudTrail files under the expected prefix.
Create the SQS queue for new object notices
Create a standard SQS queue in the same region as the bucket, and add an access policy that lets the S3 service send messages to it, limited to your bucket and account. Then configure S3 event notifications on the bucket so that object creation events for the CloudTrail prefix are sent to the queue. Confirm the queue policy in the console, then upload or wait for a new CloudTrail file and check that a message appears in the queue's message count. The check is a visible message that references the new object key.
Add a dead-letter queue for failed messages
Create a second SQS queue to act as a dead-letter queue and attach it to the main queue through a redrive policy, so that messages Graylog cannot process after repeated receives move aside instead of cycling forever. Keep the dead-letter queue in the same account and region. Confirm the redrive setting on the main queue and note the empty state of the dead-letter queue as your baseline. An empty dead-letter queue after your first test events is the check you will reuse during operations.
Create the IAM role with trust and permissions
Create an IAM role for the Graylog host or instance profile to assume, with a trust policy that names the principal your Graylog nodes run under. Attach a permissions policy that grants sqs:ReceiveMessage and sqs:DeleteMessage on the queue and s3:GetObject on the CloudTrail prefix of the bucket. Graylog's CloudTrail input documentation names these three actions as the minimum. Do not grant broader S3 or SQS actions unless your security team requires them. Check the role with the policy simulator or a test call before you add it to Graylog.
Choose manual or automatic authentication
Decide how Graylog obtains AWS credentials. Manual authentication means you supply an access key and secret for a principal with the permissions above, which you then rotate and protect. Automatic authentication uses the role attached to the instance or environment, which removes stored long-lived keys. Graylog 7.1.0 added support for automatic authentication, so confirm against the current input documentation which option your version and hosting model support. Record the decision and the owner of the credentials, because it affects rotation and audit.
Create the input or run the 7.1 setup wizard
In Graylog, open the input page and add the AWS CloudTrail input, entering the queue URL, region and the authentication method you chose. On Graylog 7.1 or later, the CloudTrail input setup wizard can guide these fields in sequence, which reduces the chance of mismatched region or queue names. Start the input and check its state in the input list. Done means the input reports running with no startup errors in the server log, and the metrics show the input polling the queue.
Assign a stream and index set
Route CloudTrail events to a dedicated stream so that access rules, alerts and dashboards can target them without filtering the whole index. Assign that stream to an index set whose rotation and retention match your log policy, so CloudTrail data is kept for the period your organization requires. Confirm the stream rules and index set on the input's routing settings before the first events arrive. Changing the index set later does not move existing messages, so verify this choice now.
Validate that events arrive and clear the queue
Generate a known AWS API action, such as listing buckets or describing an instance, and wait for the next CloudTrail delivery, which is not instant. Search Graylog for the event name, account ID or user identity, and confirm the message appears in the assigned stream with parsed fields. Then check that the main SQS queue drains and the dead-letter queue stays empty. If messages remain in flight or the dead-letter queue fills, review the IAM policy, the bucket notification prefix and the input log before you treat the setup as complete.
How CloudTrail fields and account scope map into Graylog
Each CloudTrail record arrives in Graylog as one message, and the fields the team searches on decide whether investigations work. The fields most teams need are the event name (eventName), the source address (sourceIPAddress), the caller identity under userIdentity, the account that owned the event (recipientAccountId), and the region (awsRegion). Before building streams, alerts or dashboards on these names, pull one sample event from the trail in the environment and check the exact field names and nesting as Graylog stores them after the input and any extractors run. Field naming can differ between the raw record and the message Graylog produces, so do not assume a schema. Confirm the mapping against the input's documentation for your Graylog version, using the AWS CloudTrail input documentation as the starting point.
Account scope is decided upstream, in AWS, before Graylog sees anything. A multi-account trail delivers the logs of several accounts to one S3 bucket, and an organization trail covers every member account in AWS Organizations. In both cases the shared SQS notification queue receives events for all covered accounts, so one Graylog input can ingest the whole scope. That is efficient, but it means the input's IAM role reads logs for every account the trail covers, and any account you did not intend to include will appear in the same index set. Decide which accounts belong in the trail, and whether separate trails per environment would give cleaner stream rules and retention, before the input is pointed at the queue.
Record the account scope as a design decision with three checks: the list of accounts expected to appear, the values of recipientAccountId you see in a test window, and the streams that route each account or region. A missing account usually points to a trail or bucket policy gap rather than a Graylog fault, so compare the trail's configuration with what arrives. Plan these checks as part of the verification step so that a later review can show which accounts feed the cluster and why.
CloudTrail and IAM elements mapped to Graylog settings
This table maps each AWS element in the CloudTrail pipeline to the Graylog input setting or stream it feeds, and names who owns each side. Read it top to bottom as the path a log file takes from the trail to a search result. Confirm every field name against the input form in your Graylog version, since the setup wizard added in Graylog 7.1 may change labels and defaults.
| AWS or Graylog element | What it does in the pipeline | Graylog setting or stream it feeds | Owner on each side |
|---|---|---|---|
| CloudTrail trail | Writes API activity log files to an S3 bucket on a schedule | Source of all events; no direct Graylog setting | AWS account owner (cloud team) and security team |
| S3 bucket and prefix | Stores the CloudTrail log files that Graylog retrieves | Bucket and prefix fields in the CloudTrail input; confirm the prefix matches the trail path | AWS owner sets the bucket; Graylog admin enters the values |
| SQS queue | Receives a notification for each new log file written to S3 | Queue URL or name in the input; the input polls this queue | AWS owner creates and subscribes it; Graylog admin configures the input |
| S3 event notification to SQS | Sends an object-created message from the bucket to the queue | No Graylog field; a prerequisite that must deliver messages before the input has anything to read | AWS owner; confirm the queue policy allows the bucket to send |
| IAM role used by Graylog | Grants the identity Graylog uses to read the queue and bucket | Authentication method in the input; confirm whether the wizard uses an instance role or keys | AWS owner defines the role; Graylog admin selects the authentication option |
| IAM actions sqs:ReceiveMessage and sqs:DeleteMessage | Let Graylog poll the queue and remove processed notifications | Required for polling; missing actions stop ingestion without changing the S3 data | AWS owner grants; verify with a test read after the role is attached |
| IAM action s3:GetObject | Lets Graylog read each CloudTrail log file named in a notification | Required for retrieval; confirm the bucket resource scope covers the trail prefix | AWS owner grants; security team reviews the resource scope |
| AWS region | Sets where the queue and bucket are reached | Region setting in the input; confirm it matches the queue and bucket region | AWS owner confirms; Graylog admin enters the value |
| Graylog stream and processing pipeline | Routes CloudTrail messages to a stream with its own index set | Target stream and index set chosen after the input is created | Graylog admin and security analysts agree on routing |
Securing the CloudTrail pipeline
The CloudTrail pipeline touches your AWS account, an SQS queue, an S3 bucket and the Graylog input configuration, so each link needs its own access boundary. The cards below cover the decisions that keep that chain narrow and the checks to run before you rely on it.
Scope the IAM role to three actions
The Graylog role needs only sqs:ReceiveMessage, sqs:DeleteMessage and s3:GetObject. Bind those actions to the specific queue ARN and the specific trail bucket prefix, and grant nothing else, including list or write permissions. Review the policy after each Graylog upgrade, since the required permissions are set by the input documentation and can change between releases. Confirm the current list against the input documentation before you deploy.
Limit who can edit the input
Anyone who can change the input can repoint it at a different queue, bucket or role, which changes what Graylog reads and which account it trusts. Restrict input creation and editing to a small platform group, and keep the setup wizard and configuration changes under change control. Review input configuration changes in Graylog's audit trail and compare them against the AWS side, so a changed queue name never goes unnoticed.
Protect stored AWS credentials
Where you use access keys rather than an instance or assumed role, the secret is stored in Graylog's configuration and must be treated as sensitive. Rotate keys on a schedule, avoid long-lived keys on shared hosts, and confirm that configuration backups and support bundles do not expose the secret. Ask your team to confirm how the wizard handles automatic authentication in your version before choosing between keys and roles.
Encrypt and route the data path
Traffic from Graylog to AWS endpoints runs over HTTPS, so the server needs outbound access to the SQS and S3 endpoints for your region. Route that traffic through a controlled egress path if your network policy requires it, and confirm that any proxy or VPC endpoint you use supports the calls. Check that the Graylog web interface and API are served over TLS, because the input configuration and its credentials pass through them.
Keep CloudTrail evidence in S3
The S3 bucket is the system of record for CloudTrail files, while Graylog indices are a searchable copy that rotate and expire on your index set settings. Set retention and object lock on the bucket according to your own requirements, separately from Graylog retention, so that deleting or resizing an index never removes the source files. Confirm with your legal or compliance owner how long the source logs must be kept.
Why CloudTrail events stop arriving in Graylog
This table lists the failures that most often stop CloudTrail events from reaching Graylog, with the likely cause and the fix for each. Work through the rows in order, since the notification path (trail, S3, SQS) must be correct before IAM and routing problems matter. Confirm each item against the current AWS CloudTrail Input documentation for your Graylog version.
| Symptom | Likely cause | Fix |
|---|---|---|
| No messages reach the input at all | S3 event notifications are not configured to publish to the SQS queue, or the queue policy does not allow S3 to send messages | Confirm the bucket emits object-created notifications to the queue, and that the queue access policy allows the bucket to send messages. Upload a test log file and check the queue for a message. |
| Input runs but reports no messages | The IAM role lacks one of the three required actions: sqs:ReceiveMessage, sqs:DeleteMessage or s3:GetObject | Add all three actions to the role the input uses, scoped to the queue and the trail bucket. Restart the input and recheck its status after the next delivery. |
| Queue depth keeps growing | Graylog polls messages but does not delete them, usually because sqs:DeleteMessage is missing or the input cannot reach the queue endpoint | Grant sqs:DeleteMessage, check outbound network access to SQS from the Graylog server nodes, and confirm the queue's dead-letter or retention settings match your plan. |
| Notifications fail or arrive in another region | The queue or bucket sits in a different AWS region from the one the input is configured for, so the calls go to the wrong endpoint | Place the queue and bucket in the region the input expects, or set the input's region to match where they live. Verify the region in both the AWS console and the input configuration. |
| No events from a multi-account or organization trail | The trail is created in one account, but the queue and IAM role live in another, or the trail does not deliver to a bucket the input's role can read | Confirm the organization trail writes to a bucket the input role can read across accounts, and that the queue receives notifications for that bucket. Test with a log file from a member account. |
| Events arrive but appear in an unexpected stream or index | Stream rules or the index set do not match the CloudTrail messages, so they fall through to the default stream or index | Check the input's routing and the stream rules that match CloudTrail fields. Confirm the target index set and retention, then search for recent messages in that index set to verify routing. |
| Events are duplicated or delayed | Retries after a failed delete or processing error redeliver the same notification, and polling intervals spread out delivery | Confirm sqs:DeleteMessage works, review input logs for processing errors, and allow for the polling delay before judging the input healthy. |
Questions teams ask before setting up CloudTrail in Graylog
How much does it cost to collect CloudTrail logs in Graylog?
The cost comes from four places, and only some of them are Graylog's. Graylog Open is free to download and run, according to the Graylog pricing page, so the software itself carries no license fee at that tier. The remaining costs are:
- AWS charges for the trail, the S3 storage that holds the log files, and the S3 and SQS requests the input makes. No AWS rates were supplied for this page, so check current AWS pricing for your regions and log volume.
- Infrastructure for the Graylog nodes, MongoDB, and the search backend that stores the data.
- Team time for IAM, S3, SQS and Graylog configuration, plus ongoing reviews.
A DeployOpen quote follows from a written scope that lists the accounts, regions and retention targets involved.
How much Graylog storage will CloudTrail data need?
Start with the daily volume of CloudTrail data in your accounts, then multiply it by the number of days you must keep it and by 120 percent. Graylog's system architecture guidance uses this formula to size log storage. Graylog also advises giving the message journal between 3 and 5 times the daily ingest on SSDs. Get the daily volume from a representative period in CloudTrail and S3 rather than estimating it from the number of accounts. Retention is often set by your compliance or audit obligations, so confirm it with the owners of those requirements before you size the cluster. Retention can also be split across index sets, so older data may use different storage. Ask for the sizing calculation in any quote so you can check the assumptions.
Can our own team set up the CloudTrail input without a partner?
Yes, a team with AWS administration skills and Graylog administration skills can set up the input in-house. The work has two halves: the AWS side (the trail, the S3 bucket, the SQS queue that receives new-file notifications, and an IAM role) and the Graylog side (the input configuration, its routing and testing). Graylog 7.1.0 added a CloudTrail input setup wizard, which shortens the Graylog half. The tasks that usually take the longest are reviewing the IAM policies with security staff and testing with real accounts. A partner is useful when you want a fixed scope, a documented result and handover, or when many accounts must be onboarded at once. DeployOpen works under written, fixed-scope agreements.
What do the AWS and Graylog owners need to provide?
Both owners need to agree on the design before work starts. The AWS owner typically supplies:
- The CloudTrail trail, the S3 bucket that receives its files, and the SQS queue that gets notifications of new files.
- An IAM role for the input that grants sqs:ReceiveMessage, sqs:DeleteMessage and s3:GetObject. Review these permissions against your least-privilege policy.
- Access to the accounts and regions in scope, plus a decision on multi-account trails.
The Graylog owner supplies administrator access to the cluster, a network path from the Graylog nodes to AWS endpoints, and a target stream and index set for the data. Both teams should agree on who approves IAM changes and who receives the test results.
How long does validation take with many AWS accounts?
Validation time depends mainly on how many accounts, regions and trails are in scope, because each trail and queue path needs its own check. A single account with one trail is the simplest case. Multi-account setups add IAM and queue configuration for each account, plus tests that events from every account arrive, parse and land in the right stream. The time also depends on how quickly the AWS and Graylog owners approve the IAM policies and change windows. No standard duration is established for this work, so ask for the timeline in your written scope and confirm which accounts are included in the first phase. Plan for a test period after go-live as well, so you can confirm that log volume and parsing match your expectations.
Does the CloudTrail input require a Graylog Enterprise license?
The CloudTrail input is documented as part of Graylog Open, but confirm this in the documentation for the exact version you run before you commit to a design. Graylog Open is licensed under SSPL-1.0, which is not an OSI-approved open source license, so the license terms matter for your legal review. Enterprise adds features such as the data lake with archive restore, data tiering, a correlation engine and 24/7 support, according to Graylog's pricing page. Those features may matter for CloudTrail retention or detection, but they are separate from the input itself. Ask Graylog or your DeployOpen scope document to name the features and the edition each needs.
What should we check when CloudTrail logs stop arriving in Graylog?
Work from the path the data takes. First, confirm that the trail is still delivering new files to the S3 bucket. Next, check that the SQS queue receives notifications and that messages are not piling up, because a backlog points to a polling or permission problem. Then confirm that the IAM role still has sqs:ReceiveMessage, sqs:DeleteMessage and s3:GetObject, and that a recent policy change or key rotation did not remove them. Finally, check the Graylog input status, its logs, and the stream routing rules. Keep a short runbook for these steps, with the owner of each check named, so that the on-call team can separate an AWS fault from a Graylog fault quickly.
How DeployOpen scopes a CloudTrail onboarding
A fixed-scope engagement starts by pinning down which AWS accounts, trails and regions the Graylog input will cover, because that decision drives every later step. The team inventories the organization's accounts and any existing trails, decides whether a single organization trail or per-account trails fits the governance model, and records which regions must be captured. Each trail is mapped to one S3 bucket and one SQS queue, and the scope document names the bucket owner, the KMS key policy if encryption is used, and the account that hosts the queue. Nothing is built until these items, along with the deliverables and timeline, are agreed in writing.
The IAM design comes next. DeployOpen proposes a dedicated role for the Graylog input that carries only sqs:ReceiveMessage, sqs:DeleteMessage and s3:GetObject on the specific queue and bucket, plus whatever decryption permission the chosen key requires. The client confirms how the Graylog nodes will obtain that role, whether through an instance profile or static credentials, and who approves changes to the policy. The Graylog input configuration, the stream routing that separates CloudTrail events from other sources, and the index set retention for them are documented together so that search access and retention match the organization's rules.
Validation and handover close the engagement. The test plan generates known API calls in a non-production account, confirms the matching events arrive in the intended stream, checks that processed fields are populated, and verifies that the queue drains as expected. The handover package covers the runbook, the permission list and the test evidence. Ongoing support, upgrades of Graylog and the input, and periodic reviews of trail coverage are available as retainers once the initial scope is complete.
How DeployOpen delivers Graylog integrations
DeployOpen plans, builds and hands over the CloudTrail input for Graylog inside your own AWS accounts, with the trail, queue, bucket policy and IAM role scoped before any configuration starts. Engagements are fixed-scope, so the agreed deliverables and timeline are written down before work begins.
How we scope
- We list the AWS accounts, regions and trails in scope, including whether one organization trail or several single-account trails feed Graylog.
- We confirm the S3 bucket, SQS queue and IAM role names, along with who owns each resource and who approves changes.
- We document the Graylog version, input configuration and network paths in the written scope, so the quote follows from defined work.
How we build
- We create or verify the CloudTrail trail, its S3 delivery and SQS event notifications in a non-production account first.
- We scope the IAM role to the minimum actions the input needs, including receive and delete on the queue and read access on the objects.
- We test field parsing, stream routing and index set placement in Graylog using sample CloudTrail events before enabling production traffic.
How we hand over
- We deliver a runbook covering the trail, queue, role and input settings, with each resource identified by name and owner.
- We record the verification steps used to confirm events arrive, parse and land in the intended stream, so your team can repeat them.
- We walk your operations and security staff through the setup, including what to check after a Graylog or AWS change.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer, covering changes to the Graylog input after upgrades.
- We can review IAM permissions and queue backlog during scheduled checks, so gaps are caught before they affect investigations.
- Retainer scope, response expectations and reporting are agreed in writing, and we confirm them against your environment before work starts.
