How the Graylog AWS CloudWatch Kinesis input gets AWS logs into Graylog
The supported way to bring CloudWatch and VPC flow logs into a self-hosted Graylog is the AWS Kinesis/CloudWatch input. This input does not read CloudWatch directly. AWS must first stream the messages through Kinesis, and the Graylog input then reads that stream. Your design therefore has two halves: the AWS side, where subscriptions and streams are created and permissioned, and the Graylog side, where the input is configured to match. Both halves sit in accounts and networks you control, and the work is planned there before any configuration begins.
The input handles three formats, and choosing the right one for each source matters because it determines how messages arrive and how they are parsed. CloudWatch Logs covers messages from log groups, CloudWatch Flow Logs covers VPC flow records, and Kinesis raw logs covers data that another producer has already placed on a stream without the CloudWatch envelope. Inventory every log group and flow log you need, note which format each one uses, and decide whether one stream or several will carry them. Confirm the exact format options and any field handling in the Graylog input documentation for your version before you map sources to inputs.
Before you build, confirm three things with your AWS team: which accounts and regions generate the logs, which role the Graylog node will assume to read the stream, and how the Graylog node reaches the AWS endpoints from your network. Those answers drive the IAM policy, the subscription design and the firewall or proxy rules. Also ask for the stream throughput your sources produce, because it sets how many shards and how much Graylog capacity you need. If the sources are better served by the S3, CloudTrail or Security Lake inputs, decide that now, since switching methods later means rebuilding the IAM and transport design.
What you need before configuring the Kinesis input
Most delays in CloudWatch ingestion come from prerequisites checked too late. The cards below cover the Graylog version and edition, the input port, the AWS account and streaming path, IAM, and network egress, in the order you should confirm them.
Graylog version and edition
Confirm your Graylog release against the current documentation, which covers version 7.2, before you plan the input. The AWS Kinesis/CloudWatch input is a documented input type, so check that your edition exposes it and that your search backend matches the compatibility matrix for that release. Graylog Open uses the SSPL-1.0 license, so record the license terms your organization accepts before you build the pipeline on top of it.
Reachable Graylog input port
The Kinesis input runs on your Graylog nodes, so the port it listens on must be settled before the input is created. Decide which node or load-balanced address receives the traffic, and confirm the port does not clash with other inputs. If Graylog sits behind a load balancer, check how that balancer handles the input's traffic before you rely on it.
AWS accounts, regions and log groups
List every AWS account and region that produces logs you need, along with the CloudWatch log groups and VPC flow logs in each. Multi-account and multi-region designs multiply the streams, roles and credentials you must manage. Record which log groups carry sensitive data, since those choices affect access reviews later.
Kinesis stream and subscription path
Graylog reads CloudWatch messages from Kinesis, so a Kinesis stream must receive the data first. Plan the subscription from each log group or flow log to that stream, and confirm the stream's capacity and retention fit your event volume. Note that the input depends on this streaming path and cannot pull CloudWatch Logs on its own.
IAM permissions for the input
The input needs AWS credentials or a role that can read from the Kinesis stream and the related CloudWatch resources. Apply least privilege, scoping each permission to the specific stream and log groups. Confirm the exact IAM actions in the Graylog input documentation for your release rather than assuming them, and document who owns the role and how its keys rotate.
Network egress to AWS endpoints
Graylog nodes must reach the AWS service endpoints for the Kinesis and CloudWatch APIs in your chosen region. In restricted networks, check whether egress passes through a proxy, a NAT gateway or a firewall allowlist, and confirm the endpoint list in the Graylog documentation. Test connectivity from every Graylog node, not only one, because a single failing node can cause intermittent gaps.
Configuration sequence for the Kinesis/CloudWatch input
This sequence connects one CloudWatch log group to a self-hosted Graylog through Kinesis, starting in AWS and ending with test messages searchable in a Graylog stream and index set. Work through it in order with a single log group, so each check isolates one failure point before you widen the scope.
Create the Kinesis stream and delivery path
In the AWS account that owns the log group, create a Kinesis data stream to carry the CloudWatch messages. Then add a CloudWatch Logs subscription filter on one test log group that delivers events to that stream. The AWS account owner or cloud team performs this step. It is done when the stream's incoming record metric rises after the test log group receives new events. Record the stream name, region and log group name, because the Graylog input and later checks depend on them.
Create the IAM role and policy
Create an IAM role for the Graylog input with a policy that allows reading from the test stream. Create a separate role that lets CloudWatch Logs write to the stream when the subscription filter is set up. Scope both policies to the specific stream and log group resources, and confirm the exact actions the input needs in the Graylog input documentation before you attach them. It is done when a test read from the stream using these credentials returns records without access errors.
Confirm the network path to AWS
Ask the network team to confirm that every Graylog server node that may run the input can reach the Kinesis and CloudWatch endpoints for the chosen region, along with any token service your credential method uses. Check outbound rules at the firewall, proxy and security groups. It is done when a test connection from each candidate node succeeds using the same credentials. Record any proxy settings, because they affect the input configuration in the next step.
Create the AWS Kinesis/CloudWatch input
In Graylog, open the input setup and select the AWS Kinesis/CloudWatch input. Choose the log type that matches the data: CloudWatch Logs for log group events, CloudWatch Flow Logs for VPC flow logs, or Kinesis raw logs for other messages on the stream. The log type controls how messages are parsed, so set it deliberately. Enter the region, stream name and credentials, assign the input to a node, and start it. It is done when the input reports a running state with no errors on that node.
Route test messages to a stream
Create or choose a stream for the test log group and assign it to a dedicated index set for CloudWatch data, so retention and rotation can be set for this source. Add a stream rule that matches a field the input sets for the log group or source, and test the rule against a sample message before relying on it. Wait for the first events to arrive. It is done when the stream's message count increases and the messages appear under the stream and its index set.
Verify test messages in search
Run a search scoped to the stream and to a time range covering the test events, then filter on the log group name or log type field. Compare the message count with the number of events the test log group recorded in the same window, allowing for delivery delay. Open one message and check that the timestamp, message text and CloudWatch fields look correct for the chosen log type. It is done when the search returns the expected messages. Only after this check passes should you add more log groups to the subscription filter.
How CloudWatch and flow log fields land in Graylog
This table maps each source type the Graylog Kinesis/CloudWatch input supports to the message content you should expect, the stream or index set it should route to, and the fields your team will search on. Use it as a planning checklist, then confirm exact field names against the input output in your test environment before building dashboards or alerts.
| Source type | Expected message content | Route to stream or index set | Fields to search on |
|---|---|---|---|
| CloudWatch Logs | Log events from a log group, delivered through the Kinesis stream the subscription filter writes to | Dedicated stream per log group family, with its own index set sized for that application's volume | Log group and log stream names, message text, timestamp; confirm exact field names in test |
| CloudWatch Flow Logs | VPC flow records (version, account, interface, source and destination addresses and ports, protocol, action) as delivered by CloudWatch | Separate stream and index set, since flow volume is usually much higher than application logs | Source and destination address, port, protocol, action (ACCEPT or REJECT); confirm field names in test |
| Kinesis raw logs | Unwrapped payloads written directly to the Kinesis stream by your own producer, without CloudWatch envelope | Stream that matches the producing application, with parsing pipeline rules set for its format | Depends on the producer's schema; define field names in the pipeline and verify them in test |
| Log group to stream mapping | One log group should map to one Graylog stream so retention and access can be set per group | Stream rules on the log group or source field, checked in a test search before go-live | Log group name and account identifier used to route and filter |
| Index set retention | Flow logs and application logs often need different rotation and retention settings | Index set with rotation and retention chosen per data type, based on your retention policy | Not a search field; confirm rotation settings against the index model in your environment |
| Field naming and parsing | Message fields can differ from what teams expect, and naming may change between input versions | Processing pipeline rules that normalize field names before dashboards depend on them | Agree a naming convention with analysts and document it with the pipeline |
When Kinesis fits better than the S3, CloudTrail or Security Lake inputs
Choose the Kinesis/CloudWatch input when your application, Lambda and VPC logs already flow into CloudWatch Logs and you want them in Graylog with minimal new AWS plumbing. The input reads CloudWatch log messages through Kinesis, so the AWS side must first stream those messages into a Kinesis stream that Graylog can consume. The input also handles CloudWatch Flow Logs and Kinesis raw logs, which covers network visibility without building a second export path.
- Log types: CloudWatch Logs groups, VPC Flow Logs delivered to CloudWatch, and raw records already placed on a Kinesis stream.
- Delivery pattern: Near-continuous streaming from the AWS side into Kinesis, read by Graylog.
- Audience: Platform and application teams who own the log groups and need them searchable in Graylog.
- Ownership: Whoever manages the subscription filters and the Kinesis stream, plus the IAM role Graylog uses to read it.
Confirm in the Graylog AWS Kinesis/CloudWatch input documentation which stream and subscription settings your Graylog version expects before you design the pipeline.
Choose the CloudTrail input with S3 and SQS when your audit record is an organization trail that writes compressed event files to an S3 bucket and you want Graylog to react to new objects. The trail and its notifications are the key difference: Graylog works from object-level events rather than from a log stream, so the pipeline depends on the bucket, the SQS queue and the role that reads them. Security and audit teams usually own this path because CloudTrail is the account activity record they must defend.
- Log types: AWS management and data events recorded by CloudTrail, delivered as files.
- Delivery pattern: Batch objects in S3, announced through SQS notifications.
- Audience: Security operations and audit teams who need account activity for investigations.
- Ownership: The team that controls the organization trail, the bucket policy and the queue, which may differ from the team that runs Graylog.
Confirm multi-account trail handling and the setup steps in the Graylog CloudTrail input documentation before choosing this path for an organization with several accounts.
Choose a Security Lake path when your organization has already standardized security data in Amazon Security Lake and wants Graylog to consume that normalized store rather than each source separately. The value is consolidation: one data lake holds the findings and logs that multiple AWS services and partners publish, and the central security team decides the schema and retention. This path depends on Graylog features and the Security Lake data format at the time you build, so treat its details as a point to verify.
- Log types: Normalized security data that Security Lake collects from enabled AWS sources.
- Delivery pattern: Data held in the lake and read by the consuming tool, which is a different model from streaming.
- Audience: Central security or data teams that govern the lake for the whole organization.
- Ownership: The team that administers Security Lake, its subscribers and its retention, not the team that runs Graylog nodes.
Check the current Graylog documentation for any Security Lake input and its license requirements before you plan, because support and edition details change between releases.
The decision often comes down to who owns the pipeline and what each option costs to run. Kinesis suits teams that operate CloudWatch and want a direct stream, while S3 and SQS suit audit programs that keep CloudTrail as the system of record. Security Lake suits organizations that have chosen a central lake as their security data platform. Costs come from AWS services, such as Kinesis throughput, S3 storage and requests, and SQS messaging, as well as the Graylog sizing needed to index what arrives.
- Cost drivers: Volume of CloudWatch or CloudTrail events, the number of accounts, and retention periods on both AWS and Graylog.
- One-time work: IAM roles, streams or queues, and the first pipeline validation.
- Recurring work: AWS charges, Graylog storage and the team time to watch ingestion.
- Quote items to request: Who builds and who operates each AWS component, and what is priced separately from the Graylog license.
Ask your AWS account owners which log sources already exist before you select an input, because duplicate delivery of the same events inflates both cost and search noise.
Many organizations end up with two or three AWS inputs, because a CloudWatch application stream, an audit trail and a central lake each serve different questions. Running them side by side is workable if each log type has one authoritative source and the streams are routed to separate indices or streams in Graylog. The risk is double ingestion, where the same event arrives through CloudTrail and a Security Lake export, which distorts counts and alerts.
- Assign one source per log type: Decide which path is authoritative for CloudTrail, VPC flow data and application logs.
- Route by stream: Keep each input's messages in its own stream so searches and alerts stay scoped.
- Watch for overlap: Compare event counts between paths during a test window.
- Document ownership: Record which team maintains each AWS component and its IAM permissions.
- Plan retention per source: Index sets for each path can carry different retention.
Before going live, confirm the inputs you intend to run are available in your Graylog edition and version.
Securing the IAM role, the stream and the Graylog input
The Kinesis/CloudWatch path touches four places where access can be too broad: the AWS role that reads the stream, the Kinesis stream itself, the Graylog input configuration and the Graylog backends that store what arrives. Tightening each one limits what enters Graylog and who can change how it gets there.
Least-privilege IAM for the reader role
Give the role that Graylog uses to read from Kinesis only the read actions and the specific stream or log groups it needs, rather than broad CloudWatch or Kinesis permissions. Confirm the exact action list in the Graylog input documentation for your version, then test the role with a narrow policy before widening it. Review the policy whenever a new log group is added.
Restricting who can change the input
The input decides what data enters Graylog, so changing its configuration is a security-relevant action. Limit input management to a small group of administrators and keep regular users in read-only roles. Record who edited the input and when, and confirm which role permissions Graylog Open provides for this in your version.
Encrypting data in transit to the node
Confirm how log data travels from Kinesis to the Graylog node and whether the path between AWS and your network is encrypted end to end. Terminate TLS where your design requires it and document any segment that is not encrypted. Verify the configuration against the Graylog documentation for your version rather than assuming defaults.
Scoping log groups to limit intake
Subscribing only the log groups that have a defined purpose keeps unneeded data, and any sensitive fields it carries, out of Graylog. For illustrative purposes, an application log group holding customer identifiers may need a separate review before it is streamed. Agree the list of included log groups with the data owners and document it alongside the input.
Protecting the Graylog backends
The Graylog server depends on MongoDB for configuration and metadata and on the Data Node or OpenSearch cluster for stored logs. Restrict network access to both, enable their own authentication and keep their credentials out of shared configuration. Confirm the supported versions for your Graylog release in the compatibility matrix before hardening them.
Broader controls and planning
The items above cover the ingestion path only. Audit trails, role design for the wider platform, sharing rules and encryption of stored data need a separate review. Use the Graylog security planning guidance to cover those areas before the input goes into production.
Why CloudWatch messages stop arriving in Graylog
This table lists the failures that most often leave CloudWatch or flow log messages missing from Graylog, with the likely cause, a check and a fix for each. Work through the rows in order, since an early failure such as a stream with no subscription data hides every problem further down the path. Confirm exact permission names and log type options against the Graylog AWS Kinesis/CloudWatch input documentation for your version.
| Symptom | Likely cause | Check | Fix |
|---|---|---|---|
| No messages at all | The Kinesis stream receives no subscription data, so there is nothing for Graylog to read | Confirm the CloudWatch Logs subscription filter points at the Kinesis stream and that the stream's incoming record metrics are above zero | Create or correct the subscription filter and confirm the log group is producing events in the same region |
| Input shows running but reads nothing | The IAM identity used by the input lacks read permissions on the Kinesis stream or the related log resources | Compare the policy attached to the input's credentials with the actions the Graylog documentation lists for the input | Attach the missing read permissions to the role or user, then restart the input and watch for new messages |
| Flow log messages missing or malformed | The input is set to the CloudWatch Logs type while the data is VPC flow logs, or the reverse | Look at the raw message in a test search and compare its format with the log type selected in the input | Select the log type that matches the source (CloudWatch Logs, CloudWatch Flow Logs or Kinesis raw logs) and recreate the input if the type cannot be changed |
| Input is stopped or in error | The input was never started, failed on startup, or the Graylog node that runs it is not the one you are checking | Open the input's state on the System, Inputs page and read the server log for the Graylog node that reports the error | Correct the reported configuration problem, such as a wrong region or stream name, and start the input on the intended node |
| Connection failures in logs | Firewall, security group or proxy rules block outbound access from the Graylog nodes to AWS service endpoints | Test HTTPS reachability to the Kinesis and CloudWatch endpoints for your region from each Graylog node, not only from your workstation | Open outbound HTTPS from the Graylog nodes to the required AWS endpoints or route them through your approved egress path |
| Messages arrive but do not appear in the expected place | The input routes messages to a default stream, or a stream rule does not match the fields the input produces | Search the input's messages without a stream filter and inspect the stream assignments shown on a sample message | Adjust the stream rules to match the fields present in the message, or set the destination stream in the input and in pipeline rules |
| Messages appear in a search but not on dashboards or alerts | Searches, dashboards or event definitions target a different stream, index set or time range | Confirm the index set and stream of the sample message, then compare them with the stream used by the dashboard widget or event definition | Point the widget or event definition at the correct stream and widen the time range to include the messages |
| Messages stop after a period of working | The stream or subscription was changed, the credentials were rotated, or an index set has reached its retention or rotation limit | Review recent changes to the AWS subscription, IAM keys and Graylog index rotation settings, and check the index set for recent deletions | Restore the subscription or credentials, document the change in the input's runbook, and adjust index retention to keep the intended window |
How the data path from CloudWatch to Graylog is built
The data path has four stages that each need their own capacity decision: CloudWatch Logs or VPC flow logs produce events, a Kinesis stream carries them, the Graylog AWS Kinesis/CloudWatch input reads them into the Graylog server, and processing and index storage hold them for search. The Graylog input reads messages that Kinesis must already stream to Graylog, so the stream and its delivery path are a prerequisite, not an afterthought. Plan the stream shard count, the input's configured read and retry behavior, and the Graylog node that will receive traffic before you turn on any log group. Confirm the current input settings and supported stream formats in the AWS Kinesis/CloudWatch input documentation for the version you run, because these names change between releases.
Backpressure shows up inside Graylog first. Messages that arrive faster than the processing pipelines and Data Nodes can index accumulate in the message journal on each Graylog node, and the journal is the buffer that keeps data from being lost during short slowdowns. The Graylog system architecture guidance advises placing the journal on SSDs and sizing it at 3 to 5 times daily ingest. Pair that with the storage rule of daily ingest times retention days times 120 percent. Ask your team to measure real CloudWatch volume for a representative week, including bursts from VPC flow logs, because flow logs can grow quickly in busy VPCs. Under the Kinesis path, a stalled Graylog node can cause data to wait upstream, so set alerts on journal fill and on the age of the oldest Kinesis record, and test a planned outage before production.
Questions teams ask before setting up CloudWatch ingestion in Graylog
What drives the cost of streaming CloudWatch logs into Graylog?
The cost has two parts that you should price separately: the AWS side and the Graylog side. On AWS, the charges come from the Kinesis stream, the CloudWatch subscription and filter setup, and the volume of log data forwarded, so the bill follows how much you send and how many log groups you include. On the Graylog side, the main driver is storage: every forwarded message is indexed and kept for your retention period. Ask your AWS account owner to model the Kinesis and CloudWatch charges for your actual log volume, and ask Graylog sizing questions against your daily ingest and retention. No fixed figure is given here because the result depends on your regions, volumes and retention policy.
How long does CloudWatch ingestion setup take?
The duration depends mainly on how many log groups, accounts and regions are in scope, and on how many of them need parsing rules. A single account with a few log groups and one stream is a much smaller job than a multi-account organization where each team owns its own log groups and approval steps. Other drivers include whether IAM roles already exist, whether the Graylog server is already running in production, and whether the network path from AWS to Graylog needs new firewall rules. Ask any partner for a written scope that lists accounts, regions, log groups and the number of input configurations before agreeing to a timeline.
What do we need from our AWS and Graylog owners?
The work needs input from two teams. From the AWS owners, you need permission to create or review Kinesis streams, CloudWatch subscription filters and IAM roles with the permissions the input requires, and an agreed list of log groups and VPC flow log sources. From the Graylog owners, you need access to the Graylog administration interface, a place to create the input and its index set, and a decision on which stream and retention policy the new data should use. Confirm who approves IAM changes and who can open the network paths between AWS and Graylog before the work starts.
Can our in-house team set up CloudWatch ingestion without a partner?
Yes, a team with experience in both AWS networking and Graylog administration can usually build it internally. The Graylog AWS Kinesis/CloudWatch input is documented, and the work is mostly configuration, IAM permissions and testing. The areas that tend to take the most time are the IAM policy scoping, verifying that messages arrive and parse correctly, and sizing the index sets for the volume you expect. Teams without recent AWS or Graylog experience often choose a fixed-scope engagement for the build and keep operations in-house. Either way, agree on who owns the input configuration after go-live.
Which Graylog inputs are for CloudWatch and when do we need Enterprise?
The Kinesis/CloudWatch input handles CloudWatch Logs, CloudWatch Flow Logs and raw Kinesis messages, and it works with Graylog Open as documented in the Graylog docs. Other AWS inputs, such as the CloudTrail input that reads from S3 and SQS, are a different option and have their own setup and licensing questions. Confirm the edition for each input against the Graylog documentation and pricing page before choosing one. Enterprise adds capabilities such as the data lake with archive restore, data tiering and 24/7 support, which matter when you need longer retention or vendor support around the pipeline.
What breaks most often in a CloudWatch to Kinesis pipeline?
The most common problems are a stream that is not delivering messages to Graylog, IAM policies that are either too broad or missing a required action, and log groups that were added to AWS after the subscription was set up. Parsing issues come next, because CloudWatch messages arrive in a structure that differs from the syslog-style sources many teams know. Check the stream first, then the IAM role, then the network path, and finally the message fields in Graylog. Keep a short test plan that sends known events through each log group before you rely on the data for alerts.
Should we use the Kinesis input or the S3 and CloudTrail path?
Use the Kinesis/CloudWatch input when your logs are already in CloudWatch Logs or VPC flow logs and you want them streamed to Graylog near real time. Use an S3-based path when your logs are written to a bucket, such as CloudTrail trails, and you want Graylog to read them from storage with SQS notifications. The right choice depends on where each log type is produced, how much latency you can accept and who controls the buckets. Many organizations use both, one for each source. Confirm the exact input behavior in the Graylog AWS documentation before you commit to a design.
How DeployOpen scopes and delivers CloudWatch ingestion for Graylog
DeployOpen delivers CloudWatch ingestion under a fixed-scope engagement with the scope, deliverables and timeline agreed in writing before any work starts. The written scope should name each log source in the first place: which CloudWatch log groups, which VPC flow logs, whether any raw Kinesis streams are involved, which AWS accounts and regions are covered, and which Graylog streams and index sets each source will land in. A quote follows from that scope, so the more precisely the source list is defined, the more accurately the work can be estimated. Ask the partner to show how each source maps to an input, an IAM principal and a stream before you sign.
The deliverables typically cover the Kinesis/CloudWatch input configuration in your self-hosted Graylog, the IAM design for the streaming path, test evidence showing that messages from each source arrive and parse as expected, and a runbook for your operations team. The IAM design should document the roles and trust relationships the streaming path uses, the least-privilege permissions granted to each, and how credentials are rotated or removed when a source is retired. Confirm the exact format of the test evidence and the runbook contents in the statement of work, because these are the items your auditors and on-call engineers will ask for later. Retainer support for upgrades, input changes and periodic checks is available as a separate, recurring agreement that you can scope once the initial build is accepted.
All work happens inside your own AWS accounts and your own self-hosted Graylog environment. DeployOpen does not take custody of your log data or hold standing access to your production accounts beyond what the written scope grants for the engagement period. Confirm with your partner how access is granted, recorded and revoked, and who in your organization approves the IAM changes. DeployOpen prepares systems, configurations and evidence that can support your compliance assessments, but it does not issue audit reports, attestations or certifications. Those come from independent auditors and certification bodies, so plan for them to review the evidence you hold.
