Collecting GitLab logs and diagnostics for support

Collecting GitLab logs and diagnostics for support before you escalate

When a self-managed GitLab incident stalls, the usual cause is missing evidence, not a missing answer. Before you escalate to GitLab or to a partner, collect the exact GitLab version, the timestamps of the failure and the matching logs from each component that touched the request. Support engineers can rarely reproduce a problem from a description alone, so a case that arrives with a dated, versioned, component-level record moves much faster than one that says "GitLab is slow since Tuesday." The fastest first capture is gitlab-ctl tail, which follows the logs of every bundled service on a Linux package installation, including Nginx, Puma, Sidekiq, Gitaly and PostgreSQL. Start it before you reproduce the fault, then save the output for the window in which the failure occurred. Record the GitLab version with sudo gitlab-rake gitlab:env:info and the status of services with sudo gitlab-ctl status, because a fault during an upgrade, a failed job or an authentication error often traces to a specific release or a component that restarted. Confirm the installation method with your own team first, since Helm, Operator and Docker deployments keep logs in different places.

This process is worth starting early in four situations: after an upgrade that crosses one of the required upgrade stops, when response times degrade under normal load, when CI jobs fail in a pattern, and when users or runners hit authentication faults such as failed SAML or LDAP sign-ins. Ask your team to note who changed what and when, including gitlab.rb edits and runner changes, because support will ask. DeployOpen plans and prepares this evidence as part of implementation, support and runbook work. It does not issue support on GitLab's behalf, and it does not replace the support entitlement your tier includes, which you should confirm with GitLab before you need it.

Main log locations by component

The Rails application that serves the web UI and REST and GraphQL APIs writes most of its request-level activity to logs under the Linux package log directory. Open these first when users see HTTP 500 errors, slow pages, failed logins or API calls that return unexpected responses. Correlate timestamps and request IDs across files before deciding the fault sits in the application.

Production log: /var/log/gitlab/gitlab-rails/production_json.log holds structured request entries and is the usual starting point for application errors.
Application log: /var/log/gitlab/gitlab-rails/application.log records application-level events and warnings.
Error log: /var/log/gitlab/gitlab-rails/production.log carries unstructured Rails output and stack traces that the JSON file may not show.
API traffic: /var/log/gitlab/gitlab-rails/api_json.log helps isolate automation or CI clients that repeatedly call the API.
Service status: run gitlab-ctl status to confirm the Puma and related services are running before reading logs.

Confirm each file name and path against the documentation for your installed version, since names have changed across releases.

Sidekiq runs asynchronous work such as webhooks, mail, repository housekeeping, pipeline processing and email notifications. Look here when events arrive late or not at all, when pipelines stay pending, or when scheduled cleanup seems to stop. Failed jobs are often visible here before any user reports a problem.

Sidekiq log: /var/log/gitlab/gitlab-rails/sidekiq.log records job starts, completions and failures with job class names.
Sidekiq JSON log: /var/log/gitlab/gitlab-rails/sidekiq_json.log gives the same events in structured form for filtering by job class or error.
Queue symptoms: growing queues or stalled pipelines usually point here, even when the web tier looks healthy.
Repeated retries: the same job class failing over and over often indicates a configuration, credential or network problem with an external target.
Service check: gitlab-ctl tail sidekiq streams live output while you reproduce the problem.

Before escalating, record the job class, the error text and whether the failures started after a configuration change or upgrade.

Gitaly handles Git operations such as clone, fetch and push against repository storage. Check its logs when pushes or clones time out, when repository pages show errors while the rest of the UI works, or when storage paths behave inconsistently after a restore or migration. Problems on separate Gitaly nodes must be read on those nodes, not only on the main server.

Gitaly log: /var/log/gitlab/gitaly/current is the main log for storage operations and gRPC errors.
Gitaly JSON log: /var/log/gitlab/gitaly/log.json.1 (rotated files may use different suffixes) holds structured entries for filtering.
Git HTTP and SSH traffic: if clones fail before reaching Gitaly, check the Workhorse and NGINX tabs as well.
Storage location: confirm the configured repository storage path in gitlab.rb so you know which disk to inspect for errors.
Separate nodes: in multi-node layouts, collect logs from each Gitaly host, since a single node may show only a symptom.

Note the repository path, the operation that failed and the time window, because these details shorten the investigation.

The database and cache tiers store the application's state and session data. Read their logs when the application reports database timeouts, lock waits, connection refusals, or cache failures that appear intermittent. Confirm first whether the problem sits in the database itself or in connections from the application.

PostgreSQL log: /var/log/gitlab/postgresql/current records server errors, slow statements and connection problems.
Redis log: /var/log/gitlab/redis/current shows connection handling, memory pressure and persistence messages.
Application view: database errors in the Rails logs often mirror what PostgreSQL logs, so match timestamps between tiers.
External services: if you use a managed or separate database, the logs live with that service and must be requested from its provider or administrator.
Confirm the topology: check gitlab.rb to see whether these services run on the instance or elsewhere.

Avoid sharing raw database output without reviewing it, since statements can include sensitive data.

NGINX receives client connections and forwards them to GitLab Workhorse, which proxies requests to the application and handles large Git and upload traffic. Look here when you see HTTP 502 or 504 responses, certificate warnings, upload failures, or when the same request succeeds from inside the network but fails through the proxy. These logs show the request as the client experienced it.

NGINX access log: /var/log/gitlab/nginx/gitlab_access.log records every request with status codes and upstream timing.
NGINX error log: /var/log/gitlab/nginx/gitlab_error.log shows connection failures to upstreams and TLS problems.
Workhorse log: /var/log/gitlab/gitlab-workhorse/current captures proxy errors and long-running requests.
Load balancers: if a load balancer sits in front, collect its logs too, since they may show a different status than GitLab.
Timing mismatch: compare upstream time values with application timings to see where delay accumulates.

Record the exact status code and request path, because they separate proxy failures from application errors.

Audit events record who changed what, such as permission updates, sign-ins, project visibility changes and token creation. Security and compliance teams use these records for investigations and evidence, while production logs help operators answer what the system did. Treat these files as records that need protection, retention rules and controlled access, because they can contain personal data and identifiers.

Audit JSON log: /var/log/gitlab/gitlab-rails/audit_json.log holds structured audit events for administrative and user actions.
Authentication log: /var/log/gitlab/gitlab-rails/auth.log records sign-in and authentication attempts, useful when access questions arise.
Application security log: check the Rails directory for security-related files that match your configured features.
Retention: confirm how long your Linux package installation keeps rotated logs and whether you must export them earlier.
Forwarding: many teams ship these files to a central platform, which is covered in the sibling topic on SIEM forwarding.

Ask your security team which events they need before you collect, so you do not gather more personal data than the case requires.

Capturing gitlab-ctl tail output during a live failure

This sequence produces a timestamped, redacted capture of GitLab component logs covering a reproduced failure, with a recorded time window and user action that a support engineer can match to the affected component.

Confirm instance state with gitlab-ctl status
Run sudo gitlab-ctl status on the affected node before reproducing anything. The administrator records which services report as running, which report as down or restarting, and the host name and GitLab version from sudo gitlab-ctl version in the case notes. This establishes a baseline so the support engineer can tell a pre-existing fault from one the reproduction causes. Confirm the output is saved with the time it was taken.
Prepare a timestamped output file
Create a dated file name in a directory with enough free space, for example by writing to a path such as /var/tmp/gitlab-tail-<host>-<UTC timestamp>.log. Check free disk space on that filesystem first, because a busy instance can write quickly during a failure. Confirm the file exists and is writable by the account that will run the capture, and note the path in the case record.
Start the tail capture and redirect output
Run sudo gitlab-ctl tail in a terminal multiplexer session such as tmux or screen, so the capture survives a dropped SSH connection, and redirect output with sudo gitlab-ctl tail >> <file> 2>&1. Keeping a second terminal open for interactive checks is useful. Confirm the capture started by seeing the first lines appear in the file, and note the start time in UTC.
Reproduce the failure and record the window
Perform the exact user action that triggers the problem, such as a push, a pipeline start or a page load, and note the start time, end time, user account, project and the request identifier if the user can provide one. While the reproduction runs, check that the file size is increasing, for example with ls -l run twice. A file that does not grow usually means the capture is attached to the wrong node or the component is silent, which should be investigated before continuing.
Stop the capture cleanly
Once the failure has appeared and been observed, stop the tail with Ctrl+C in the session that runs it, then confirm the process has exited. Run gitlab-ctl status again to capture the post-failure state, and record the end time. A clean stop ensures the final lines are flushed to the file before the file is copied or compressed.
Check the capture for sensitive values
Search the file for tokens, passwords, private keys, personal data and full URLs containing credentials before it leaves the instance, for example by grepping for common secret patterns and for user email addresses. Replace matches with placeholders in a redacted copy, and keep the original under the organization's retention rules. The check is complete when the redacted copy has been reviewed and the list of replacements is recorded with the case.
Copy logs to a separate host when the node is unstable
If the affected node is unresponsive, out of disk space or failing over, copy the capture and the surrounding logs to a separate administration host or approved storage location using the transfer method your security policy allows. Verify the copy by comparing file sizes and checksums with the originals. Confirm the copy is complete before any cleanup on the broken node, because the source may be lost during recovery.

What to collect before opening a GitLab support case

A support engineer can only act on what arrives with the case. The six items below cover the version and upgrade history, configuration, health state, logs, environment and reproduction, so the first reply is a diagnosis rather than a request for missing data. Agree in writing beforehand what DeployOpen will support and who handles escalation to GitLab.

Version, edition and upgrade history

Record the exact GitLab version and edition (Free, Premium or Ultimate), the install method, and every upgrade made in the last few months with dates. GitLab requires upgrades to pass through required upgrade stops, so a skipped stop is a common root cause that support will ask about first. Confirm the path against the upgrade paths documentation before you open the case, and note any rollback or restore performed during the incident window.

gitlab.rb and gitlab-secrets handling

Share a copy of /etc/gitlab/gitlab.rb with secret values redacted, and list any non-default settings you changed. Never send /etc/gitlab/gitlab-secrets.json or raw database credentials, since they contain keys that protect encrypted data. Keep both files backed up in your own secure store, and let support describe what it needs before you copy anything out of the environment.

gitlab-ctl status and health output

Capture the output of gitlab-ctl status and gitlab-ctl check-config from the affected node, along with the time you ran them. These show which bundled services are running, restarting or down, and whether configuration is valid. Run them during the incident and again after recovery so support can compare the two states rather than relying on a single snapshot.

Component logs for the incident window

Collect logs from the components involved, not the whole directory. Typical sources include Rails and Workhorse logs for web requests, Sidekiq for background jobs, Gitaly for repository access, and PostgreSQL and Redis for data-layer faults. Export a fixed window that includes a few minutes before the first symptom, and confirm log locations for your install method before collecting.

Environment topology details

Describe the operating system and version, the number of nodes, where PostgreSQL and Redis run (bundled, external or managed), storage type for repositories and object storage, and which reference architecture you sized against. Support handles a single-node Linux package install very differently from a multi-node cluster, so an accurate topology saves a round trip. Note any proxy, load balancer or network path between users and GitLab.

Reproduction steps with timestamps

Write the smallest set of steps that reproduces the problem, with the user role, project type, and exact action taken. Add timestamps in one time zone, ideally UTC, and include the request ID or job ID from the UI or logs where one appears. A reproduction that names a time and an identifier lets support match the case to log lines directly, which is usually the fastest path to a cause.

Which capture method fits which incident type

This table compares four ways to capture GitLab logs for a support case: live tailing, reading files after the fact, the bundled support tooling and central log forwarding. Read each row against the incident you are handling, because the right method depends on whether the problem is still happening and whether the evidence has already rotated out of the node. Confirm command names and log paths against your installed GitLab version before you rely on them.

Method	When it fits	What it misses	Effort on the team
gitlab-ctl tail (live capture)	A fault you can reproduce now, such as a failing push, webhook or runner registration during a test	Anything that happened before you started capturing, and components you did not think to tail	Low to set up, but someone must be at the terminal while the problem occurs
Reading individual log files after the fact	Incidents already over, where you know the component, such as Puma, Sidekiq, Gitaly or nginx	Logs rotated out or overwritten on the node, and correlation across components unless timestamps line up	Low to start, higher when you must piece together several files by hand
Built-in GitLab support tooling	Requests from GitLab Support or a partner that ask for a standard diagnostic bundle of configuration and logs	Data outside the bundle, and anything you must redact before sending (confirm what the tool includes in your version)	Moderate, mainly reviewing the output for secrets and sensitive project data before it leaves your network
Central log forwarding to a SIEM or log platform	Recurring or intermittent faults, and incidents that need history across days or several nodes	Events that never reached the forwarder, and components not configured to ship (confirm which log types your setup forwards)	Highest, since shipping, parsing and retention must be designed and maintained before an incident
Support case with a known reproduction window	Problems that can be recreated on demand, where you can record a timed capture plus a bundle	Behavior that depends on user load or data you cannot replay in a test	Moderate, since you coordinate timing with the people who can reproduce the fault

Handling secrets and personal data in diagnostics

Before you attach anything to a support case, assume that the files you collect contain credentials and personal data. Log lines, gitlab.rb and the output of a gitlab-ctl tail capture can carry access tokens, webhook URLs, internal hostnames, user email addresses, IP addresses and repository paths that reveal project names or customer work. Some of these items are harmless alone but identify people or systems when combined. Treat every bundle as sensitive until someone has reviewed it, and keep the raw collection in a restricted location inside your own environment.

Some files should never leave your infrastructure, even in redacted form. Do not share `gitlab-secrets.json`, which holds the keys GitLab uses to encrypt database fields and tokens, and do not share database credentials, Redis passwords, object storage keys, SSH host or deploy private keys, or the contents of any certificate key file. Supporting staff rarely need these values to diagnose a fault. If a ticket appears to require one, ask for the specific question to be answered in a way that does not expose the value, or arrange a controlled session in which your engineer runs the check and reports only the result. Also check for secrets embedded in gitlab.rb, environment files and reverse proxy configuration before you copy them.

Agree redaction rules in writing with GitLab or your implementation partner before the first file is sent. The agreement should name what is replaced (for example, a placeholder for tokens, hostnames and email addresses), whether repository and group paths are kept, how user identifiers are pseudonymized, and who performs the review. Ask whether the support team can work from a sample you prepare, and confirm where bundles are stored and when they are deleted. The GitLab hardening recommendations cover the configuration areas that hold these secrets, and they are a useful checklist for what your redaction step should look for. Confirm current guidance with the vendor, since the hardening documentation is described as a work in progress.

Security risks in diagnostic collection

A support bundle is a snapshot of a production system, so collecting it creates its own exposure. The cards below cover the risks that most often turn a routine support case into a security incident, and one practical control for each.

Over-sharing secrets in captures

Log output, configuration dumps and job traces can contain database passwords, tokens, OAuth client secrets and the contents of gitlab-secrets.json. Before anything leaves your network, search the bundle for credential patterns and remove or redact matches. Ask GitLab support whether they need configuration files at all, because many cases only require log excerpts with timestamps and the failing request ID. Rotate any secret that is found in a bundle sent outside your control.

Capture files left on shared hosts

Tail output and archives written to /tmp or a shared admin home directory can be read by other local users and picked up by backup agents. Create a dedicated, root-owned working directory with restrictive permissions before capture starts, and set a short cleanup task that runs after the case closes. Confirm which backup jobs and log shippers read that path so an investigation file does not get copied into long-term storage.

Logs kept longer than policy allows

Once a diagnostic copy exists, it can outlive the retention period set for the live logs, which may include usernames, IP addresses and project names. Record who created each bundle, why, and when it must be deleted, and apply the same retention rule to support copies as to production logs. Check the retention settings in your own log platform, since GitLab's defaults are not a substitute for your organization's policy.

Unauthenticated transfer of diagnostics

Emailing a bundle, posting it to a public file share or uploading it through an unverified link exposes the data in transit and lets anyone who obtains the link read it. Send bundles only through the channel your support agreement names, encrypt the archive before transfer, and verify the recipient's identity through a known contact. Keep the decryption key out of the same message as the file so a single leak does not expose both.

Capture changing the system under investigation

Raising log levels to debug, enabling verbose tracing or restarting components to reset state can alter timing, fill disks and hide the original fault. Limit changes to what the support engineer requests, write each change down with its time and the previous value, and revert it as soon as the capture ends. Check free disk space on the log volume first, since verbose output can fill it within the capture window and stop the instance.

Questions teams ask before escalating a GitLab incident

Can our team collect GitLab logs and diagnostics for support ourselves?

Yes. Most teams can gather the core evidence themselves by running gitlab-ctl status, capturing gitlab-ctl tail output during the failure, and copying the relevant files from the log directory under /var/log/gitlab. The harder part is knowing which component produced the symptom and capturing the window before rotation removes it. Teams without a Linux administrator who knows the Omnibus layout usually get more value from a short, scoped review of what they have collected. Confirm which installation method you use first, because a Helm chart or Operator deployment reads logs from container stdout rather than files on a host.

How much effort does a support log collection take?

Effort depends on how many components are involved and how quickly the incident can be reproduced. A single-node Linux package install with one clear symptom is a small job: capture status, the relevant component logs, the gitlab.rb configuration with secrets removed, and the GitLab version. A multi-node reference architecture with separate Gitaly, PostgreSQL, Redis and Sidekiq hosts multiplies the collection steps, and each host needs its own capture with synchronized timestamps. Ask any partner to state the number of hosts, the component list and the review steps in a written scope before work starts.

What makes log collection harder as the environment grows?

Three factors drive most of the extra work. The first is node count, since every host adds log sources and clock alignment problems. The second is incident frequency: a team that escalates often needs a repeatable collection procedure, not a one-off script. The third is secret handling rules. Configuration files such as gitlab.rb and some logs can contain tokens, hostnames or personal data, so your security policy may require redaction, an approved transfer channel or a review before anything leaves your network. In air-gapped environments, transfer also needs a documented route, as described in the GitLab offline environments documentation.

Does a partner collect diagnostics differently from our own team?

The core evidence is the same, but a partner usually applies a fixed checklist, documents each step, and reviews the data against known failure patterns before recommending an escalation. A partner also works from the written scope, so the collection is reproducible and the evidence is organized for GitLab support or for your own change records. The partner does not see more of your system than your access allows. Ask how they handle redaction, where collected data is stored, how long it is retained and who on your side approves each transfer.

What do we need to provide before a partner can start?

A partner needs a named owner on your side who can approve access and decide on the escalation path. They also need access to the affected hosts or cluster, a way to reach the GitLab administrator account for version and configuration checks, and a change window if collection requires restarting services or enabling debug logging. Agree on what happens if the incident is ongoing, since live capture may need to run while your team keeps users informed. Record these requirements in the written scope so that access does not become the bottleneck during an incident.

How long does preparation take depending on our environment?

Preparation time is set mainly by access approvals, the number of hosts and the state of existing logging. A team that already forwards logs to a central platform and keeps a documented collection runbook can prepare quickly. A team with no runbook, several hosts and approval steps for every transfer will spend most of the time on access and sign-off rather than technical collection. No fixed duration applies to all environments. Ask for a preparation plan with milestones tied to your approval steps, not a generic estimate.

What happens if the relevant logs have already rotated?

Rotated logs are often recoverable from the compressed archives kept in the same log directory, but only if rotation has not removed them. Check the retention settings on the host and any central log store before assuming the data is gone. If the evidence is lost, the practical step is to record the symptom, timestamps, GitLab version and the output of gitlab-ctl status now, then enable more detailed capture before the next occurrence. Central forwarding and a documented retention period reduce this risk considerably, so treat it as a planning item for the next change window.

How DeployOpen scopes and supports GitLab diagnostics

DeployOpen approaches support log collection as part of running self-managed GitLab inside your own infrastructure, so evidence is gathered in a controlled, repeatable way before anything leaves your network. We scope the work in writing, build the collection steps around your installation method and log shipping setup, and hand over a documented procedure your team can run during an incident.

How we scope

We confirm your GitLab install method, edition and node layout before agreeing what diagnostics the written scope covers.
We agree which components are in scope, such as Puma, Sidekiq, Gitaly, PostgreSQL and Redis, and where their logs live on your hosts.
We record data handling rules, including what may be shared with GitLab or a partner and how secrets and personal data are excluded.

How we build

We build repeatable gitlab-ctl tail capture and log bundling steps tailored to your Linux package or Helm deployment.
We test the collection procedure on your non-production instance so it runs without changing production configuration during an incident.
We set file permissions and retention on collected bundles so diagnostics are stored and transferred under your access controls.

How we hand over

We deliver a written runbook that lists each log location, the collection command, and the information to include in a support case.
We walk your operations team through the procedure and record who is authorized to run it and who opens the escalation.
We document the version, edition and configuration details your team should attach to every case before escalating.

How we support

Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
Ongoing support, upgrades and re-testing of the collection procedure after GitLab upgrades are available as a retainer.
Quotes follow from the written scope, so your team can ask for each collection step, handover item and retainer option to be priced separately.

Collecting GitLab logs and diagnostics for support