Zammad and Checkmk alert tickets

The core behavior to design around is simple: a problem notification from Checkmk creates a Zammad ticket, later notifications about the same problem update that ticket, and a recovery notification closes it. Checkmk can send real-time status alerts to Zammad by email or REST API, and several alerts can update a single ticket. The point of this section is to make sure your team understands that lifecycle before it configures anything, because the deduplication behavior is what keeps infrastructure incidents from flooding the same queue that handles user requests.

Checkmk supports two transport options, and the choice shapes your operating model. Email is the simpler path to start with: Checkmk sends a message to a Zammad inbound address, and Zammad parses it into a ticket or an article. Authentication is largely handled by the mail infrastructure, and parsing depends on the message format Checkmk produces. Failure visibility is weaker, because a bounced or silently dropped message may only show up as a missing ticket. The REST API path uses a Zammad API token and an explicit HTTP call, which gives you clearer authentication control, stronger response codes for failures and a more deliberate payload structure. It also means the token becomes a secret you must store, rotate and scope to the minimum permissions needed.

Whichever transport you choose, the mechanism for linking a new alert to an existing ticket depends on the identifiers the notifications carry. Zammad needs a stable reference, such as a host and service combination or a custom field value, that appears identically in the problem notification and in every later update and recovery. If the identifier changes between notifications, or is not present in the body that Zammad reads, the alerts will create new tickets instead of updating the original. Do not assume the mapping works from general descriptions. Confirm the exact fields and their behavior in the Checkmk and Zammad documentation, then check them in a test instance with a deliberately triggered problem, a repeat notification and a recovery.

Plan the test so it covers the failure cases, not only the happy path. Trigger a problem, let it repeat, recover it, and then raise it again to confirm that a new incident opens a new ticket rather than reopening an old one that was closed. Check how Zammad handles notifications that arrive out of order, and confirm what happens when the mail server or API endpoint is unavailable. Record which team member owns the test results, because those results determine whether the transport and identifier design is ready for production alerting.

Decide early who owns the Checkmk notification rules, who maintains the Zammad inbound channel or API token, and who responds to tickets created by monitoring. A ticket that updates automatically is only useful if an agent knows whether the incident is still active. Agree on the priority and group assigned to monitoring tickets, and document how an engineer should add context to an open alert ticket without breaking the link used for later updates.

Alert tickets should not be created under a real person's customer identity. If an alert arrives as a ticket owned by an employee's customer record, the ticket inherits that person's name, email history and notifications, and the next time that person writes to support, their conversation can be merged with, or buried under, machine-generated status changes. The safer pattern is a dedicated system or service customer account, created for Checkmk only, with a name that makes its origin obvious to agents, such as a monitoring service identity. Confirm in your Zammad instance that this account cannot log in interactively, that it is excluded from customer satisfaction surveys, and that its email address is a mailbox you control, because Checkmk's email method and any reply behavior depend on it. Choose the receiving group before you configure the alert path. Most teams route alert tickets to a dedicated infrastructure or operations group rather than the general support group, so that agents who are on call can filter for it and so that user requests are not diluted by monitoring noise. Decide who is allowed to reassign alert tickets, who may close them manually, and whether the group needs a shared inbox view. Group membership and roles are set in Zammad's administration, and the decision should be documented next to the Checkmk notification rules so the two stay aligned when staff change. Priority rules determine whether an infrastructure incident is seen before a password reset. Define how Checkmk alert states map to Zammad priorities, for example a critical host or service state creating a high priority ticket, and a recovery message only updating the existing ticket. Then decide whether alert tickets sort above or below user requests in the default views. Test these rules with a non-production Checkmk host, because the mapping is only useful if the on-call agent sees the ticket in the first screen they open. Agents see alert tickets in the same queue as user requests unless you separate them with views, groups or overviews. That is usually the right trade-off for small teams, as long as the views filter by group, state and priority and the alert ticket titles include the host and service name. Test how a shared queue behaves under volume: a flapping service can generate many updates, so confirm whether each update appears as a new article, whether notifications fire for every article, and how your team wants to handle duplicates. If several Checkmk sites or environments send alerts into one Zammad instance, the identity and routing decisions multiply. Give each site or environment its own identifier in the ticket title or a tag, decide whether production and test alerts may share the same receiving group, and confirm that alerts from one site cannot update tickets created by another. Ask your integration owner to document the matching rules, since the same host name in two sites can otherwise cause one site's recovery message to close another site's incident. Plan this before onboarding the second site rather than after the first duplicate appears.

Before any Checkmk notification reaches the production Zammad queue, run the full alert lifecycle against a non-production host that you can break on purpose. The goal is to prove that the integration creates one ticket per incident, updates that ticket when the problem repeats, closes it when recovery arrives, and makes failures visible to the people who must act on them. Start by choosing a test host in a non-production Checkmk site and a non-production Zammad instance with the same channel (email or REST API) that production will use. Testing the wrong channel proves very little, because the email path and the REST path differ in how they carry host, service and state information into Zammad.

Run three basic scenarios in order. First, force a problem state on a test service and confirm that exactly one ticket appears in the intended group with the host and service identified in the subject or body. Second, let the problem repeat or change state, for example from warning to critical, and confirm that the existing ticket receives an update rather than a second ticket. Third, return the service to OK and confirm that the recovery notification moves the same ticket to the closed state configured for your integration. Record the ticket number at each step. A duplicate at any step means the matching rule between Checkmk notifications and Zammad tickets needs correction before go-live, not after.

Then test the harder case: recovery arrives after an agent has already changed the ticket. Set the ticket to a state such as pending or in progress, or reassign it, and then send the recovery notification. Confirm whether Zammad closes the ticket, leaves the agent's state in place, or creates a new ticket. Your support team needs a documented answer to this, because the wrong outcome either hides an open incident or buries a resolved alert in a fresh queue item. Also test a new problem that arrives after a ticket was closed by recovery, to confirm that a new incident opens a new ticket as intended.

Load testing comes next. Send a burst of alerts from one test host, sized to match the largest realistic flapping pattern you expect, and then repeat the burst across several test hosts at the same time. Watch for lost notifications, delayed updates, duplicate tickets and changes to ticket ordering. Confirm the mail queue or API rate limits on both sides, and confirm the Zammad background job behavior under the burst. Ask your Zammad partner and your Checkmk administrator to agree the burst sizes in writing, since no fixed capacity figure applies to every environment. Finally, deliberately break delivery, for example by pointing a notification at an unreachable mail server or an invalid API token, and confirm that operators see the failure in Checkmk's notification log, in a monitored mailbox, or in an alert on the integration itself.

Keep the evidence in a change record that an auditor or the next engineer can follow. Store the Checkmk version, Zammad version, notification rule definitions, the Zammad channel or API user configuration (without secrets), the test host and service names, and the timestamp and ticket number for each scenario. Include screenshots or exports of ticket history, the Checkmk notification log for the failed delivery test, and the sign-off from the operator who confirmed visibility. Record the exact acceptance criteria you applied, such as the number of tickets expected per incident, and state any deviation with its owner and due date. This record lets the team repeat the test after each Checkmk or Zammad upgrade, which is when integrations most often break silently.