Keycloak monitoring and metrics

Monitoring is only trustworthy once you have watched it fail on purpose. Before go-live, run a short set of acceptance tests that prove the metrics endpoint is exposed in the right place, that Prometheus actually collects from every node, that health checks remove a bad node from rotation, and that the alerts you rely on fire under realistic stress in a lower environment. Each test below has an expected result you should write down before you run it, so the outcome is a pass or fail rather than a judgment call.

Start with exposure. Keycloak serves Prometheus-format output at /metrics on its management interface when it is started with metrics enabled. Confirm two things from outside the cluster. First, a request to the management interface returns metrics text. Second, the same path requested through the public hostname that serves sign-in is refused. Run both requests from the network where your Prometheus server sits and from an internet-facing vantage point, because a reverse proxy rule that looks correct on paper can still forward the path. Also confirm that the admin console and Administration REST API are on a separate hostname or context path, as the production guide recommends, since that separation is what keeps the metrics path off public routes. Ask your platform team to show the proxy configuration that enforces it.

Next, prove that collection works across restarts. Restart one node and watch its target in the Prometheus targets view. It should move to down while the process is unavailable and return to up once the node is ready and serving metrics again. Check that the series for that instance resume rather than appearing under a new label, since a changed instance name will break dashboards and alert rules that filter on it. If you run Keycloak on Kubernetes, repeat the test with a pod deletion so you also see how service discovery handles a replacement pod. Record how long the target stays down, because that gap is the blind spot your alerting must cover.

Then test the cluster behavior. Stop one node in a multi-node deployment and confirm two separate reactions. The load balancer or orchestrator health check should stop sending new requests to that node, and existing users on other nodes should keep signing in without errors. The monitoring side should also react: the target goes down, and the alert you defined for lost instances fires within the window you agreed. A health check that removes the node but no alert is a silent degradation, so treat a missing alert as a failed test. Confirm the health endpoints are wired to the correct probes, liveness for restarting a stuck process and readiness for taking traffic, and that a node still starting up is not marked ready early.

Finally, exercise the signals that matter for an identity service. In a test realm with a disposable user, generate a burst of failed password attempts and confirm the login failure counter moves and that your dashboard shows the spike. Then, in a lower environment only, add artificial latency to the database or create cache pressure by reducing cache size or running a load test, and confirm the latency and cache alerts trigger at the thresholds you set. Finally, stop the monitoring stack itself and confirm something still tells a person: a dead-man switch, an external probe, or an alert manager heartbeat should page the on-call team. Keep a short acceptance checklist: metrics only on the management interface, public path refused, targets recover after restart, failover removes a node and alerts, login failure and latency alerts fire in test, and loss of monitoring raises an alert.

Once Keycloak is in production, monitoring only helps if someone has decided which signals page a person and which ones go to a dashboard. Start with sign-in outcomes. Alert on the rate of failed logins against a baseline you measured during the first weeks of live traffic, not on a fixed number copied from another system, because a normal Monday morning and a password-spraying attempt can look similar in raw counts. Keep the spike alert separate from the steady-state alert so that a sudden jump is not hidden by a high but stable background rate. Note that in the 26.8 release, login failures are persisted to the database by default. That adds write load and table growth, so confirm your database capacity and retention plan before you upgrade, and watch database write latency alongside the failure counter.

Next, decide which request latency percentiles matter to your users and track them explicitly. Many teams watch the median and the 99th percentile for token and login endpoints, but the right choice depends on your sign-in volume and your service-level commitments, so write the chosen percentiles into the monitoring standard. Pair latency with cache health. Keycloak's embedded Infinispan caches hold sessions, realm data and other state, and rising cache misses or evictions often explain slow sign-ins before anyone sees an error. Database connectivity and node availability complete the core set: alert on failed database connections, on nodes dropping out of the cluster, and on health check failures from your load balancer or orchestrator. Enable the metrics endpoint on the management interface and confirm which histograms your version exposes before you build dashboards around them.

Monitoring changes with every Keycloak upgrade. Metric names, label sets and default values can shift between releases, so a dashboard that worked on one version can show empty panels or silent alerts on the next. Before each upgrade, read the upgrade notes for metric and observability changes, then compare the scraped metric names in a staging environment against the production set. Treat that comparison as a required step in the upgrade test cycle. The Keycloak project plans four minor releases a year, so this check will recur several times annually, and it should be scheduled as routine work rather than handled as an emergency after a dashboard breaks.

Document the monitoring setup well enough that a new engineer could run it without asking the original builder. The record should cover the scrape configuration, including targets, intervals and the management interface address, and the owner of each dashboard. For each alert, record the threshold, the reason for choosing it, the expected response and a runbook that explains the first three checks to make. Runbooks should name the Keycloak logs, the database and cache views and the escalation contact. Store this documentation with the realm and client inventory so changes to either are reviewed together.

Hold a review after each minor release, and again after any change to the realm count, client count or sign-in volume. In the review, confirm that every alert still fires in staging, that thresholds still match observed traffic, that dashboard owners are current and that runbooks match the running version. Record the outcome as evidence for your operations and compliance files. Ask your support provider, or your own team, to confirm which metrics changed in the release notes and whether any default now writes new data. These reviews are the most reliable way to keep monitoring trustworthy as Keycloak evolves.