Migrate Grafana API keys to service accounts without breaking scripts and CI
To move scripts and CI jobs off legacy API keys, you inventory every caller, decide the organization each one needs, assign a role that grants no more than that caller uses, and then swap credentials one caller at a time with the old key still active until the new token is confirmed. That is the work: an inventory of callers, a per-organization scope decision, role assignment and a rotation plan. Start with the inventory, because a key that nobody remembers creating is usually the one that breaks a nightly job after cutover. For each caller, record the Grafana URL, the organization it writes to or reads from, the endpoints it calls, who owns the job and how it stores the credential today, whether in a CI variable, a vault or a file on a host.
Scope and role decisions follow from that list. A legacy API key belongs to one organization, so a caller that touches several organizations needs one service account or token per organization, and you should not give one account broad rights because it is convenient. Map each caller to the least role that covers its calls, and treat the None role as the default for callers that only need to reach resources they were explicitly granted. Then set a token expiry for each token within the limits your Grafana version enforces, and record the owner and renewal date in your inventory. Effort depends on the number of callers and organizations, so ask any partner for a quote built from that list. Confirm the exact service account, token and expiry behavior for your Grafana version in your own test environment before you change production.
Rotate without downtime by running both credentials in parallel. Create the service account token, update one caller in a non-production pipeline, confirm the job succeeds with the new token and that the audit trail shows the service account rather than the old key, then roll the change to production callers in small groups. Keep the legacy key active until every caller in your inventory has been seen using its replacement, then delete it and record the removal. Plan for the failures this sequence exposes: callers that embed credentials in dashboards or provisioning files, scripts that read GF_SECURITY_ADMIN_PASSWORD or similar settings instead of tokens, and token expiry dates that fall during a release freeze. Set calendar reminders for each expiry and test the rotation procedure once before you need it under pressure.
What maps cleanly from API keys to service accounts and what does not
This table compares how a legacy organization API key and a service account token behave across scope, roles, expiry, ownership and migration. Read each row as a checklist: the first column tells you what to inspect in your current scripts and CI jobs, and the right-hand columns show what changes. Confirm expiry limits and role mappings against the Grafana version you run before you rewrite any automation.
| Item | Legacy API key | Service account and token |
|---|---|---|
| Scope | Bound to one organization at creation | Bound to one organization; add a token per organization that the job needs |
| Role assignment | Carries a single organization role such as Viewer, Editor or Admin | Holds an organization role, and the None role can be used for least privilege |
| Least privilege | Cannot be set below the role chosen at creation without recreating the key | Set the role to None and grant access through folder or dashboard permissions where possible |
| Expiry | Expiry is set when the key is created and cannot be extended | Tokens have their own expiry; confirm the maximum allowed in your version |
| Naming and ownership | Name is free text and the key is not tied to an identity | Account name identifies the job or owner, and tokens carry their own names for rotation |
| Audit trail | Actions are hard to attribute to a specific script or team | Actions are attributed to the service account, which helps review |
| Automatic carry-over | Not applicable, since keys are replaced rather than converted | Keys do not convert into tokens; each script must be given a new token and updated |
| Rotation without downtime | Usually requires replacing the key everywhere at once | Create a second token, update consumers, then revoke the old token |
| Version check | Behavior is stable but deprecated in newer releases | Confirm expiry limits and role mappings for your Grafana version before cutover |
Documented paths for creating service accounts and tokens
The web interface is the fastest path for a single service account and its first token. An administrator opens the service account area, creates the account, assigns it a role in the organization that needs it, and generates a token. The UI makes the scope visible on screen, which helps when you are checking what a new script will be allowed to do. It also leaves no repeatable record unless someone writes the steps down, so it suits proofs of concept, one-off setup and emergency access rather than fleets of automation.
- Best fit: a single account for a single script, or a token you need today to validate a migration step.
- Scope check: confirm the organization selected before you save, because a service account belongs to one organization.
- Role choice: select the lowest role that works, and test whether the None role is enough before granting more.
- Token capture: copy the token when it is shown, since it is not displayed again and must go straight into your secret store.
- Audit gap: UI clicks are not version controlled, so record the account name, organization, role and expiry in your change ticket.
Before you rely on a UI-created account, confirm the menu names and options in the Grafana documentation for your installed version.
The HTTP API lets a script or pipeline create service accounts and tokens programmatically, which makes setup repeatable across environments and lets you run it under change control. Requests authenticate with an identity that already has permission to manage service accounts, so the bootstrap credential is itself a sensitive secret. API paths and request bodies have changed across Grafana releases, so treat any example you copy as a starting point rather than a contract.
- Best fit: creating the same accounts in development, staging and production, or rotating tokens on a schedule.
- Bootstrap credential: protect the administrative credential used to call the API, and remove it once automation no longer needs it.
- Idempotency: check whether an account already exists before creating it, so reruns do not fail or duplicate identities.
- Output handling: write returned tokens directly to your secret manager and never to pipeline logs.
- Version check: confirm each endpoint, method and payload field in the Grafana API documentation for your version before you build on it.
Run the calls against a non-production instance first and confirm the returned role and organization match what you requested.
Provisioning loads configuration from YAML or JSON files at startup, and Grafana documents further as-code options including Terraform and Git Sync. This path suits configuration you want reviewed, versioned and reproduced, such as data sources and dashboards. Service account and token provisioning depends on the Grafana version and tooling you use, so confirm in the provisioning documentation and your provider documentation which resources are supported before you design the workflow around them.
- Best fit: environments that must be rebuilt from source, such as disaster recovery or new regions.
- Review trail: changes arrive through pull requests, which gives security and platform teams an approval point.
- Secret separation: keep token values out of Git, and inject them from a secret store at deploy time.
- Drift control: decide whether manual UI changes are allowed, since provisioned resources can conflict with edits made in the interface.
- Tooling maturity: check the provider or Git Sync status for your version before committing to it for identity resources.
Confirm which resources you can manage as code, and which still need the UI or API, before you plan the cutover.
How to move callers from API keys to tokens in phases
This sequence moves every script and CI job from a legacy Grafana API key to a service account token without losing access, so each caller is verified on its new credential before the old key is revoked and every token has a named owner.
Inventory every API key and its callers
Start by listing every API key in each Grafana organization, with its role, creation date, last-used date where your instance shows it, and the name of the owner. Then find the callers that use each key, including cron jobs, CI pipelines, provisioning tools, dashboard backup scripts and team-built integrations, by searching repositories, pipeline definitions and secret stores for the key name and the Grafana URL. The step is complete when every key maps to at least one caller and every caller has a named team that can be contacted. Keys with no identifiable caller become candidates for revocation after review, not silent deletion.
Pilot with one non-critical script
Choose a single low-risk caller, such as a read-only dashboard export or a report that a failed run does not page anyone about, and migrate it first. The owning team creates a service account in the target organization, adds a token, and points the script at the new credential in a test environment before changing production. The pilot is done when the script runs successfully on the token for several scheduled cycles and the team records any surprises, such as endpoints it calls that the service account role cannot reach. Those findings shape the role design in the next step.
Create tokens with the narrowest role and org scope
For each remaining caller, create a service account in the organization that the caller actually uses, and assign the lowest role that lets it do its job. Service accounts are scoped to one organization, so a caller that touches two organizations needs one account in each. Use the None role as the starting point for callers that only need to be denied access by default and then granted specific folder or dashboard permissions, and confirm in your Grafana version how the None role behaves for each API path the caller uses. Set a token expiry that fits the rotation schedule your team can actually maintain, and confirm the maximum expiry allowed by your instance configuration. The step closes with a review record for each account listing its organization, role, token expiry and owner.
Run old and new credentials in parallel
Deploy the new token alongside the existing API key in each caller's configuration, so the caller can read either credential and falls back to the legacy key if the token fails. Keep both working through a full cycle of the job's normal schedule, including any weekly or month-end runs. Monitor authentication failures and the API call logs your reverse proxy or Grafana logs provide, and check that the token is the credential being used. The parallel period ends when the new token has completed every scheduled run without a fallback to the old key, and the team has confirmed the outputs match what the old key produced.
Cut over each caller and rotate by adding before removing
Switch each caller to the new token alone, one team at a time, starting with the callers that are least critical. Use the same add-before-remove pattern for every future rotation: create the new token, deploy it to the caller while the current token still works, verify a successful run, and only then revoke the old token. This ordering means a job never has zero valid credentials during a change. Record the cutover date, the token name and the verifying run for each caller, so the next rotation follows the same procedure. The cutover is complete when no caller reads the legacy key in its logs or configuration.
Revoke old keys after confirmation
Once a caller has run successfully on its new token for the full observation window agreed with its owner, revoke the corresponding legacy API key. Revoke keys in batches only after every caller in the batch is confirmed, and keep a record of the key names and the date of revocation. If a caller fails after revocation, the rollback path is to issue a new token to that caller, not to restore the old key, so the team should confirm the owner can do this quickly. The step is done when the API key list for each organization contains only keys that have a documented owner and purpose, or none at all.
Document owners and set the review cycle
Maintain a register of every service account and token with its organization, role, scope, expiry date, owning team, the systems that use it and the rotation procedure. Store the register in a location the security and platform teams can both audit, and link each entry to the pipeline or job that consumes the token. Agree with your security team how often ownership is confirmed and how the register is reconciled against the live Grafana instance, and ask your platform vendor or partner to confirm how your Grafana version reports token last-used times. The step is done when every token in the instance appears in the register and the next review date is set.
Rotating tokens without downtime in scripts and CI
The core rule for rotating a Grafana service account token without downtime is overlap: create the new token while the old one is still valid, move every script and pipeline to the new value, prove the jobs run, and only then revoke the old token. Store each token in the secrets system your team already uses for CI variables and deployment credentials, such as your CI provider's secret store or a vault, rather than in repository files or shell profiles. Keep one secret entry per service account and per consumer where practical, so you can rotate one pipeline without touching the others. Record the owner, the service account, the organization it belongs to and the date of issue beside each entry, because rotation is much harder when nobody knows which jobs depend on a token.
Rotation works in order because the old and new tokens coexist for a window you control. Issue the replacement, update the secret store, then trigger a non-production run and a scheduled production run before revoking anything. Where a pipeline reads its token at start-up, a running job keeps its old value until it finishes, so schedule the cutover between jobs. Confirm the token expiry behavior for your installed Grafana version, including whether a maximum lifetime is enforced and what the default is, before you set reminders. Check the Grafana authentication and access control documentation for your edition, and confirm how tokens are validated when several Grafana servers sit behind a load balancer.
Test the rotation the same way you test a deployment. A useful check is a small read-only request, such as listing dashboards or querying a data source health endpoint, sent with the new token from each pipeline's runner network. Confirm that the request succeeds, that the token has only the role and organization scope it needs, and that the old token now returns an authorization error. For high availability, the Grafana high availability docs describe the shared database that holds server state, so run the check against more than one node and confirm the token behaves the same way on each. Have your platform team confirm how the installed version stores tokens in the database and whether any alerting configuration needs changes during rotation.
What drives the cost and duration of the migration
This table lists the factors that expand or shrink the work of replacing legacy API keys with service accounts, and whether each cost is paid once or keeps recurring. Read each row as a question to answer about your own estimate before requesting a quote. The main takeaway is that the inventory of callers and organizations usually sets the effort, while token rotation and owner reviews continue after cutover.
| Driver | Why it changes cost or effort | One-time or recurring | How to keep it down |
|---|---|---|---|
| Number of API keys and callers | Each key maps to a script, job or system that must be found, tested and switched to a token | One-time (inventory and cutover) | Export the key list and tag each key with its caller before changing anything |
| Organizations involved | Service accounts are created per organization, so each org needs its own accounts and tokens | One-time (setup and verification) | Confirm which orgs hold live callers and retire unused ones first |
| Caller type: CI, scripts, provisioning or external systems | CI jobs and external systems need secret delivery changes, while scripts may only need a new variable | One-time (cutover) | Group callers by type so each group is changed with one repeatable pattern |
| Distinct roles required | Each token needs the least role its caller uses, and mixed-role callers need extra design | One-time (role design), then recurring review | Map every API call to the minimum role, then reuse a small set of role profiles |
| Token expiry limits | Expiry settings decide how often tokens must be replaced, which affects the rotation schedule | Recurring (rotation) | Confirm the expiry limits in your Grafana version and set them to match your rotation process |
| Secrets tooling maturity | Teams with a central secret store can rotate tokens automatically, while others edit each caller by hand | One-time (tooling) and recurring (rotation) | Store tokens in the secret manager your teams already run before cutover |
| Testing needs | Each caller needs a check that it still reads and writes what it did before, including failure paths | One-time (test design), then recurring for changes | Build a read-only smoke test per caller and run it in a non-production instance first |
| Ownership and documentation | Each service account needs a named owner, and the owner list must be kept current | One-time (documentation), then recurring (owner reviews) | Record owner, purpose, role and expiry in one register at creation time |
What usually goes wrong when API keys are retired
Retiring legacy API keys tends to fail in predictable ways: callers nobody remembered, tokens with more power than they need, and credentials that expire in the middle of the night. The cards below cover the six risks we see most often, with the check or design choice that prevents each one.
Unknown callers break after revocation
A legacy key often serves a cron job, a dashboard exporter or a forgotten deployment script that never appears in any inventory. Revoking it first and searching later turns that caller into an outage. Before revoking anything, review the key's recent usage where your logs allow it, ask each owning team to list the scripts and pipelines they run, and revoke only after a parallel service account is working for each one. Keep the old key active until that check is complete.
Tokens granted broader roles
Teams often recreate a legacy Admin key as a service account with the same role, which copies the old blast radius into a new object. Assign the None role by default, then add only the organization-scoped permissions each caller needs, such as reading one folder or writing annotations. Confirm the role in the service account settings for every token, because a broad role on a token used in CI can change dashboards and data sources across the instance.
Tokens with no owner
A token that nobody is responsible for has no one to ask when it fails, rotates or needs a new scope. Give every service account a name that states its purpose, an owning team and a contact, and record the caller it serves in your inventory. Review ownership during each rotation cycle, and remove service accounts whose callers are gone instead of leaving them active as a precaution.
Expiry causing silent job failures
A token that expires on a Friday evening can stop a backup export or a reporting job without any visible error in the caller's own logs. Set an expiry that matches your rotation calendar, and make sure the caller records HTTP 401 responses as failures that alert someone. Ask Grafana which maximum token lifetime your version allows before you design the schedule, because the limit determines how often each caller must be updated.
Rotating every caller at once
Replacing a token in all scripts and pipelines in one change window means a single mistake stops everything, and rollback means restoring several secrets at the same time. Rotate in stages: create the new token, deploy it to one caller, confirm successful requests, then move the next caller. Keep the old token valid during the overlap so a failed step can be reversed without a second emergency change.
Secrets copied into logs or repositories
Tokens pasted into shell history, CI build logs, ticket comments or committed configuration files outlive the migration and remain readable by people who should never see them. Store each token in your secrets manager or CI secret store, reference it by name in pipeline definitions, and mask it in job output. Scan repositories and log exports for the Grafana token prefix after rotation, and revoke any token that has appeared where it should not.
Questions teams ask before moving API keys to service accounts
What drives the cost of moving API keys to service accounts?
Cost depends mainly on how many scripts, pipelines and integrations call Grafana and how safely each one can be changed. Each caller needs its own service account, a token with an agreed expiry, a role matched to its actions and a new secret in your secrets tooling. The one-time work covers the inventory, the change to each caller and the cutover. Recurring work covers token rotation on the schedule you set and periodic reviews of who still holds access. Ask any quote to itemize those parts, name the Grafana version and deployment model it covers, and list the callers in scope. A fixed scope agreed in writing before work starts makes any change in caller count visible in the price.
How long does it take to move API keys to service accounts?
The duration depends on the number of callers, how well they are documented and how many teams must approve changes. A handful of scripts owned by one team is a different job from many CI pipelines spread across several groups. The inventory and ownership decisions usually set the pace, because a key cannot be replaced until someone knows what uses it and who can deploy the change. Ask for a timeline that separates the inventory, the caller changes, a period in which the legacy key and the new token both work, and the final revocation of legacy keys. The timeline in the written scope should be the basis for planning, not a figure taken from general experience.
What must our team provide before the migration starts?
Three things decide whether the work runs smoothly: a caller inventory, named owners and a secrets store. The inventory lists every script, pipeline, integration and monitoring tool that sends requests with an API key, including the organization it targets and the actions it performs. Each entry needs an owner who can approve a credential change and schedule a restart. The secrets store must allow new tokens to be written, read and rotated without pasting them into tickets or chat. Your team should also provide administrative access to the Grafana instance and a non-production instance for trial runs. Confirm that all three exist before scoping, because gaps usually appear during the cutover.
Can our own team move API keys to service accounts without help?
Many teams can do this internally if they have someone who administers Grafana, an owner for each caller and a safe place to test. The work is mostly careful inventory, small configuration changes and a coordinated cutover rather than new engineering. Outside help is most useful when the caller list is long, when callers belong to teams with different change windows, or when internal staff lack time to run a test period. Confirm that your team can change every caller and test in a non-production instance before deciding. DeployOpen plans, implements and documents this kind of change inside your own infrastructure, and most engagements are fixed-scope, so you can compare a written scope against your internal estimate.
How should we handle API keys used by provisioning tools?
Start with provisioning because automation often uses a key for years without anyone reviewing it. Grafana's provisioning files define data sources, dashboards, plugins and alerting resources at startup, and tools such as Terraform or Git Sync can apply changes on a schedule. Identify each job that writes to Grafana, then assign it a service account token with the narrowest role that still permits its changes. Store the token in your secrets tooling and reference it at runtime instead of committing it to a repository. Migrate one tool at a time and check the result after each run, because a failed provisioning job can leave dashboards or alert rules out of date. Review the provisioning documentation for the file formats your instance uses.
What should we confirm about our installed Grafana version first?
Confirm the exact version, how it was installed and whether it is still inside Grafana's support window before planning the change. Grafana's upgrade guidance describes major releases once a year, minor releases every other month and patch releases at least monthly. Each minor release is supported for 9 months, and the last minor release of a major version for 15 months. An unsupported or near-end-of-support version may need an upgrade before the credential work begins, so the upgrade path should be decided first. Check the version in your server's status information or in the image tag your deployment uses, then compare it with the releases page before writing the scope.
Can we rotate tokens without downtime during the cutover?
Yes, if the legacy key and the new token are both accepted for a short, agreed period while each caller is switched. The usual sequence is to create the service account and token, add the new secret to each caller, confirm a successful call, and revoke the legacy key only after no caller uses it. Watch for callers that keep a key in memory, cached files or long-running jobs, since they continue using the old credential until restarted. In a high availability setup with several Grafana servers behind a load balancer, confirm that token changes are visible on every node before revoking anything. Verify in your version's documentation that the two credentials can coexist, and schedule any necessary restarts inside your normal change windows.
How DeployOpen scopes and delivers the migration
DeployOpen plans, implements and hands over self-hosted open-source platforms inside your own infrastructure, so the Grafana service account migration runs in your environment under your access controls. Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
How we scope
- Inventory every legacy API key, the scripts and CI jobs that use it, and the organization each key touches.
- Define the target service account per organization, role and purpose, including where the None role fits least-privilege needs.
- Agree token expiry limits, rotation ownership and the cutover window in the written scope before any change is made.
How we build
- Create service accounts and tokens in a staging Grafana, then test each script and pipeline against them before production.
- Stage the rotation so new tokens are issued and consumed while old keys still work, avoiding a single outage point.
- Verify API calls per organization and confirm that revoked keys no longer authenticate, with results recorded for review.
How we hand over
- Document the service account inventory, role assignments, token expiry settings and owner for each token.
- Provide the rotation runbook, including how to renew tokens and how to revoke one quickly if it leaks.
- Walk your operations and security teams through the new model so they can add accounts without reintroducing legacy keys.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer after handover.
- Retainer work can cover periodic token reviews, expiry checks and changes to automation as scripts and pipelines evolve.
- Confirm support terms, response expectations and included reviews with DeployOpen in the written quote.
