LiteLLM with HashiCorp Vault and cloud secret managers

The access mode decides whether LiteLLM may read secrets from a store, write secrets into it, or both. LiteLLM's secret manager settings include access_mode with three values: read_only, which is the default, write_only, and read_and_write. Pick the mode for each store separately, because a single Vault mount, AWS Secrets Manager prefix, Azure Key Vault or Google Secret Manager project may hold provider keys that the gateway only needs to read, while another location receives virtual keys that the gateway creates. Treat the mode as the first control on what the gateway can do to your secrets, before any role or policy you add on the store side.

Provider API keys for OpenAI, Anthropic, Bedrock, Azure or other upstream services are normally created by your platform or security team and then consumed by the gateway at startup or when a model is called. For these, read_only is the right mode. The gateway has no reason to change a provider credential, and a read-only setting means a compromised or misconfigured proxy cannot overwrite or delete the keys that every team depends on. Confirm with your own runbook how a provider key is rotated, because rotation usually happens outside the gateway, and the gateway should pick up the new value through its normal read path.

Virtual keys are the opposite case. When LiteLLM issues a new virtual key and store_virtual_keys is enabled, the gateway must write that key into the configured store under the configured prefix. Write_only fits a location where the gateway should only deposit new keys and never read back what is already there, which limits exposure if the proxy is compromised. Read_and_write is needed when the gateway must also read a stored virtual key later, for example to confirm it exists or to reuse it. Choose the narrower mode that still lets the workflow run, and test the key lifecycle, including creation and any later lookup, against a non-production store before you grant the mode in production.

The store-side role should match the mode. For HashiCorp Vault, AppRole policies can be limited to the read capability on the provider-key path and the create or update capability on the virtual-key path. AWS, Azure, Google and CyberArk offer equivalent per-secret or per-path permissions, and you should confirm the exact permission names with each vendor's documentation rather than copying a policy from elsewhere. Grant only the operations the mode requires, and avoid broad roles that also allow listing, deleting or changing access policies, because those operations are not part of the gateway's job and would enlarge the blast radius of a leaked credential.

Document the chosen mode for every store and every team in a single table that your security team signs off. Team-level secret manager settings can point a team's virtual keys to a different store, so each team entry needs its own access_mode and prefix review. Schedule a periodic check that compares the mode in the gateway configuration with the role actually bound in each store, because drift between the two is a common source of unexpected write access or failed key creation. Ask your implementation partner or internal team to show the test evidence for each mode before go-live.

Team-level secret manager settings let each team bring its own key management configuration, and LiteLLM applies that configuration when it creates virtual keys tied to the team. The decision you face is whether all teams share one secret store or each team gets its own. Per-team settings make the second option possible, but they do not make it the right choice by default. Before you write any team configuration, decide where the provider API keys and virtual keys for each team will live, who can read them, and who can change the settings that point LiteLLM at them.

One shared store versus per-team stores. A single shared store, such as one HashiCorp Vault mount or one AWS Secret Manager account and region, is simpler to back up, audit and rotate, and it keeps one set of access policies to review. Per-team stores give a team separate access boundaries and separate rotation schedules, which can matter when a team is a regulated business unit or a tenant with its own contractual obligations. The cost is more policies, more backends to monitor and more places a misconfigured setting can send a key. Many organizations pick one shared store for most teams and reserve separate stores for the few teams with a hard isolation requirement.

Naming and prefix discipline. Team settings rely on the prefix that LiteLLM uses for stored virtual keys and on the paths or names of the secrets it reads. Agree on a naming scheme before the first team is onboarded, for example one that embeds the team identifier and the environment, and enforce it in a review step rather than by convention. Confirm whether two teams can ever resolve to the same prefix, and whether a team's settings can point at another team's path. Overlapping prefixes are the most common way a key ends up visible to the wrong group, so test this with a non-production key before rollout.

Who owns what. Platform admins own the gateway, the default key management system, the backend credentials LiteLLM uses to authenticate (for Vault, the AppRole or other method chosen for the proxy), and the access mode for each backend. Team admins own the choice of model, budgets and the team's own key lifecycle inside the boundary the platform has approved. Write this split down as a responsibility matrix, and make sure team admins cannot change the secret manager fields that point to a store outside their approved set. Also decide who approves a new team configuration and who can revoke a virtual key when a person leaves.

What to confirm in your LiteLLM version. Team-level secret manager settings are an Enterprise capability in the secret manager documentation, so confirm your license covers them. Then verify, in your own deployment and on the version you run, how team settings interact with the global settings: whether a team configuration fully replaces the default, whether access_mode applies per team, how an unset team field falls back, and whether the prefix is applied as you expect. Check the current LiteLLM secret managers documentation and release notes for your version, and record the results as a test case. Do not assume behavior carries over from another release line.

The test plan should prove three things before the gateway serves production traffic: provider keys load from the secret store after a restart, virtual keys are written to the store under the expected prefix, and the gateway fails visibly when the store cannot be reached. Run every check against the same key_management_system, access_mode and team-level settings that production will use, not against a simplified development configuration. Before the first run, write down the expected result for each provider, each team and each secret path, including which AppRole or policy applies. Confirm in the LiteLLM secret manager documentation for your release which settings are read once at startup and which are read on each request, because that determines what a restart test actually proves.

For the restart test, remove the provider keys from the deployment, restart the proxy pods, and send a test request through a route for each provider. Each request should succeed, and the gateway logs should show that the key came from the configured store. Repeat this for every team that has its own key management configuration, since a team-level setting that fails silently would leave that team's traffic depending on a different source. For the virtual key test, create a key for a team through the admin UI or API, then read the store directly with a separate operator identity. Confirm that the entry exists under the configured prefix, sits in the path belonging to that team, and that the gateway's access_mode permits the write you expect. If access_mode is read_only, confirm that the gateway cannot create entries at all.

For the failure test, revoke the AppRole secret ID or disable the policy that the gateway uses, then force a fresh read by restarting a pod or waiting for the next lookup, depending on your design. Send requests and record the exact error text, the HTTP status returned to clients, and whether your alerting fires. The expected outcome is a clear authentication error from the store path, with no fallback to a cached or plaintext value that nobody intended to keep. Restore the AppRole and confirm that the gateway recovers without a manual redeploy. If recovery requires a restart, document that as an operational step in the runbook. Ask your team to name who receives the alert and who has the authority to restore access.

After the functional tests, scan for leftover plaintext. Check the environment of running pods, the container specs and Helm values in Git, CI job logs, and the gateway's own logs, including any request or debug logging that could echo a key. Use a secret scanner with patterns for each provider's key format and for your virtual key prefix, and treat any hit as a finding to fix before go-live. Rotation tests should run in a non-production gateway first. Rotate one provider key and one virtual key there, confirm that the old value stops working, and confirm that the new value loads the way your design intends, whether that is by restart or by a cache refresh. Confirm the cache behavior for your version before scheduling the same rotation in production.