OpenSearch index lifecycle and data retention

Retention rules are usually written as durations per data class, such as keeping a category of records for a stated period after an event. Index State Management does not work with data classes. It evaluates each index against a policy's states and conditions, using index age, document count, index size and rollover thresholds. Before writing any policy JSON, build a mapping table with one row per requirement. Each row should name the data class, the index pattern that carries it, the retention start event, the retention length, the action that ends retention and the ISM policy that will enforce it. A requirement that cannot fill every column is not yet ready to be enforced.

Group requirement categories by the retention behavior they need, then create one policy template for each group. A template pairs an index pattern with a policy through an ISM template, and the priority field decides which template wins when several match. Templates apply to indices created after they exist, so existing indices need an explicit policy assignment. The retention start date is the main point of friction. ISM ages an index from its creation time, and after a rollover the age of the new backing index starts again. If the compliance owner counts retention from the event timestamp in the data, an index-age rule can keep records too long or delete them too early. Confirm this for each source before setting min_index_age values.

Legal holds and extended retention need their own design, because a standard delete state will remove held data on schedule. One approach is to give held indices a policy with no delete action, assigned explicitly through the change policy API. Another is to route held data into a separate index pattern that no delete template matches. Either way, new rollover indices must inherit the hold, so test that a rollover of a held index produces a successor that stays protected. Record who approved each hold, the scope, and the release procedure. Confirm with your compliance owner whether a hold must be logged outside the cluster, and whether the audit logs from the Security plugin satisfy that need.

Overlapping patterns are the most common source of silent policy conflicts. A hold pattern, a general log pattern and a security pattern can all match the same index name. Keep a register of every index pattern and its priority, and give each template a unique priority so no two overlapping templates tie. After any change, create a sample index with a representative name in a test environment and confirm which policy attaches, then check the managed index status for each live index. Verify the exact matching and priority behavior in the documentation for your OpenSearch version before relying on it, because the rules that decide conflicts are the ones most likely to surprise an operations team.

A retention policy is only trustworthy after someone has tried to break it. Before you rely on Index State Management to roll over, shrink, move and delete indices, run a staged test that confirms each index picks up the intended policy, that every state transition does what the policy says, and that a failure stops the delete instead of letting it run. Treat this as an acceptance test for the lifecycle design, and keep the results as evidence for the retention owner and any later compliance review. Start with policy attachment. Create a test index that matches each index pattern in your index templates and confirm it receives the expected policy ID, using the explain view for each index. Then add a deliberately non-matching index, such as one created by a script that skips the template, and confirm it shows no policy. That second check matters because the most common silent failure is an index that misses a template and never gets a policy, so it is never rolled over, moved or deleted. Build a report that lists every index without a policy and schedule it to run after each deployment. Next, exercise the transitions in staging with a copy of production-shaped data. Force a rollover and confirm the new write index exists, the alias points to it, and document counts add up across the old and new indices. Run a shrink and check the target shard count, the document count against the source, and that the source is closed or removed as the policy specifies. Then move an index to a lower-cost tier, if you use one, and run a query against it. A moved index that stops serving queries is an availability defect, not a storage success. The failure tests answer the question of whether a policy fails safely. Create a policy whose state never advances, for example by pointing a transition at a condition that cannot be met, and confirm that alerting on a stalled state fires before anyone assumes the data is aging out. Next, configure a delete action that runs before a snapshot succeeds, then simulate a failed snapshot by making the repository unreachable, and check whether the delete still happens. In a well-designed policy, data should not be removed while the backup it depends on is missing. Finally, fill the disk on a test data node, or simulate it with the vendor's disk watermark settings, and observe whether delete actions still run and whether the cluster blocks writes as expected. Finally, prove recovery, not just deletion. Restore a snapshot of an index that has already passed its delete age into an isolated cluster, confirm the document count and a sample of queries match expectations, and record how long the restore took on your hardware. Confirm the time window your retention requirement allows for recovery, and write down the people who approve a restore. Ask your implementation partner to deliver these test cases as scripts with pass and fail criteria, so the tests can be rerun after every OpenSearch minor upgrade.

The first weeks after go-live show whether the lifecycle design holds. Index State Management runs each policy against the indices attached to it, so the operating team needs a standing check on managed indices instead of waiting for a storage alert. Use the explain view to see each index's current state, the action in progress and any failed step. Indices that sit in one state longer than the design expects usually point to a rollover condition that never triggers, a missing alias, or an allocation problem blocking a move between tiers. Alert on failed actions and on indices that stop advancing. Track total index size and shard count per tier each week, because shard counts that drift upward often signal template or rollover settings that no longer match the data volume.

Snapshots are the restore path for anything the policies delete or move, so confirm they succeed on the schedule the retention matrix assumes. Snapshot management policies create and expire snapshots on a timetable, and each run should be checked for success, partial results and repository errors. Watch repository free space and the age of the newest successful snapshot for each index pattern. Snapshot expiry should match retention classes, because a snapshot kept longer than the matrix allows can hold data the business has agreed to delete. Schedule a restore test on a non-production cluster at a fixed interval, and record how long it took and any gaps, so the runbook reflects what actually happened.

Every change to a policy, index template or alias needs a review against the retention schedule before it reaches production. A policy edit does not always reach indices that are already attached, and a broad index pattern can pull in data that belongs to another retention class, so the reviewer should list which indices each change touches. Treat OpenSearch upgrades the same way. Minor releases arrive about every eight weeks, so apply each one to a copy of the cluster inside the maintenance window and retest the lifecycle policies. Create a test index, step it through every state, and confirm that snapshots and deletions run as written. Record the version tested and the results before approving the upgrade for production.

Four documents make the handover durable. The retention matrix lists each data class, its index pattern, its tier path, its retention period and the legal or business basis for that period. The policy and template inventory maps every ISM policy, index template and alias to the pattern it governs and to its version in source control. The snapshot and restore runbook covers repository setup, schedules, expiry rules, restore steps and the record of past tests. Finally, name an owner for each retention class, including who approves changes and who answers when a deletion is questioned. Confirm with legal, records and security teams which retention periods are binding before the matrix is signed, because the platform will enforce whatever the matrix says.