Open-Xchange high availability

A failover test plan should show two things before go-live: the platform keeps serving users while a component is down, and when it cannot, it fails in a way your operations team can see and recover from. Write one entry per test with three fields: the expected user-visible result, the measurement, and the rollback step. Measure from the user's side with a scripted check that logs in, sends and receives a mail, reads calendar and contacts, and opens a file in Drive or a PIM attachment. Run every test in a staging environment built from the same Helm chart version and values files you plan to use in production, and pin those versions before the first test begins.

Start with controlled pod termination for middleware and frontend replicas. Delete one replica at a time with kubectl while the scripted check runs. The expected result is that new logins and mail flow continue, with at most a brief error for requests in flight at the moment of termination, and that Kubernetes replaces the pod and readiness returns to normal. Measure the error window from the synthetic check, not from the cluster dashboard. The rollback is to restore the original replica count and confirm every pod is ready before the next test. Next, stop one MariaDB Galera node. Quorum should hold. Write test mailboxes and calendar entries through the application, rejoin the node, then compare row counts and checksums across members before you record the test as passed.

Redis needs a test that reports actual behavior, not assumed behavior. Stop the Redis instance that holds caching and session data, and record whether users keep their sessions or must sign in again. Confirm the answer against your own setup, since the default littlered operator deployment on Redis 7.4.x and a custom or unmanaged setup can behave differently. For storage, interrupt the path to the distributed file system or object store for a short, pre-agreed window. Drive and attachment reads and writes should fail with clear errors, mail and calendar should keep working, and no file should be corrupted after access returns. The rollback restores the network path, and you verify a sample of files by checksum. For the ingress layer, disable the Istio gateway or the ingress-nginx path, record how traffic behaves during the outage, and reapply the original configuration as the rollback.

If multi-site operation is in scope, run a full site-level failover last, because it touches every component above. Confirm database and storage replication status before the test, direct traffic to the secondary site, and run the complete check set. Plan failback as its own test with its own pass criteria. Then write every result into the handover pack: test identifier, date, build and chart versions, operator, expected and observed behavior, log or screenshot references, rollback time, open defects, and a named owner for each item. Agree the pass criteria with the client before testing starts, so the handover pack supports the go-live decision and nobody has to reinterpret results afterwards.

The operations team needs a monitoring baseline before the first mailbox goes live, because high availability only works if someone sees a degraded node before users do. For MariaDB Galera, watch the node state reported by wsrep_local_state_comment, the member count in wsrep_cluster_size, and flow control through wsrep_flow_control_paused. A node that stays paused slows every write in the cluster, even when it still answers queries. For Redis, which holds caching and session data for OX App Suite v8, track availability, memory use against the configured limit and eviction counts. Evictions under memory pressure can drop sessions and force users to sign in again. Agree alert thresholds with your own team, then test each alert once by taking a node down in a non-production environment.

Next, cover the layers around the data stores. On Kubernetes, alert on OX deployments with fewer available replicas than expected, on pods that restart repeatedly and on nodes under resource pressure, since a crash loop often appears here before it appears in a user report. The persistent tier that holds OX Drive files and PIM attachments needs capacity trending and latency tracking, whether it is NFS, Ceph RadosGW, Scality RING or AWS S3. Growth is easier to forecast from mailbox and file counts, but only if someone reviews the trend on a fixed schedule. At the edge, the Istio ingress gateway should expose request counts and 5xx error rates by route, so a failing backend shows up as a rising error ratio on a specific path.

Backups must cover two separate things: the MariaDB data store and the persistent storage tier. For MariaDB, agree the backup method with your database team and confirm that it produces a consistent copy of the Galera cluster. Confirm also that binary logs allow recovery to a specific point in time. For the storage tier, the method depends on the backend you chose, and each option has its own replication, snapshot and versioning features. Confirm retention periods and the restore procedure with the platform owner. Decide whether Redis data needs backup at all, or whether rebuilding its cache and sessions after a loss is acceptable for your users. Finally, restore a copy into a separate environment, because a backup that has never been restored is an assumption, not a recovery plan.

Failover tests are not a one-time go-live activity. Each Helm chart upgrade, Kubernetes version change or Istio change can alter pod scheduling, service routing or how the database connections reconnect, so repeat the failover scenarios after each one. Keep the same test list every time: drain a Kubernetes node, stop one Galera node, fail one Redis instance, and confirm that sessions survive and mail keeps flowing. Record the outcome and the time each service took to recover, and compare the results against the previous run. The upgrades and support topic covers how to choose releases and stay inside the support window, so this section assumes that decision is already made and focuses on proving the cluster still fails over correctly.

The handover should leave the team with runbooks and an architecture record they can use without the implementation partner in the room. The runbooks should cover replacing a Galera or Redis node, expanding storage, renewing certificates at the Istio layer, restoring from backup and rolling back a failed Helm release. The architecture record should list every component and version in use, including the Kubernetes, Istio, MariaDB and Redis versions, the storage backend, the network paths, and the named owner of each monitoring alert and backup job. Place both documents under change control and agree who approves updates. Ask for both to be reviewed by the team that will operate them, and test one runbook end to end before the handover is signed off.