Apache APISIX Ingress Controller on Kubernetes

Start with a written verification plan that ties each Kubernetes object to the gateway state it should produce. For every Ingress resource and every APISIX custom resource in your design, record the route, upstream, plugin chain and TLS binding the controller should create. Then test the chain end to end: confirm the generated route exists in the gateway's configuration, send requests to the host and path and check that they reach the intended upstream service and not a neighbor with a similar name. Include negative cases, such as a path that should return a 404 or a method that a plugin should reject. Plugin behavior needs its own checks: rate limits, authentication and header rewrites should be tested with requests that should pass and requests that should fail. Keep the expected results in version control so that a later change can be compared against the same baseline.

Failure tests show whether the gateway behaves the way you chose, which is not always the default. Delete a test Ingress resource, then apply an invalid one, such as a malformed route or a reference to a missing Secret or Service. Observe whether the gateway keeps serving the last good configuration, drops the route or stops applying changes. Decide this before testing, because the answer determines your alerting and rollback steps. Record what the controller reports in its resource status and logs, and what your monitoring sees. Then restore the valid resource and confirm that traffic recovers without manual edits to the gateway. Repeat these tests after each controller or gateway version change, since the handling of invalid input can differ between releases.

Restart the controller pods and the gateway pods separately, and then together, while sending steady test traffic. Measure dropped requests, the time before the routes are served again, and whether any configuration is lost or duplicated after the restart. In traditional mode, where the data and control planes share one deployment and etcd stores the configuration, also test what happens when etcd is unavailable. Isolate or stop one etcd member first, then the whole cluster for a short window, and check whether existing traffic continues on the configuration already loaded. Check whether configuration changes made during the outage are applied once etcd returns. Confirm the expected behavior with your own etcd setup and the deployment mode you chose, because the result depends on the design, not on the controller alone.

Run the old and new ingress side by side for the same host before any traffic moves. Use a test hostname or a weighted DNS or load balancer split, and compare responses from both paths for the same requests. Check that TLS certificates are served correctly, including the certificate chain, the server name indication match and the expiry date, and that the new path uses the same certificate source your security team approved. Verify that headers are passed as designed: forwarded client address, host, correlation identifiers and any headers added or removed by plugins. Compare response headers too, since security headers are often lost in migration. Document every difference and decide whether each one is expected before cutover.

Confirm every command and every log message against the exact APISIX and Ingress Controller versions you install before you put them in a runbook. Documentation for one release line can differ from another, and log wording changes between releases. Pin the versions in your test plan, record the results of each test with the date, the versions and the configuration revision, and keep those records with your change tickets. Ask your implementation partner or internal platform team to walk through the failure cases with the people who will be paged, so that the expected behavior is agreed before an incident rather than during one.

Once the APISIX Ingress Controller is serving traffic, the operating routine decides whether it stays reliable. Start by monitoring four things: the health of the controller pods and the gateway pods, configuration sync errors reported by the controller, certificate expiry for every TLS secret the routes reference, and upstream health as seen by the gateway. Pod restarts and readiness flaps are the earliest signs of trouble, so alert on them rather than waiting for user reports. Sync errors matter more than most teams expect. If the controller cannot translate a Kubernetes resource into a route, the gateway keeps serving the last good configuration, which can hide a broken change for a long time. Confirm which metrics and logs your installed versions expose, and check the APISIX observability documentation for the exact names before writing alert rules.

Backups depend on the deployment mode. If you run traditional mode, the etcd cluster holds the gateway state, so back it up on a schedule and test a restore into a non-production cluster. In that case the Kubernetes objects are not the only source of truth for runtime configuration. Whatever the mode, store the Kubernetes manifests or Helm values that define the gateway resources, the routes, the TLS secrets and the controller settings in version control. Rebuilding the environment should require only the repository, the secrets store and the cluster. Confirm with your platform team where each secret lives and who can read it, because a backup that includes credentials needs the same protection as the original.

Treat the controller and the gateway as two separately versioned components. The Ingress Controller is a separate project from the gateway, and the downloads page lists its own release line, so pin the version of each in your deployment definition and record both in your change log. Upgrade in a staging cluster first, with the same ingress class, plugins and route set as production, and run your test suite against it. Read the release notes for each version before you move, paying attention to changes in resource schemas, plugin behavior and default settings. The downloads page does not mark any release as long-term support, so there is no vendor-defined support window to plan around. Your team must set the upgrade cadence, decide how far behind the current release it is willing to run, and assign an owner to track security advisories.

Documentation is what lets someone other than the original builders operate the ingress. Record who owns each ingress class and which controller instance serves it, so two controllers never claim the same class without a decision. Keep a route inventory that maps each Kubernetes Ingress or APISIX resource to its host, path, upstream service and owning team. Maintain a plugin list for each route, including authentication, rate limiting and any custom plugins, because a plugin change on a shared route can affect many teams. Finally, write a rollback procedure that names the previous controller and gateway versions, the commands or Helm revision to restore, the order of steps, and the check that confirms traffic is healthy again. Rehearse it in staging at least once.

The support model for this stack is covered in the separate upgrades and support topic, including what a support agreement can cover and how it relates to your release cadence. Decide before go-live whether your team will carry the routine described here alone or with a partner, because that choice determines who is on call for controller failures, who approves upgrades and who keeps the runbooks current. Confirm the scope of any support arrangement in writing before you need it.