GitOps drift is more than a difference between a Git repository and a Kubernetes cluster: it is a signal that the intended operating model, the live system, or the reconciliation process has changed. This practical checklist explains how to detect drift, decide whether it is intentional, investigate safely, and restore the desired state without overwriting a legitimate emergency change.
Overview
In a GitOps workflow, a version-controlled repository defines the desired configuration for applications, policies, and supporting Kubernetes resources. A GitOps controller compares that desired state with the live cluster and works toward reconciliation. Drift occurs when those two states no longer match, whether because someone edited the cluster manually, a controller failed, a dependency changed, or the repository itself was updated incompletely.
Not every difference should be repaired immediately. An emergency patch may be intentional and still need to be recorded in Git. A generated field may differ because Kubernetes or a controller manages it at runtime. A secret may be represented indirectly, so a visual comparison can be misleading. The first task is therefore classification, not synchronization.
Use this workflow whenever a drift alert appears, a synchronization fails, a cluster receives an emergency manual change, or a platform team is preparing an access review, migration, or post-incident audit.
Argo CD, Flux, and cloud-managed GitOps at a glance
The right choice depends on the team’s operating model, existing cloud platform, notification needs, and desired separation between application delivery and cluster administration. Product behavior and command syntax can change, so verify details against the version deployed in your environment.
| Approach | Typical operating model | Drift and reconciliation workflow | What to verify |
|---|---|---|---|
| Argo CD | A Kubernetes-facing delivery service with a web interface, CLI, APIs, and application-level health and sync views. | Review an application’s sync status and resource differences, then sync approved changes. Automated sync and self-healing behavior should be enabled only when the team understands the consequences. | Application projects, destinations, RBAC, sync waves or ordering, health checks, exclusions, and notification integrations. |
| Flux | A set of Kubernetes controllers that reconcile source, configuration, and deployment resources declaratively. | Inspect the relevant custom resources and controller events or logs. Trigger or wait for reconciliation after correcting the source or dependency, then confirm ready and healthy conditions. | Controller ownership, source and Kustomization or Helm relationships, intervals, dependencies, suspend settings, and event delivery. |
| Cloud-managed GitOps | A cloud provider’s managed integration for connecting repositories and clusters, often alongside the provider’s identity and monitoring systems. | Use the provider console, API, or CLI to inspect association status, synchronization errors, and resource differences. Follow the provider’s reconciliation and rollback model. | Supported Kubernetes versions, repository authentication, regional limitations, IAM or RBAC mapping, audit logs, notification options, and ownership boundaries. |
Checklist by scenario
When the source repository changed
- Identify the commit, pull request, or merge that preceded the drift alert.
- Confirm the affected environment and cluster are consuming the expected branch, tag, path, or revision.
- Review rendered manifests rather than relying only on the source YAML. Helm values, Kustomize overlays, substitutions, and generators can alter the final output.
- Check whether the change was approved for this environment and whether a promotion step was skipped.
- Compare the intended change with the controller’s reported difference before syncing.
When someone edited the cluster manually
- Find the actor, timestamp, command, admission record, or change ticket associated with the edit.
- Classify the change as emergency mitigation, routine administration, experimentation, or unauthorized modification.
- Capture the live configuration before changing it again.
- If the edit is valid, reproduce it in the repository through the normal review process. If it is not valid, reconcile from the approved source.
- Do not assume that a successful
kubectl applymeans the change is durable; a GitOps controller may restore the repository state.
When the controller is not reconciling
- Check controller pods, deployments, events, and recent logs in the controller’s namespace.
- Confirm the controller can reach the repository, registry, Kubernetes API, and any required secret or identity provider.
- Look for authentication failures, rate limits, certificate problems, invalid manifests, and insufficient permissions.
- Check whether reconciliation has been suspended intentionally or blocked by a dependency.
- Review resource health separately from synchronization status. A controller can report that a desired manifest was applied while the workload remains unhealthy.
When secrets appear to be different
- Determine whether the repository stores encrypted, templated, externally referenced, or sealed values rather than plaintext data.
- Avoid placing secret values in issue comments, terminal transcripts, or drift reports.
- Confirm that the controller has access to the decryption key, external secret provider, or required workload identity.
- Check whether a rotation changed the live value without updating the source reference or version.
- Use metadata, checksums, references, and provider audit logs to investigate without exposing credentials.
When dependencies are involved
- List dependent resources such as namespaces, custom resource definitions, service accounts, policies, certificates, and external endpoints.
- Verify that the required API versions and custom resource definitions exist in the target cluster.
- Check ordering and readiness conditions for dependencies before retrying reconciliation.
- Separate a source error from a cluster-capacity or runtime error. Scheduling failures, quota limits, and unavailable nodes are not fixed by repeating a Git sync.
When the cluster has just changed
- Review whether a Kubernetes upgrade, controller upgrade, admission policy change, or migration altered defaulting or API behavior.
- Compare the controller version and Kubernetes API compatibility with the versions used by the cluster.
- Test the rendered resources against a non-production cluster or validation process where possible.
- Preserve the drift report and before-and-after manifests for the change record.
What to double-check before remediation
Use the following sequence to turn a drift alert into a controlled decision.
- Identify the authority. Decide whether Git, an external secret system, a cloud control plane, or a deliberately manual process owns each disputed field. Do not reconcile resources whose ownership is unclear.
- Record the live state. Save resource metadata, events, status conditions, and a redacted manifest. Include the namespace, resource name, cluster, controller, and observed revision.
- Review the difference. Separate meaningful fields, such as an image tag, replica count, role binding, or network policy, from generated metadata and runtime status. Confirm whether an admission controller or another operator writes the field.
- Check blast radius. Determine whether the change affects one workload, a namespace, a shared policy, or multiple clusters. A small-looking change to a shared chart or base can have broad consequences.
- Choose the correction path. For an intentional change, update Git and use the normal review path. For an unauthorized or unsafe change, restore the approved revision after confirming that doing so will not remove an emergency mitigation. For a controller failure, fix access, source, dependency, or health problems before forcing a sync.
- Approve and reconcile safely. Use a pull request, change record, or equivalent approval. Prefer a targeted synchronization when the controller supports it, and avoid bulk reconciliation while the scope is uncertain.
- Verify recovery. Confirm the controller reports the expected revision and a healthy or ready state. Then check rollout progress, endpoints, events, logs, metrics, and dependent services.
- Close the loop. Record the cause, decision, commands or interface actions, reviewer, verification evidence, and any follow-up needed to prevent recurrence.
For resource-level inspection, teams commonly use Kubernetes commands such as kubectl get, kubectl describe, and kubectl diff. The exact GitOps commands vary by controller and release. For example, Argo CD users may inspect an application and its diff with the Argo CD CLI, while Flux users typically inspect the relevant custom resources and controller logs. Confirm command flags and permissions in the installed version before using them in production.
Drift can also reveal broader workflow issues. If the difference originated in infrastructure or identity configuration, pair this review with a Terraform plan review checklist or a cloud IAM access review. If a deployment failure is involved, use the CI/CD pipeline troubleshooting guide to inspect the delivery path.
Common mistakes
- Syncing before understanding the difference. A forced reconciliation can erase a legitimate emergency fix or amplify a bad repository change.
- Treating every field as Git-owned. Kubernetes defaults, controllers, operators, and admission systems may legitimately manage particular fields.
- Ignoring generated output. Reviewing source templates without rendering environment-specific output can hide the actual change.
- Confusing sync with health. Applied resources can still fail to schedule, start, serve traffic, or satisfy readiness checks.
- Changing controller settings during an incident without recording them. Suspension, self-healing, exclusions, and retry changes affect future behavior and should be visible to the team.
- Exposing secrets in diagnostics. Redact values and inspect references, checksums, versions, and access logs instead.
- Using broad permissions to solve a narrow failure. Fix the missing resource or narrowly scoped permission, then review the access change. The secrets management workflow can help when credentials or rotation are involved.
- Failing to test rollback. A repository revert is only useful if the controller can retrieve it and the resulting manifests remain compatible with the cluster.
When to revisit
Review this GitOps drift checklist whenever a drift alert fires, a synchronization fails, or an operator makes an emergency cluster change. Revisit it before seasonal planning cycles, cluster migrations, Kubernetes or controller upgrades, repository reorganizations, and access reviews. It is also worth scheduling a twice-yearly check of controller commands, reconciliation behavior, status and health fields, Kubernetes API compatibility, RBAC boundaries, and notification integrations.
Make the review actionable by assigning an owner for each GitOps installation and documenting the authoritative source for applications, infrastructure, secrets, and policy. Keep a small test scenario that covers an intentional repository change, a detected manual edit, a failed dependency, and a safe rollback. After each exercise or real incident, update the runbook with the observed controller version, useful diagnostics, approval path, and verification steps.
Before closing the next drift alert, ask four final questions: What changed? Who owns the desired state? What is the smallest safe correction? How will we prove the cluster recovered? Those answers provide a repeatable path from detection to verified reconciliation without treating every difference as an emergency.