Why GitOps Actually Works (And When It Doesn't)

This is the honest version of that trade-off, after three years running Argo CD in production: where the model earns its keep, where it fights you, and what to budget for if you want multi-service rollouts to actually be safe.

Key takeaways

• GitOps rests on four principles from the CNCF's OpenGitOps working group: declarative, versioned and immutable, pulled automatically, continuously reconciled.
• The biggest win isn't the automation. It's the audit trail: every production change is a diffable commit with a reviewer attached, and git log beats any dashboard at telling you what changed at 2 a.m.
• Reconciliation controllers can fight autoscalers and anything else that legitimately mutates state outside Git, which quietly reintroduces the manual toil GitOps was supposed to remove.
• Budget real engineering time for sync-wave ordering and health-check tuning. Multi-service rollouts don't become safe by default just because they're declarative.

What GitOps actually is

Strip away the tooling and GitOps is four principles, formalized by the CNCF's OpenGitOps working group. A system managed by GitOps must have its desired state expressed declaratively; that state must be versioned and immutable, "stored in a way that enforces immutability, versioning and retains a complete version history"; software agents must pull the desired state automatically rather than have it pushed to them; and those agents must continuously reconcile, observing actual state and applying the desired state whenever they diverge (OpenGitOps Principles, CNCF).

Argo CD is the concrete implementation of that fourth principle: "Argo CD is implemented as a Kubernetes controller which continuously monitors running applications and compares the current, live state against the desired target state" (Argo CD Documentation). When the two diverge, Argo CD marks the application "OutOfSync" and can resync automatically or wait for a manual trigger, depending on how you've configured that application.

Where it genuinely works

For anything stateless, and for most Kubernetes-native application deployments, the model holds up exactly as advertised. Roll back a bad deploy by reverting a commit. Answer "what's actually running in production" by reading a Git repo instead of running a live query against every cluster. Give an auditor a commit history instead of a shared-screen walkthrough of who remembers changing what.

The biggest win in three years of running this in production hasn't been the automation itself. It's been the audit trail. Every change to production is a diffable commit with a reviewer attached. When something breaks at 2 a.m., git log is faster than any dashboard at telling you what changed and who approved it, because the answer is already sitting in version control instead of scattered across deploy logs, Slack threads, and someone's memory of a manual kubectl command from three weeks ago.

Where it breaks

The promise falls apart wherever "desired state" isn't actually the whole story. A few recurring cases:

• Database migrations. A migration is a one-time imperative action, not a state you can declare and continuously reconcile toward. GitOps tooling can trigger a migration job, but the migration itself lives outside the reconciliation loop and needs its own safety net.
• Feature flags tied to external systems. If a flag's true state lives in a third-party service (a feature-flag SaaS, a database row toggled by a support tool), Git isn't actually the source of truth anymore. Declaring it in Git and letting an external system override it independently is two sources of truth pretending to be one.
• Infrastructure that mutates itself. Horizontal pod autoscalers, cluster autoscalers, and anything that legitimately changes replica counts or resource allocations outside of a human's commit will drift from whatever static value is declared in Git. A reconciliation controller that doesn't know about the autoscaler's intent will fight it, flipping the replica count back and forth as each system tries to enforce its own version of "correct."

Drift-detection tooling flags all of this as configuration that changed out-of-band, but it rarely tells you why. Figuring out whether a given drift is a legitimate autoscaler action or an actual incident takes a human who understands both systems, every time, until you've built explicit exceptions into the reconciliation loop. That exception-building is real engineering work, and skipping it is how teams end up with a GitOps setup that pages someone every time the HPA does its job.

What to actually budget for

Adopting GitOps well means planning time for the parts that don't show up in the pitch:

• Sync-wave ordering. Multi-service rollouts where one service depends on another being healthy first need explicit ordering, not just "everything reconciles eventually." Get this wrong and you'll reconcile a dependent service before its dependency is actually ready.
• Health-check tuning. The default health check for a resource type is often too shallow to catch a genuinely broken deploy. A pod can report "Healthy" to Argo CD while still failing every real request. Writing accurate custom health checks for your actual failure modes is ongoing work, not a one-time setup task.
• Explicit exceptions for anything that mutates itself. Decide up front which resources the reconciliation loop should ignore (via ignore-differences configuration or similar) so autoscalers and other legitimate external mutators don't get fought by the controller.

The honest recommendation

Use GitOps for everything that's genuinely declarative: stateless services, most Kubernetes-native configuration, anything where "what should be running" is a complete description of correctness. Keep a clear, explicit boundary for anything that isn't, migrations, externally-owned state, self-mutating infrastructure, rather than forcing it into the same reconciliation model and then fighting the tool. And budget real engineering time for sync-wave and health-check tuning up front. That's the work that makes multi-service rollouts actually safe, not the fact that the config lives in Git.

Frequently asked questions

Does GitOps replace the need for a CI pipeline? No. CI still builds, tests, and pushes an artifact; GitOps takes over from "here's a new image tag" and handles getting it into the cluster safely. They're complementary, not substitutes for each other.

How do you handle secrets in a GitOps model that requires everything in Git? Don't store plaintext secrets in Git regardless of the GitOps model. Use a sealed-secrets or external-secrets-operator pattern so the encrypted reference lives in Git while the actual secret material stays in a secrets manager, satisfying the declarative-and-versioned principle without exposing anything.

Is Argo CD's automatic sync always the right default? Not for every application. Automatic sync is right for low-risk, easily-reversible services. For anything where a bad automatic sync could be expensive to unwind, manual sync with a deliberate review step is the safer default, even though it trades away some of the automation appeal.