GitOps interview questions

The short version is: Git holds the desired state for apps and infra, and an agent inside the cluster keeps reality matching that repo. A deploy isn't kubectl from CI — it's a reviewed commit the controller applies.

# commit a manifest change → controller detects → applies → cluster matches Git
git commit -am "bump web to v1.2.3" && git push

I'd boil it down to four things: desired state is declarative, it's versioned in Git, an agent pulls it (CI doesn't push with cluster creds), and reconciliation runs continuously so drift gets caught and fixed.

declarative   # YAML/Helm/Kustomize in Git
versioned     # every change is a commit
pulled        # agent inside the cluster
reconciled    # level-based loop, not one-shot apply

Desired state is whatever Git — or a chart/OCI artifact — says should exist. Observed state is what's actually on the API server right now. Reconciliation is just closing that gap until they match.

argocd app get web          # Sync / Health
argocd app diff web         # desired (Git) vs live

It's the object that tells the controller 'watch this Git/OCI path and keep that destination cluster/namespace matching it.' Argo calls it an Application; Flux usually pairs a source with a Kustomization or HelmRelease.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata: { name: web }
spec:
  source: { repoURL: https://git/org/cfg, path: apps/web }
  destination: { server: https://kubernetes.default.svc, namespace: web }

Sync is the reconcile step — apply what's in Git so the live cluster matches. OutOfSync means Git and cluster disagree. Healthy is separate: Synced just means the YAML matches, not that the app is happy.

argocd app get web
# Sync: Synced | OutOfSync
# Health: Healthy | Degraded | Progressing

Drift is when someone — or something — changed the cluster and it no longer matches Git. The reconciler notices and either alerts or self-heals by re-applying the desired state.

argocd app diff web
# or: kubectl edit deploy/web  # then watch controller revert if auto-sync

CI still does the heavy lifting I'd expect: build, test, push images, and open or update the config commit. What it shouldn't need is prod kubeconfig. The in-cluster agent stays the continuous applier.

docker push registry/web:$SHA
# update overlays/dev image tag in config repo
# Argo/Flux applies — CI never kubectl apply -f to prod

Push means CI holds kubeconfig and runs kubectl or Helm against the cluster. Pull means a controller inside the cluster watches Git and applies changes. I'd rather keep cluster credentials in the cluster, and I get continuous drift correction for free.

With push, every pipeline and every environment ends up holding cluster creds — a leaked CI token is basically cluster access. Pull keeps write access inside each cluster, makes every deploy a Git change, and turns reconcile into ongoing self-heal instead of a one-shot apply. That's why I push images from CI but let Argo or Flux own the apply.

kustomize edit set image app=app:v2
git commit -am "promote app:v2" && git push
# agent reconciles — CI never ran kubectl against prod

I'd describe it as the controller periodically comparing desired vs observed and converging them. It's level-based, not event-based — so a missed webhook or a hand-deleted resource still gets fixed on the next sync.

argocd app sync web
# Flux:
flux reconcile kustomization apps --with-source

Drift is any live mutation that diverges from Git. With self-heal on, the next reconcile reverts it. Without it, you see OutOfSync and you get to investigate before anything rewrites the cluster.

argocd app diff web
argocd app set web --self-heal --auto-prune

Sync answers 'does live match Git?' — Synced or OutOfSync. Health answers 'is it actually working?' — Healthy, Progressing, Degraded. You can be Synced and Degraded: the YAML is right, the app is crash-looping.

argocd app get web
# Sync: Synced | Health: Degraded  ← investigate pods/events, not Git

Prune deletes live resources that aren't in Git anymore. Without it, GitOps only creates and updates — remove a manifest and the orphan sticks around in the cluster.

syncPolicy:
  automated: { prune: true, selfHeal: true }
# Flux: spec.prune: true on Kustomization

Tags like latest — or even v1.2.3 — can move under you. A digest is an immutable content address. Pinning @sha256:… means desired state is exact, reproducible, and plays nicer with signature verification.

images:
  - name: app
    digest: sha256:abc123…

It's a CRD that pairs a source — repo path, Helm chart, or OCI artifact at a revision — with a destination cluster/namespace and a sync policy. Argo renders the manifests and reports Sync plus Health.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata: { name: web }
spec:
  source:
    repoURL: https://github.com/org/config
    path: apps/web/overlays/prod
    targetRevision: main
  destination: { namespace: web, server: https://kubernetes.default.svc }
  syncPolicy:
    automated: { prune: true, selfHeal: true }

Flux is more of a toolkit. You've got a GitRepository or OCIRepository as the source, then Kustomization or HelmRelease objects that reconcile paths or charts. dependsOn handles ordering, and image-automation can bump tags by committing back to Git.

kind: GitRepository
metadata: { name: config }
spec: { url: https://github.com/org/config, ref: { branch: main } }
---
kind: Kustomization
metadata: { name: web }
spec:
  path: ./apps/web
  prune: true
  sourceRef: { kind: GitRepository, name: config }

I'd say both are solid CNCF-graduated GitOps engines. Argo is app-centric with a strong UI, Projects, and SSO — great when the platform team wants visibility and multi-tenancy in one product. Flux is lean and CRD-native; teams that live in Git and the CLI often prefer it. I pick on UI needs, tenancy model, and what the ops team already knows — not on 'which is more GitOps.'

Argo's Application CR owns source, destination, and sync policy in one object, and the UI is something non-platform engineers will actually open. Flux splits Source → Kustomization/HelmRelease → ImagePolicy, which maps cleanly to Git and doesn't depend on a central UI — but onboarding usually needs more YAML literacy. ApplicationSets and Flux generators solve the same fan-out problem differently. Neither replaces progressive delivery; you still layer Rollouts or Flagger. Credential model, RBAC, and how you bootstrap the first controller matter more than feature bingo.

# Prefer Argo when: SSO + UI + AppProjects multi-tenancy matter
# Prefer Flux when: Git-native CRDs, OCI sources, image automation compose better
# Always: config in Git, pull reconcile, prune + self-heal, no kubectl from CI to prod

Interviewer often follows with: If you had to migrate from one to the other, how would you avoid dual-writing forever?

App-of-apps is a root Application that syncs child Applications — one entry point to bootstrap a cluster. ApplicationSets generate those children from generators (list, git directories, clusters), so onboarding an app or cluster is data instead of hand-written YAML.

App-of-apps is perfect for bootstrap and a small, stable tree. It gets painful once you have dozens of apps times environments times clusters — every new destination is another Application commit. ApplicationSets (or Flux generators) template Applications from a generator: a Git folder per app, a cluster list, or a matrix. The root still exists, but the explosion of children is generated and pruned when the generator input disappears. I'd still keep clear ownership via AppProjects or Flux namespaces, and the generator inputs themselves live in Git.

kind: ApplicationSet
spec:
  generators:
    - clusters: {}
  template:
    metadata: { name: 'web-{{name}}' }
    spec:
      source: { repoURL: https://github.com/org/config, path: apps/web }
      destination: { name: '{{name}}', namespace: web }

Interviewer often follows with: How do you keep a bad generator change from wiping every child app?

Waves are annotations that order resources so CRDs and namespaces land before workloads. Hooks — PreSync, PostSync, SyncFail — run Jobs for migrations or smoke tests around the sync. Waves order things; hooks add lifecycle steps.

Without waves, Argo can apply a Deployment before its CRD or operator exists and thrash. Negative waves run first — namespaces, CRDs, operators — zero is the default, positive waves later for apps and NetworkPolicies that select workloads. Hooks are separate Jobs annotated as PreSync for migrate, PostSync for smoke, SyncFail for alert. A failed hook can block the sync, so you want backoff, TTL, and idempotent Jobs. In Flux the equivalent is dependsOn between Kustomizations plus healthChecks — same idea, different API.

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
# Job runs migrate; only then wave 0 Deployments sync

Interviewer often follows with: What breaks if a PreSync migration isn't backward-compatible with the old pods still running?

An image updater watches the registry and commits the new tag or digest back to the config repo. The reconciler applies that commit. The cluster still only changes because Git changed — CI never ran kubectl.

kind: ImagePolicy
spec:
  imageRepositoryRef: { name: app }
  policy: { semver: { range: ">=1.0.0" } }
# image-automation controller opens/commits the bump

AppProjects constrain which source repos, destinations, and cluster-scoped resources an Application can use. They're the tenancy boundary so team A can't sync arbitrary YAML into team B's namespaces.

kind: AppProject
metadata: { name: team-a }
spec:
  sourceRepos: ['https://github.com/org/config.git']
  destinations:
    - { namespace: 'team-a-*', server: https://kubernetes.default.svc }

I'd use overlays when I'm patching a shared base with small env deltas. Helm values fit parameterized charts and third-party apps. A lot of GitOps setups use Helm for vendors and Kustomize for first-party apps, or wrap Helm with Kustomize.

# overlays/prod/kustomization.yaml
resources: ['../../base']
images: [{ name: app, digest: sha256:… }]

I model each environment as a path or overlay — occasionally a branch, but overlays are usually cleaner. Promotion is a PR that bumps the target env's image digest or chart version. Humans review the Git change; the agent applies it.

cd overlays/prod && kustomize edit set image app=app@sha256:abc…
git commit -am "promote app@sha256:abc to prod"

Most teams I work with keep app code and cluster config in separate repos so a deploy is a config commit and app CI never needs cluster credentials. Inside the config repo, per-env overlays sharing a base keep things DRY. Separate repos buy stronger RBAC boundaries, but you pay in duplication.

The split that matters is code vs desired-state config. App pipelines build and push images — and preferably sign and attest — then a second change updates the config repo. Mono-repo config with overlays for dev/stg/prod is simplest for shared bases and CODEOWNERS. Repo-per-team or repo-per-env helps when blast radius and write access need hard isolation. Branch-per-env usually turns into merge hell and drift. My default pitch: separate config repo, overlays for envs, promote by digest not mutable tags, CODEOWNERS on prod paths.

config/
  apps/web/base/
  apps/web/overlays/{dev,stg,prod}/
  platform/                 # controllers, policies
# CODEOWNERS: overlays/prod → @platform-oncall

Interviewer often follows with: How do you stop a developer from merging a prod overlay change without platform review?

I never commit plaintext. Sealed Secrets or SOPS keep ciphertext in Git; External Secrets Operator keeps only a reference in Git and pulls the value from Vault or cloud SM at runtime. If we already have a central secrets platform, I'd lean ESO.

Sealed Secrets encrypts to a cluster-scoped public key and only the controller decrypts — fine for small setups, painful once you're juggling keys across clusters. SOPS encrypts files with age or KMS before commit; it works across tools but key access has to be gated. With ESO, an ExternalSecret CR references Vault or a cloud secrets manager, the Secret gets materialized in-cluster and refreshed, Git stays free of secret material, and rotation lives upstream — though etcd then holds the synced Secret, so encrypt etcd or use ephemeral projection where you can. Pattern I like: GitOps owns the ExternalSecret YAML; Vault owns the value and the audit trail.

# Sealed Secrets
kubeseal < secret.yaml > sealed-secret.yaml && git add sealed-secret.yaml
# ESO — Git holds a reference only
kind: ExternalSecret
spec:
  secretStoreRef: { name: vault, kind: ClusterSecretStore }
  target: { name: db }
  data: [{ secretKey: password, remoteRef: { key: apps/web/db } }]

Interviewer often follows with: How do you rotate a secret without a downtime-causing reconcile race?

I treat the config repo like production: branch protection, required reviews, CODEOWNERS on prod overlays, signed commits if we need them, plus Argo AppProjects or Flux RBAC so a team's Application can only sync to their namespaces and allowed sources.

There are two planes — Git RBAC and cluster RBAC. On Git: protect main, require reviews for overlays/prod, restrict who can push to the repo the controller trusts, and prefer deploy keys or GitHub App tokens scoped read-only for the controller. On the cluster: the controller ServiceAccount should be the only principal that can mutate GitOps-managed namespaces; developers get get/list/watch and maybe exec, not edit. Argo AppProjects constrain source repos, destinations, and cluster resources. Without both planes, either anyone can PR-bomb prod or anyone can kubectl-edit around Git.

kind: AppProject
metadata: { name: team-a }
spec:
  sourceRepos: ['https://github.com/org/config.git']
  destinations:
    - { namespace: 'team-a-*', server: https://kubernetes.default.svc }
  clusterResourceWhitelist: []   # no cluster-scoped by default

Interviewer often follows with: How do you allow an emergency hotfix without turning off all the controls permanently?

Install the GitOps controller and credentials to read the config repo, apply a root app-of-apps or Flux Kustomization pointing at the platform path, then let it reconcile everything else — CRDs, policies, apps. Stateful data still needs its own restore plan.

argocd app create root --repo https://github.com/org/config \
  --path clusters/prod --dest-namespace argocd --sync-policy automated

I'd keep GitOps owning the desired manifests, but hand traffic shifting to Argo Rollouts or Flagger. They canary or blue-green against Prometheus metrics, auto-promote or roll back, while Git still records the intended version. GitOps deploys the revision; the rollout controller owns the cutover.

A plain Deployment rolling update is basically all-or-nothing once maxUnavailable allows it. Progressive delivery adds analysis templates — error rate, p99 — at each weight step. Blue-green keeps two full stacks and flips a Service or Ingress; canary shifts a percentage via mesh, Ingress weights, or Rollouts' proxies. On failure the controller reverts traffic without necessarily reverting Git — then you fix Git so desired state stays honest. When I'm explaining blue-green, I sketch the cutover out loud.

# Rollouts AnalysisTemplate queries Prometheus error ratio
# Flagger: progressDeadlineSeconds + metric thresholds
# Git still pins image digest; controller only shifts weight

Interviewer often follows with: If analysis passes but business KPIs tank, how do you fold those signals in?

I'd use a management cluster — or per-cluster agents — with ApplicationSet cluster generators or Flux multitenancy so one template fans out per registered cluster, with per-cluster overlays or Helm values. Label clusters by env, region, team, and constrain Projects so teams only sync their scope.

Common patterns: a hub Argo registering spoke clusters via cluster secrets — one control plane and a centralized UI, but a big blast radius if the hub is compromised; instance-per-cluster Flux or Argo — better isolation, harder fleet visibility; ApplicationSet with a cluster generator or git files generator for region overlays. Per-cluster differences belong in values files or cluster-labeled overlays, not forked repos. For secrets I'd rather use ESO with per-cluster SecretStores than replicate Sealed Secrets keys. Progressive delivery and policy like Gatekeeper or Kyverno should ship as platform layer on every cluster, not bolted on per app.

generators:
  - clusters:
      selector:
        matchLabels: { env: prod }
template:
  spec:
    source:
      path: 'apps/web/overlays/{{metadata.labels.region}}'
    destination: { name: '{{name}}', namespace: web }

Interviewer often follows with: How would you do a controlled region-by-region rollout instead of syncing every prod cluster at once?

I'd start with app status and conditions, then diff live vs desired. After that I'm hunting a failing hook, an immutable field fight, a missing CRD, or an RBAC-denied apply. Fix the blocker first — prune or replace only once I understand the diff.

What I see most: PreSync Job crashlooping, someone trying to change an immutable selector, CRD not in an earlier wave, Application pointed at the wrong revision, compareOptions ignoring (or not ignoring) server-side fields, admission webhooks rejecting applies, or two Applications fighting over the same resources. In Flux I'd check Ready on the Kustomization, flux logs, and dependency blockers. I never reach for --force first — I need to know whether Git or live is wrong. If someone kubectl-edited, either commit the intentional change or turn self-heal back on after I've confirmed.

argocd app get web
argocd app diff web
argocd app history web
kubectl -n argocd logs -l app.kubernetes.io/name=argocd-application-controller --tail=100

Interviewer often follows with: When is replace or force justified versus just fixing the manifest?

With self-heal, the next reconcile reverts them. Without it, the app goes OutOfSync. Prevention is RBAC and admission so humans can't mutate managed namespaces, plus a culture that changes go through Git.

Self-heal treats the cluster as a disposable cache of Git — correct for apps, dangerous if operators 'fix prod' without committing. I'd pair self-heal with tight RoleBindings so developers are read-only, ValidatingAdmissionPolicy or Kyverno to deny mutations without a GitOps exception label, and alerts when OutOfSync lasts more than N minutes. Break-glass is a short-TTL RoleBinding with audit logging. Hotfixes still need to land as Git commits inside the incident window so the next reconcile doesn't undo a real fix.

argocd app set web --self-heal --auto-prune
# RBAC: only argocd-application-controller SA can update deploy/web

Interviewer often follows with: How do you handle CRDs or objects that an operator has to mutate outside Git?

Workloads come back by installing the GitOps agent on a fresh cluster and pointing it at the config repo. I still need a plan for persistent data — Velero, snapshots — and for bootstrap credentials: repo access, decryption keys, secret stores. Git restores desired state, not databases.

My DR runbook looks like: recreate the cluster and control plane; restore or recreate the controller's repo credentials and any SOPS/age/KMS access; apply the root app; wait for the platform wave — CRDs, controllers, policies; restore PVCs and DBs from backups before or as apps come up depending on RPO; then verify Sync, Health, and critical SLOs. Multi-cluster GitOps helps if you can fail traffic to a warm region that's already reconciled. Practice it — untested 'Git is our backup' usually fails on secret-zero and stateful restores. The interview signal is separating declarative cluster state from data-plane backups.

1. new cluster + CNI + storage class
2. install Argo/Flux + repo + KMS access
3. sync platform/ (CRDs, OPA, cert-manager)
4. velero restore / DB snapshot
5. sync apps/ and confirm health

Interviewer often follows with: What's your RPO/RTO if the config repo itself is unavailable?

PR pipelines render and validate manifests — kustomize build, kubeconform, Conftest or Checkov — optionally deploy to an ephemeral or staging cluster with the same overlay pattern, then promote by merging to the prod path. I never point prod at an unreviewed branch.

kustomize build overlays/prod | kubeconform -strict -
kustomize build overlays/prod | conftest test -
# Argo CD PR preview plugins / Application previews where available

When the work is inherently imperative and short-lived — one-off node surgery, interactive debugging — when state can't usefully be declared, or when you need sub-second human control without a commit. I use GitOps for desired cluster state and runbooks or jobs for emergencies, then encode the lasting fix in Git.

GitOps struggles with careful database cutovers, firmware/BIOS, secrets you won't put even as ciphertext in Git without mature ESO, ultra-high-churn job systems where every commit is noise, and environments where Git latency or review exceeds incident needs. Mature teams allow break-glass imperative access with automatic drift alerts and a hard requirement to commit the end state. Saying 'GitOps everywhere' without naming those exceptions is a red flag in a senior interview.

# incident: scale manually
kubectl -n web scale deploy/api --replicas=20
# within N minutes: commit replicas (or HPA) so Git matches live

Interviewer often follows with: How do you reconcile GitOps with Helm charts that create random-named resources each release?

I'd make ownership exclusive: one Application or Kustomization per resource set, non-overlapping paths, and prune only on the true owner. Shared platform resources belong in a platform app — product apps consume them, they don't re-declare them.

Dual ownership shows up as perpetual OutOfSync, thrashing applies, and surprise deletes when one side prunes. Fix it by splitting paths cleanly, using Argo's resource tracking annotations, preferring server-side apply with clear field managers, and forbidding wildcard apps that sync overlapping directories. ApplicationSets should generate disjoint destinations. In Flux, dependsOn expresses order without two reconcilers writing the same object. I'd add a CI check that fails if the same GVK/name appears in two synced paths.

# render all apps and assert unique namespace/name/gvk tuples
kustomize build apps/web | kubeconform -
# Argo: check resource annotations for application ownership

Interviewer often follows with: How should CRDs be owned when both platform and app charts vendor them?

Least-privilege read access for the controller — deploy key or GitHub App limited to the config repo — tokens in a sealed or ESO-managed secret, rotate on a schedule, and never reuse a human PAT with org-wide scope.

The controller identity is high value: read access to desired state can reveal infra topology; write access for image automation can push malicious commits. I prefer app installations over PATs, contents:read by default, and contents:write only for image-automation bots. If the bot can write, branch protection should still block skipping reviews on prod paths. Audit git access logs. Private Helm or OCI gets separate pull secrets. On compromise: revoke the app or key, rotate, and review recent commits the bot authored.

# Argo repository Secret: type git, url + sshPrivateKey (deploy key read-only)
# Image automation: separate bot identity with write to overlays/dev only

Interviewer often follows with: Should image-automation commit directly to main or open PRs?

Usually server-side defaults, webhook mutations like sidecars, normalized fields, or ignoreDifferences gaps. I'd diff carefully, ignore known server-populated fields, and make sure CI renders with the same tools and versions Argo uses.

Classic noise: cluster-added caBundle, defaulted clusterIP, Istio or Linkerd injecting containers, HPA fighting replicas, kubectl last-applied annotations. Argo's ignoreDifferences and RespectIgnoreDifferences matter; so do Flux SSA strategies. Align Helm and Kustomize versions between CI and the controller. Either write desired state that expects the mutation, or use policy that forbids surprise mutation in GitOps namespaces.

spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers: ['/spec/replicas']  # when HPA owns replicas

Interviewer often follows with: When is ignoring replicas the wrong fix?

I'd restrict automation to non-prod paths or PRs, require human or semver gates for prod, and separate the bot identity so it can't skip reviews on production overlays.

Continuous latest or :sha tags into prod create change fatigue and accidental deploys of broken builds. Pattern that works: automate overlays/dev, open PRs for staging, promote to prod via merge with CODEOWNERS and tests. Pin digests in prod. Rate-limit the bot and ignore build-metadata-only noise. If you're doing write-back, scope the GitHub App to contents:write only on allowed paths. Treat image automation as a promotion policy problem, not a convenience toggle.

# ImagePolicy → writes overlays/dev only
# prod overlay: digest pinned; promote via PR from staging

Interviewer often follows with: How do you keep the bot from merging its own prod PRs?

If users are hurting, I'd manually abort or promote based on alternate signals, restore metrics permissions or ServiceMonitors, and never leave a half-shifted canary overnight without an owner.

Flagger and Rollouts depend on Prometheus or webhooks. Failures are usually wrong metrics namespace, missing RBAC, histogram vs counter mismatch, or analysis windows that are too short. Break-glass: abort to stable, or promote only with explicit incident-commander approval. Lasting fixes: synthetic checks as backup analysis, alert on AnalysisRun Error, and game-day the failure mode. When I'm aborting I think about traffic cutover the same way I do blue-green. Owning the analysis stack is part of progressive delivery — not an afterthought.

kubectl argo rollouts abort web
kubectl argo rollouts get rollouts web
# fix ServiceMonitor / RBAC; re-run canary in business hours

Interviewer often follows with: When is automatic promote-on-analysis-timeout dangerous?

I'd pause the ApplicationSet or auto-sync immediately, fix the generator template with a dry-run preview, and add CI that renders generators and asserts destination/path invariants before merge.

ApplicationSets multiply mistakes. Controls that help: pause reconciliation annotations, sync windows, progressive sync, and PR-rendered previews of generated Applications. Assert that prod destinations only mount overlays/prod, that cluster labels gate generators, and that ApplicationSet changes need platform review. Prefer allow-lists over globbing every cluster. After the incident, audit what synced during the window and revert commits. Pause plus generator testing plus destination invariants is what I want to hear in an expert interview.

kubectl annotate applicationset apps argocd.argoproj.io/refresh=false
# or delete bad generated Apps after pausing auto-sync
# CI: render ApplicationSet and grep -L 'overlays/prod' for prod clusters

Interviewer often follows with: How do you test ApplicationSet changes against a single canary cluster first?

I'd put CRDs and SecretStores in a platform app with prune careful or disabled for those kinds, keep app Applications consuming ExternalSecrets only, and order sync waves so stores exist before app secrets.

Prune is dangerous across shared platform resources. Pattern: platform wave 0 for CRDs, ESO, SecretStore; app wave 1 for ExternalSecret and Deployments. Finalizers and deletion policies on ExternalSecret matter — Retain vs Delete. SecretStores aren't app-owned; document that. Add sync-wave annotations and health checks. Deletion safety for secret infrastructure is a senior ops topic, and interviewers know it.

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "0"   # SecretStore
# app ExternalSecret: sync-wave "1"

Interviewer often follows with: Should ExternalSecret-managed Secrets be pruned by the app Application?

I'd monitor revision skew between clusters, alert when a cluster lags the target SHA, and use mirrored repos or multi-destination sync with an SLO on reconcile latency. Synced to an old commit isn't safe.

Each cluster's agent tracks a Git revision independently. Network partitions, rate limits, or broken repo credentials cause silent lag. I'd dashboard target vs live revision, time-since-reconcile, and sync error rate per cluster. Don't promote traffic until the DR cluster reports the required SHA and healthy apps. For airgapped DR, OCI artifacts with mirrored registries help. 'Synced' is relative to whatever commit the agent sees — make the commit ID a first-class SLI.

# alert if cluster-b revision != cluster-a revision for > 10m
# argocd app get web -o json | jq .status.sync.revision

Interviewer often follows with: How do you avoid split-brain if both regions can accept writes to the config repo?

I treat Git as the source of truth. Either commit the change, or use a controlled bypass — sync window or pause — with a ticket. Don't leave lasting live-only edits under auto-sync.

Out-of-band changes are drift. Auto-sync will revert them; manual sync without a commit loses the hotfix on the next reconcile. Process I'd use: hotfix branch → PR → sync, or a temporary ignore with an expiry. For true emergencies, disable auto-sync, apply, then reverse-commit live state into Git before re-enabling. GitOps discipline under pressure is the interview signal.

argocd app set web --sync-policy none
kubectl edit deploy/web
# capture live → git commit → argocd app sync web
argocd app set web --sync-policy automated

Interviewer often follows with: When is ignoreDifferences the wrong way to keep a kubectl edit?

I'd halt sync or auto-sync for that app immediately, restore prod from the last good Git revision or backup, and enforce destination allow-lists plus CI checks that prod paths only target prod clusters.

Wrong destination is a top GitOps outage class. Controls: AppProject destination restrictions, separate Argo instances per env, CODEOWNERS on Application manifests, and CI that asserts destination name/server plus path invariants. Prefer cluster name labels over raw API URLs people copy wrong. Post-incident, audit what was pruned or applied. Project RBAC and destination allow-lists beat hoping engineers pick the right context.

# AppProject
spec:
  destinations:
    - name: prod-east
      namespace: 'prod-*'
# CI: fail if path overlays/dev && destination prod

Interviewer often follows with: Why is a separate Argo CD for prod stronger than one instance with many clusters?

I'd inspect wave dependencies and Job health, delete or fix the stuck Job or hook, adjust waves so optional work can't block forever, and add timeouts on PreSync Jobs.

Waves order resources; hooks can block sync. Deadlocks usually come from a Job without a backoff limit, circular waits, or CRDs that aren't in earlier waves. Break-glass: terminate the hook, sync carefully, or disable auto-sync while fixing Git. Lasting fixes: review sync-wave annotations in CI, hook timeouts, and health customizations. Waves are a concurrency protocol — design them with failure modes in mind.

argocd app get web --show-operation
kubectl -n prod get jobs,applications
# fix Job or remove blocking hook; sync-wave: "-1" for CRDs

Interviewer often follows with: Should database migrations be a PreSync hook or a separate pipeline job?

I'd bootstrap minimal Project and RBAC with a one-time break-glass apply or a wave-0 platform app outside the loop, then let app-of-apps manage the rest. Never depend on a child to create its own parent prerequisites.

Classic chicken-egg: root Application needs a Project, but the Project only lives under a child path in Git. Pattern: a minimal bootstrap manifest applied once — or via Terraform — or put Project plus root in the same wave-0 path synced by a privileged installer. Document recovery if Argo itself gets deleted. Separate 'control plane install' from 'fleet desired state.'

# 1) kubectl apply -f bootstrap/appproject-root.yaml
# 2) apply root Application (points at apps/)
# 3) children sync; do not require child to create root Project

Interviewer often follows with: How do you recover if someone deletes the root Application in the cluster?

Analysis used the wrong SLIs or too-narrow a traffic sample. I'd abort or roll back on real user signal, then fix the queries to include business KPIs and multi-region scrapes before the next promote.

Lying analysis looks like success on infra metrics while checkout fails, scraping only canary pods that skip a bad code path, or windows that are too short. I want dual signals and synthetic transactions. Auto promote-on-green without human review for high-risk changes is dangerous. Abort to stable like you would in blue-green. Ownership of analysis templates belongs on the delivery platform.

kubectl argo rollouts abort web
# analysis: add checkout_success + multi-zone Prometheus
# require iterations covering peak traffic

Interviewer often follows with: How do you prevent analysis from succeeding when Prometheus returns empty series?

I'd freeze syncs, restore a known-good commit from reflog or a backup remote, point Argo at a verified revision, and block force-push on protected branches going forward.

GitOps assumes immutable history on release branches. Force-push breaks SHAs controllers track and can resurrect vulnerable manifests or drop commits. Recovery: protect main, use merge queues, recover the commit from another remote or CI artifact mirror, hard-refresh apps to a good SHA. Audit what live clusters applied during the incident. Treat a history rewrite as an integrity incident, not a git inconvenience.

git reflog show origin/main
argocd app sync web --revision <known_good_sha>
# branch protection: deny force push on main

Interviewer often follows with: Would tagging release SHAs in an OCI registry have helped here? How?

Anyone can ship — or bypass Git gates — to production. I'd split projects by env, map SSO groups to least privilege, deny override in prod, and audit policy.csv changes like IAM.

Wide Argo RBAC is cluster-admin adjacent. AppProjects scope repos, destinations, and cluster resources; RBAC scopes actions like get, sync, override, delete. For prod I want sync via automated Git only, or a break-glass group with MFA. Disable exec and override for normal developers. The GitOps UI privilege is part of the security boundary.

p, role:dev, applications, sync, dev-*/*, allow
p, role:dev, applications, sync, prod-*/*, deny
g, [email protected], role:dev

Interviewer often follows with: What does override allow that plain sync does not?

I'd dual-encrypt with old and new keys during the transition, re-encrypt all secrets in Git, update the controllers' keys, then retire the old key. Never rotate keys without a re-encrypt pass.

SOPS/age/KMS rotation needs a transition window. Controllers must hold keys that can decrypt HEAD. Procedure: add the new recipient, sops updatekeys across the repo, sync, remove the old recipient. Keep break-glass decrypt offline. Pair with External Secrets where you can so there's less ciphertext in Git. Secret rotation is a coordinated Git plus controller change.

# .sops.yaml: add new age recipient
sops updatekeys -y apps/**/secret*.yaml
# update flux decryption keys; sync; remove old recipient

Interviewer often follows with: Why might External Secrets reduce how often you re-encrypt Git?

I'd restore CRDs from Git or backup immediately, disable prune on platform CRDs, and put CRDs in a wave-0 platform app that apps depend on but don't own.

Prune plus shared platform resources is dangerous. Own CRDs and operators in a platform Application with prune careful or finalizers. Sync waves: CRDs before CRs; never let an app prune cluster-scoped shared types. Recover from Git history or etcd/Velero. Deletion safety is a first-class GitOps design concern.

# platform Application: CRDs + operators, prune=false for CRDs
# app Applications: only namespaced CRs; sync-wave after platform

Interviewer often follows with: Should applications be allowed to sync cluster-scoped CRDs at all?

I'd pin both chart version and values commit in one change — or vendor the chart — add CI that renders that exact pair, and avoid floating 'latest' chart deps in prod.

Multi-source can drift if auto-sync picks a new chart independently of values. Pin versions in Git, use a single lock commit, or package an OCI artifact that binds chart and values. CI should helm template the pair on every PR. Atomicity of desired state across sources is a design choice you have to own.

# same PR: Chart.yaml version bump + values digest
helm template web chart-1.2.3 -f values-prod.yaml | kubeconform

Interviewer often follows with: How do ApplicationSets make this pinning problem worse if you're not careful?

I'd read the constraint message, fix the manifest in Git or temporarily widen the constraint with change control, then re-sync. I wouldn't kubectl-delete half-applied objects without understanding dependents.

Admission failures leave partial syncs. Prefer dry-run or server-side apply in CI against the same constraints. Sync waves so namespaces and constraints exist first. Break-glass exceptions need tickets. Policy engines are part of the GitOps path, not an afterthought.

kubectl get k8srequiredlabels.constraints.gatekeeper.sh
argocd app sync web --dry-run
# commit label fix; sync again

Interviewer often follows with: How do you test Gatekeeper constraints in CI before merge?

I'd stop writing prod overlays automatically, promote digests via PR with soak time from staging, and rate-limit bots to non-prod paths only.

Continuous prod mutation destroys change management. Automate overlays/dev, PR to staging, merge-promote to prod with CODEOWNERS. Pin digests and record release metadata. Bisect needs discrete commits. Progressive delivery isn't the same as unbounded auto-promote to prod.

# image-automation: write overlays/dev only
# prod: digest pin; promote PR from staging after soak

Interviewer often follows with: How do you keep security patching fast without going back to hourly prod commits?