GitOps for Infrastructure: ArgoCD Patterns and Pitfalls
GitOps with ArgoCD for Kubernetes: the application-of-applications pattern, secret handling, drift detection, multi-cluster sync and progressive rollouts.
GitOps makes Git the single source of truth for infrastructure state. Instead of running kubectl apply or clicking through dashboards, you commit manifests to a repository and a reconciliation controller—ArgoCD—ensures the cluster matches what's declared in Git. If someone manually changes something in the cluster, ArgoCD detects the drift and corrects it.
This sounds simple, but production GitOps has sharp edges. Secret management, multi-cluster coordination, progressive rollouts, and application dependency ordering all require patterns beyond the basic tutorials.
Application of Applications Pattern
Managing dozens of ArgoCD Applications individually doesn't scale. The App of Apps pattern uses a parent Application that manages child Applications declaratively.
# ❌ Managing each app manually via CLI
# argocd app create frontend --repo https://... --path frontend
# argocd app create backend --repo https://... --path backend
# argocd app create monitoring --repo https://... --path monitoring
# Doesn't scale, no audit trail, no declarative management# ✅ Parent Application that manages all child apps
# root-app/application.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: platform-root
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/org/platform-apps
targetRevision: main
path: apps
destination:
server: https://kubernetes.default.svc
namespace: argocd
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true# apps/frontend.yaml — child Application
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: frontend
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "2"
spec:
project: default
source:
repoURL: https://github.com/org/platform-apps
targetRevision: main
path: manifests/frontend
helm:
valueFiles:
- values.yaml
- values-production.yaml
destination:
server: https://kubernetes.default.svc
namespace: frontend
syncPolicy:
automated:
prune: true
selfHeal: trueSync waves control ordering: infrastructure (wave: 0) deploys before databases (wave: 1) which deploy before applications (wave: 2). This prevents failures from missing dependencies.
Secret Management without Storing Secrets in Git
The fundamental tension in GitOps: everything should be in Git, but secrets cannot be in Git. Several patterns resolve this.
// Approach 1: Sealed Secrets — encrypt secrets for Git storage
interface SealedSecretWorkflow {
steps: string[];
}
const sealedSecretsApproach: SealedSecretWorkflow = {
steps: [
// 1. Create a regular Kubernetes secret
"kubectl create secret generic db-creds --from-literal=password=s3cure --dry-run=client -o yaml > secret.yaml",
// 2. Encrypt it with the cluster's public key
"kubeseal --format yaml < secret.yaml > sealed-secret.yaml",
// 3. Commit the sealed version to Git
"git add sealed-secret.yaml && git commit -m 'Add db credentials'",
// 4. ArgoCD applies it; controller decrypts in-cluster
],
};
// Approach 2: External Secrets Operator — reference from vault
// external-secret.yaml
const externalSecretManifest = `
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: db-credentials
namespace: backend
spec:
refreshInterval: 1h
secretStoreRef:
name: vault-backend
kind: ClusterSecretStore
target:
name: db-credentials
creationPolicy: Owner
data:
- secretKey: username
remoteRef:
key: production/database
property: username
- secretKey: password
remoteRef:
key: production/database
property: password
`;External Secrets Operator is the better long-term pattern. It decouples secret lifecycle from deployment lifecycle—secrets rotate in the vault without any Git commits.
Drift Detection and Self-Healing
ArgoCD continuously compares the desired state (Git) with the actual state (cluster). When they diverge, it can alert or automatically correct.
interface DriftPolicy {
autoSync: boolean;
selfHeal: boolean;
prune: boolean;
allowedDrift: DriftException[];
}
interface DriftException {
group: string;
kind: string;
jsonPointers: string[];
}
// Configure which fields are allowed to drift
// (managed by controllers, not humans)
const driftPolicy: DriftPolicy = {
autoSync: true,
selfHeal: true,
prune: true,
allowedDrift: [
{
// HPA modifies replica count
group: "apps",
kind: "Deployment",
jsonPointers: ["/spec/replicas"],
},
{
// Cert-manager updates TLS secrets
group: "",
kind: "Secret",
jsonPointers: ["/data"],
},
],
};
// Corresponding ArgoCD Application config
const ignoreDifferences = `
spec:
ignoreDifferences:
- group: apps
kind: Deployment
jsonPointers:
- /spec/replicas
- group: ""
kind: Secret
jsonPointers:
- /data/tls.crt
- /data/tls.key
`;Without ignoreDifferences, ArgoCD will constantly fight with the Horizontal Pod Autoscaler—ArgoCD sets replicas to the Git value, HPA immediately changes it, ArgoCD detects drift and resets it. This loop is one of the most common ArgoCD pitfalls.
Multi-Cluster Sync Strategy
Production GitOps usually involves multiple clusters: staging, production, and sometimes regional clusters. The Git repository structure must support environment-specific configuration.
interface MultiClusterConfig {
structure: "directory-per-env" | "branch-per-env" | "overlay";
}
// Recommended: Kustomize overlays
const repoStructure = {
"base/": {
"deployment.yaml": "Base deployment manifest",
"service.yaml": "Base service manifest",
"kustomization.yaml": "Base kustomization",
},
"overlays/staging/": {
"kustomization.yaml": `
bases:
- ../../base
patches:
- replica-count.yaml
configMapGenerator:
- name: app-config
literals:
- LOG_LEVEL=debug
- API_URL=https://api.staging.example.com
`,
},
"overlays/production/": {
"kustomization.yaml": `
bases:
- ../../base
patches:
- replica-count.yaml
- resource-limits.yaml
configMapGenerator:
- name: app-config
literals:
- LOG_LEVEL=warn
- API_URL=https://api.example.com
`,
},
};
// ArgoCD ApplicationSet for multi-cluster
const applicationSet = `
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: backend-service
namespace: argocd
spec:
generators:
- list:
elements:
- cluster: staging
url: https://staging-k8s.internal
overlay: staging
- cluster: production
url: https://prod-k8s.internal
overlay: production
template:
metadata:
name: 'backend-{{cluster}}'
spec:
source:
repoURL: https://github.com/org/platform-apps
targetRevision: main
path: 'manifests/backend/overlays/{{overlay}}'
destination:
server: '{{url}}'
namespace: backend
`;ApplicationSets generate Applications from templates, reducing duplication and ensuring consistency across clusters. Each environment gets the same base manifests with environment-specific overlays.
Progressive Rollout Integration
GitOps with ArgoCD pairs well with Argo Rollouts for progressive delivery. Instead of replacing all pods at once, traffic shifts gradually with automated analysis.
# rollout.yaml — replaces Deployment
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: backend-api
namespace: backend
spec:
replicas: 10
strategy:
canary:
steps:
- setWeight: 5
- pause: { duration: 5m }
- analysis:
templates:
- templateName: success-rate-check
- setWeight: 25
- pause: { duration: 10m }
- analysis:
templates:
- templateName: success-rate-check
- setWeight: 50
- pause: { duration: 10m }
- setWeight: 100
canaryService: backend-api-canary
stableService: backend-api-stable
selector:
matchLabels:
app: backend-api
template:
metadata:
labels:
app: backend-api
spec:
containers:
- name: api
image: registry.example.com/backend:v2.1.0# analysis-template.yaml
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
name: success-rate-check
spec:
metrics:
- name: success-rate
interval: 60s
successCondition: result[0] > 0.99
provider:
prometheus:
address: http://prometheus.monitoring:9090
query: |
sum(rate(http_requests_total{
service="backend-api",
status=~"2.."
}[5m])) /
sum(rate(http_requests_total{
service="backend-api"
}[5m]))A Git commit updates the image tag, ArgoCD syncs the Rollout, and Argo Rollouts progressively shifts traffic. If the analysis template detects degraded success rates, it automatically rolls back—no human intervention needed.
Key Takeaways
GitOps with ArgoCD makes Git the single source of truth for infrastructure, providing audit trails, rollback through git revert, and declarative cluster management. Use the App of Apps pattern with sync waves to manage dozens of applications with proper dependency ordering. Handle secrets through External Secrets Operator or Sealed Secrets—never store plaintext credentials in Git. Configure ignoreDifferences for fields managed by cluster controllers like HPA and cert-manager to prevent reconciliation loops. Structure multi-cluster configurations using Kustomize overlays with ApplicationSets to generate environment-specific applications from templates. Integrate Argo Rollouts for progressive delivery that automatically analyzes canary metrics and rolls back on degradation. The goal is a system where a git revert is your disaster recovery plan—if the current state is broken, reverting the commit restores the last known good state automatically.
