← Back to blog
Disaster RecoveryTerraformAnsibleKubernetes

Disaster Recovery as Code: Automating Your DR Strategy

· 9 min read

Every organization has a disaster recovery plan. Few have tested it recently. Fewer still can execute it without a single engineer who “knows where everything is.” And almost none can recover in the timeframe their business actually requires.

The problem is not a lack of planning. It is that most DR strategies live in wiki pages, runbooks that have not been updated since 2021, and tribal knowledge distributed across a team that may or may not be available at 3 AM on a Saturday. The solution is the same principle we have applied throughout this series: if it is not in code, it does not exist.

graph LR
    subgraph Primary
        P[Production] --> B[Automated Backups]
    end
    B --> S3[(Offsite Storage)]
    subgraph DR Site
        DR[Standby Infra]
    end
    S3 --> DR
    P -->|Failure| DETECT[Detection]
    DETECT --> DECIDE{Failover?}
    DECIDE -->|Yes| DR
    DECIDE -->|No| FIX[Fix in Place]
    DR --> VERIFY[Verify + Traffic Switch]

RTO and RPO: Start With the Business

Before writing a single line of automation, you need two numbers from your business stakeholders:

  • RTO (Recovery Time Objective): How long can you be down before the business suffers unacceptable damage?
  • RPO (Recovery Point Objective): How much data can you afford to lose? An hour? A day? Zero?

These numbers drive every technical decision. An RPO of zero demands synchronous replication. An RTO of four hours allows for cold standby infrastructure. An RTO of fifteen minutes requires hot standby with automated failover.

Most CTOs we work with discover that different systems have wildly different requirements. Your public marketing site might tolerate hours of downtime. Your payment processing pipeline might tolerate seconds. Map each critical system to its RTO/RPO pair before proceeding.

Kubernetes Backup With Velero

If you are running workloads on Kubernetes — and if you have been following this series since our GitOps foundations post, you likely are — Velero is the standard tool for cluster backup and restore.

Velero backs up Kubernetes resources and persistent volumes to object storage. Here is a basic schedule configuration:

apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: daily-full-backup
  namespace: velero
spec:
  schedule: "0 2 * * *"
  template:
    includedNamespaces:
      - production
      - staging
    storageLocation: default
    volumeSnapshotLocations:
      - default
    ttl: 720h # 30 days
  useOwnerReferencesInBackup: false

For granular control, you can label resources and back up by label selector:

spec:
  template:
    labelSelector:
      matchLabels:
        backup-tier: critical

The key insight is that Velero alone is not a DR strategy. It handles the Kubernetes layer — your deployments, services, configmaps, and persistent volumes. But your infrastructure layer (the clusters themselves, networking, DNS, load balancers) needs its own recovery path. That is where Terraform comes in.

Cross-Region Failover With Terraform

If you have followed our approach to infrastructure as code with Terraform, your entire infrastructure is already defined declaratively. DR becomes a matter of maintaining a second environment definition that can be activated when needed.

A practical pattern uses Terraform workspaces or separate state files for primary and DR regions:

# modules/kubernetes-cluster/variables.tf
variable "region" {
  description = "Cloud region for deployment"
  type        = string
}

variable "is_dr_site" {
  description = "Whether this is the DR site (affects sizing)"
  type        = bool
  default     = false
}

variable "dr_mode" {
  description = "DR mode: cold, warm, or hot"
  type        = string
  default     = "warm"
  validation {
    condition     = contains(["cold", "warm", "hot"], var.dr_mode)
    error_message = "DR mode must be cold, warm, or hot."
  }
}

# Scale down DR site when in warm standby
locals {
  node_count = var.is_dr_site && var.dr_mode == "warm" ? 2 : var.node_count
  node_size  = var.is_dr_site && var.dr_mode == "warm" ? "cpx31" : var.node_size
}

For cross-cloud failover — say, primary on Hetzner with DR on AWS — you abstract your infrastructure behind modules that accept a provider:

# environments/dr/main.tf
module "dr_cluster" {
  source    = "../../modules/kubernetes-cluster"
  providers = { hcloud = hcloud.dr }

  region     = "fsn1"
  is_dr_site = true
  dr_mode    = "warm"
}

# DNS failover via Cloudflare
resource "cloudflare_record" "api" {
  zone_id = var.cloudflare_zone_id
  name    = "api"
  content = var.activate_dr ? module.dr_cluster.ingress_ip : module.primary_cluster.ingress_ip
  type    = "A"
  ttl     = 60 # Low TTL for fast failover
}

The activate_dr variable is your failover switch. In an emergency, a single variable change and terraform apply redirects traffic. In practice, you would automate this behind a CI pipeline trigger or a monitoring-driven webhook.

Database Backup Automation

Databases are where DR gets serious. Kubernetes resources can be redeployed from Git. Stateless services restart cleanly. But your data is irreplaceable.

For PostgreSQL, a robust backup strategy combines logical and physical backups:

# CronJob for automated pg_dump
apiVersion: batch/v1
kind: CronJob
metadata:
  name: postgres-backup
  namespace: production
spec:
  schedule: "0 */6 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: backup
              image: postgres:16
              command:
                - /bin/sh
                - -c
                - |
                  TIMESTAMP=$(date +%Y%m%d-%H%M%S)
                  pg_dump -h $PGHOST -U $PGUSER -Fc $PGDATABASE \
                    | gzip > /backups/${PGDATABASE}-${TIMESTAMP}.dump.gz
                  # Upload to remote storage
                  aws s3 cp /backups/${PGDATABASE}-${TIMESTAMP}.dump.gz \
                    s3://${BACKUP_BUCKET}/postgres/${TIMESTAMP}/
                  # Verify backup integrity
                  pg_restore --list /backups/${PGDATABASE}-${TIMESTAMP}.dump.gz > /dev/null 2>&1
                  if [ $? -ne 0 ]; then
                    echo "BACKUP VERIFICATION FAILED" >&2
                    exit 1
                  fi
              envFrom:
                - secretRef:
                    name: postgres-credentials
                - secretRef:
                    name: backup-s3-credentials
          restartPolicy: OnFailure

Notice the verification step. A backup you have not verified is not a backup. This is a principle worth repeating: untested backups are worthless.

For near-zero RPO, use continuous WAL archiving to object storage with tools like pgBackRest or WAL-G. This lets you perform point-in-time recovery to any moment before the disaster.

DR Testing With Ansible Runbooks

Here is where most organizations fail. They build backup infrastructure and never test it. Ansible runbooks turn DR testing from a quarterly fire drill into a routine automated process.

# playbooks/dr-test.yml
---
- name: Disaster Recovery Test
  hosts: localhost
  vars:
    dr_test_id: "{{ lookup('pipe', 'date +%Y%m%d-%H%M%S') }}"
    slack_webhook: "{{ vault_slack_webhook }}"

  tasks:
    - name: Notify team of DR test start
      uri:
        url: "{{ slack_webhook }}"
        method: POST
        body_format: json
        body:
          text: "🔄 DR Test {{ dr_test_id }} starting. This is a planned test."

    - name: Scale up DR Kubernetes cluster
      command: >
        terraform apply -auto-approve
        -var="dr_mode=hot"
        -var="activate_dr=false"
      args:
        chdir: "{{ playbook_dir }}/../terraform/environments/dr"

    - name: Restore Velero backup to DR cluster
      command: >
        velero restore create dr-test-{{ dr_test_id }}
        --from-backup daily-full-backup-latest
        --kubeconfig {{ dr_kubeconfig }}

    - name: Wait for restore completion
      command: >
        velero restore describe dr-test-{{ dr_test_id }}
        --kubeconfig {{ dr_kubeconfig }}
      register: restore_status
      until: "'Completed' in restore_status.stdout"
      retries: 30
      delay: 60

    - name: Run smoke tests against DR environment
      command: >
        pytest tests/smoke/ --base-url=https://dr-test.internal.example.com
      register: smoke_results

    - name: Record DR test results
      copy:
        content: |
          DR Test ID: {{ dr_test_id }}
          Restore Status: {{ restore_status.stdout }}
          Smoke Tests: {{ 'PASSED' if smoke_results.rc == 0 else 'FAILED' }}
          Timestamp: {{ ansible_date_time.iso8601 }}
        dest: "/var/log/dr-tests/{{ dr_test_id }}.log"

    - name: Scale DR cluster back to warm standby
      command: >
        terraform apply -auto-approve
        -var="dr_mode=warm"
        -var="activate_dr=false"
      args:
        chdir: "{{ playbook_dir }}/../terraform/environments/dr"

Run this monthly. Automatically. Through your CI/CD pipeline. If you built your CI/CD pipelines the way we discussed earlier, adding a scheduled DR test job is straightforward.

The DR Runbook: A Living Document in Git

Your DR runbook should live in Git, right next to your infrastructure code. Not in Confluence. Not in a Google Doc. In the same repository that contains your Terraform modules and Ansible playbooks.

Structure it as a decision tree:

# Disaster Recovery Runbook

## 1. Assess the situation
- [ ] Identify affected systems
- [ ] Determine root cause category: infrastructure / application / data
- [ ] Estimate time to resolve in primary region

## 2. Decision: Failover or fix in place?
- If estimated recovery > RTO → proceed to failover
- If estimated recovery < RTO → fix in place, notify stakeholders

## 3. Failover procedure
- [ ] Run: `gitlab-ci trigger dr-failover --var ACTIVATE=true`
- [ ] Verify DNS propagation: `dig api.example.com`
- [ ] Run smoke tests: `pytest tests/smoke/ --base-url=https://api.example.com`
- [ ] Notify stakeholders of failover completion

## 4. Post-incident
- [ ] Schedule failback within 48 hours
- [ ] Conduct post-mortem
- [ ] Update this runbook with lessons learned

Because it is in Git, every change is tracked. Every update goes through a pull request. The runbook evolves with your infrastructure rather than rotting in a wiki.

Cost Optimization: You Do Not Need Hot Standby Everywhere

A common misconception is that DR must be expensive. It does not have to be. The warm standby pattern — minimal infrastructure kept running, scaled up only during failover — dramatically reduces costs.

With Terraform, your DR environment can run at 20% capacity during normal operations. When failover is triggered, terraform apply scales it to full capacity. The cloud’s elasticity is your friend here.

For non-critical systems, cold standby (infrastructure defined in code but not running) costs almost nothing. You pay only for object storage holding your backups.

Putting It All Together

A complete DR-as-code strategy integrates everything we have discussed across this series:

  1. Infrastructure defined in Terraform, with DR environments as code
  2. Kubernetes workloads backed up with Velero on automated schedules
  3. Databases backed up with verified, encrypted dumps to cross-region storage
  4. Failover automated through CI/CD pipelines, triggered by a single action
  5. Testing automated with Ansible runbooks, executed monthly
  6. Documentation living in Git, reviewed and updated through pull requests

The goal is not to prevent disasters — they will happen. The goal is to make recovery boring. A well-practiced, automated process that any team member can execute under pressure.

If your DR plan requires a specific person to be available, it is not a plan. It is a hope. Code does not take vacations, forget steps under stress, or need to be paged at 3 AM to remember which button to press.

In our next post, we will tie all of these concepts together into an internal developer platform — a self-service layer that gives your developers the power to provision, deploy, and operate their services without waiting on a central operations team.