Quartz 4

07 - Automated Backup & Disaster Recovery

4 min read

Automated Backup & Disaster Recovery

Overview

An automated, zero-downtime backup pipeline runs every night at 02:00 UTC. It creates hot SQLite snapshots, encrypts the entire stack with AES-256 using encrypted archive headers, syncs to Google Drive via Rclone, enforces a 30-day retention schedule, and updates the status badge on Homepage.

1. The 5-Phase Backup Pipeline

flowchart TD
    Cron["Cron Job (02:00 UTC: /etc/cron.d/vaultwarden_backup)"] --> Script["/usr/local/bin/backup_stack.sh"]
    
    subgraph Phase1 ["Phase 1: Hot Consistent SQLite Snapshots"]
        Script --> VW_Snap["sqlite3 vw-data/db.sqlite3 '.backup'"]
        Script --> Auth_Snap["sqlite3 authelia/db.sqlite3 '.backup'"]
        Script --> WF_Snap["sqlite3 wealthfolio-data/wealthfolio.db '.backup'"]
    end
    
    subgraph Phase2 ["Phase 2: AES-256 Military Encryption"]
        VW_Snap & Auth_Snap & WF_Snap --> Bundle["Bundle Configs + Keys + Attachments"]
        Bundle --> Encrypt["7z a -pPASSWORD -mhe=on (Header Encrypted)"]
    end
    
    subgraph Phase3 ["Phase 3: Offsite Cloud Sync & Retention"]
        Encrypt --> CloudSync["rclone copy gdrive:vaultwarden-backups"]
        CloudSync --> Prune["rclone delete --min-age 30d (Prune Old Backups)"]
    end
    
    subgraph Phase4 ["Phase 4: Dashboard Telemetry"]
        Prune --> Badge["Write /var/www/assets/backup-status.json"]
        Badge --> Homepage["Live Status on hq.ganeshr.me"]
    end

Why Hot Snapshots (sqlite3 .backup) Matter

A common mistake in homelabs is copying active .sqlite3 database files with cp. Because SQLite operates in Write-Ahead Log (WAL) mode, a live cp command often results in corrupted, unreadable databases. Using sqlite3 <db> ".backup <dest>" creates an atomic, 100% consistent snapshot with zero container downtime.

Encrypted Headers (-mhe=on)

Standard password-protected ZIP or 7z files conceal file contents but leave the names of the files visible. By enabling -mhe=on (encrypt header), an attacker inspecting the .7z file cannot even see what files exist inside the archive.

2. Backup Pipeline Configuration

  • Script Location: /usr/local/bin/backup_stack.sh
  • Cron Schedule: /etc/cron.d/vaultwarden_backup (0 2 * * * root /usr/local/bin/backup_stack.sh)
  • Encryption Password: Stored securely in SECRETS.md and ganesh-hq-bitwarden-import.json.
  • Remote Cloud Destination: gdrive:vaultwarden-backups/
  • Retention Policy: Automatically purges backups older than 30 days.

3. Disaster Recovery Walkthrough (Rebuilding From Scratch in 15 Minutes)

If the Oracle Cloud instance is ever destroyed, terminated, or corrupted, follow this exact step-by-step procedure to recover everything:

Step 1: Provision Fresh VM & Install Prerequisites

Launch a fresh Ubuntu 24.04 LTS instance, SSH into it, and install required tools:

sudo apt update && sudo apt install -y docker.io docker-compose-v2 sqlite3 p7zip-full rclone curl
sudo systemctl enable --now docker

Step 2: Configure Google Drive in Rclone

Authenticate Rclone to Google Drive:

rclone config
# Name: gdrive, Type: drive, Scope: drive

Step 3: Download & Decrypt Latest Backup

# List available backups
rclone ls gdrive:vaultwarden-backups
 
# Download the most recent archive
mkdir -p /tmp/restore
rclone copy gdrive:vaultwarden-backups/<LATEST_BACKUP_NAME>.7z /tmp/restore/
 
# Extract using your AES-256 archive password
cd /tmp/restore
7z x -p"YOUR_ARCHIVE_PASSWORD" <LATEST_BACKUP_NAME>.7z

Step 4: Restore Application Stack

sudo mkdir -p /opt/vaultwarden
sudo cp -r /tmp/restore/vaultwarden /opt/vaultwarden/vw-data
sudo cp -r /tmp/restore/authelia /opt/vaultwarden/authelia
sudo cp -r /tmp/restore/wealthfolio /opt/vaultwarden/wealthfolio-data
sudo cp -r /tmp/restore/configs/* /opt/vaultwarden/
sudo chown -R 1000:1000 /opt/vaultwarden/homepage-config

Step 5: Launch Services

cd /opt/vaultwarden
sudo docker compose up -d

Step 6: Restore Cloudflare Tunnel

Install cloudflared and restore your tunnel token:

curl -L --output cloudflared.deb https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64.deb
sudo dpkg -i cloudflared.deb
sudo mkdir -p /etc/cloudflared
echo "YOUR_TUNNEL_TOKEN" | sudo tee /etc/cloudflared/token
sudo cloudflared service install $(cat /etc/cloudflared/token)
sudo systemctl start cloudflared

All domains (hq, wealth, vault, cockpit, auth) will immediately come back online with full data intact.

4. Daily Operations Cheatsheet

Trigger a Manual Backup Test

To verify the backup pipeline without waiting for 02:00 UTC:

sudo /usr/local/bin/backup_stack.sh

Inspect Stored Backups on Google Drive

rclone ls gdrive:vaultwarden-backups

Check Backup Disk Usage on Cloud

rclone size gdrive:vaultwarden-backups

5. Next Steps & Related Links

Graph View

Backlinks