Skip to main content

Restic: Automated Backup Strategy With Restic In a Dedicated Backup-LXC

Overview

This document outlines the high-availability backup strategy for the Immich photo server. To maximize security and performance, we utilize a Vault-Container architecture. This ensures that the primary application container has no network access to the backup storage, preventing data loss in the event of a service compromise.

Infrastructure Architecture

  • Hypervisor: Proxmox VE 9.x (ProxmoxVE)
  • Primary Application: Immich (LXC ID: 200)
  • Backup Controller: Alpine Vault (LXC ID: 250)
  • Storage Target: TrueNAS, Synology running Rest Server (Vault-NAS)
  • Network Pipe: 10.10.50.0/24 (Direct 2.5GbE Link)
Host / Service Management IP Storage IP Role
ProxmoxVE 192.168.1.50 10.10.50.1 Proxmox Host
Immich-App 192.168.1.200 N/A Immich LXC (Production)
Vault-LXC 192.168.1.250 10.10.50.5 Restic Backup Controller
Vault-NAS 192.168.1.10 10.10.50.10 Restic REST Server

Configuration Steps

1. ZFS Data Mapping (Proxmox Host)

The live Immich data is passed from the ZFS tank to the Vault-LXC as a Read-Only mount point. This prevents the backup container from ever modifying or deleting live production data.

# Executed on ProxmoxVE Host
pct set 250 -mp0 /tank/subvol-200-disk-1,mp=/mnt/source/photos,ro=1

2. Restic REST Server (NAS)

The NAS runs a Restic REST server in Docker to handle incoming data. The --append-only flag can be enabled to prevent any networked client from deleting existing snapshots.

Docker Compose Snippet:

services:
  restic-server:
    image: restic/rest-server:latest
    environment:
      - OPTIONS=--append-only --private-repos
    ports:
      - "8000:8000"
    volumes:
      - /mnt/storage/backups/restic:/data

3. Backup Script (Vault-LXC)

The script below is scheduled via crontab inside LXC 250. It utilizes the high-speed 10.10.50.10 interface for data transfer.

#!/bin/bash
# /root/scripts/backup-titan.sh

export RESTIC_REPOSITORY="rest:http://vault-user:Pass123@10.10.50.10:8000/titan"
export RESTIC_PASSWORD="Encryption_Key_99"

# Perform incremental backup
restic backup /mnt/source/photos --host titan-server --tag "automated"

# Maintenance (Note: Pruning must be done locally on NAS if append-only is active)
restic snapshots

Security Model

  1. Isolation: The Immich container (Titan-App) is restricted from seeing the NAS. Even a total "root" compromise of the web service provides no path to the backups.
  2. Immutability: By using the REST server's --append-only mode, the Vault-LXC can write new data but lacks the authority to "forget" or delete old snapshots.
  3. Integrity: Restic performs cryptographic hashing on every block. Periodic restic check commands ensure no bit-rot has occurred on the NAS disks.

Maintenance & Recovery

  • Daily Check: Verify successful exit codes in /var/log/restic.log.
  • Pruning: Once weekly, a local task on the Vault-NAS runs restic prune to enforce a 7-day retention policy.
  • Restoration: To restore, mount the repository inside Vault-LXC and copy files back to the production subvolume.

How to mount a Proxmox ZFS subvolume to another LXC

This video provides a visual guide on managing Proxmox mount points and subvolumes, which is the foundational step for sharing your data between the production and backup containers.

To add this to your wiki, append the following section. This documentation will help you remember the logic behind the "Vault" architecture and how to maintain it.

Section: Automation & Scheduling

To ensure the "Vault" LXC pulls data and pushes it to the NAS without manual intervention, we utilize the system cron daemon.

1. Crontab Configuration

The backup is scheduled for 03:00 AM daily. This allows the primary application (Immich/Syncthing) to complete its own internal maintenance and database dumps (scheduled at 01:00 and 02:00) before the backup begins.

Command to edit: crontab -e

Crontab Entry:

# m h  dom mon dow   command
0 3 * * * /root/immich-backup.sh >> /var/log/restic-backup.log 2>&1

2. Log Management

Because the script redirects output to /var/log/restic-backup.log, we must ensure the file doesn't consume all disk space over time.

Log Rotation Rule (/etc/logrotate.d/restic): Create this file to keep logs for 7 days:

/var/log/restic-backup.log {
    daily
    rotate 7
    compress
    missingok
    notifempty
}

3. Verification & Health Checks

Automated backups can fail silently if the network or NAS is down.

  • Manual Log Check: Run tail -n 20 /var/log/restic-backup.log to see the last result.
  • Snapshot List: Run restic snapshots to verify that new timestamps are appearing daily.
  • Failed Backup Alerts: It is recommended to add a curl command at the end of the script to a service like Healthchecks.io. If the script doesn't run, you will receive an alert.

4. Security Lockdown

Since the script contains encryption passwords and NAS credentials, access must be restricted to the root user only.

chown root:root /root/immich-backup.sh
chmod 700 /root/immich-backup.sh
Back to top