Backup and Restore using Velero with S3-Compatible Storage¶

This guide explains how to configure Velero with any S3-Compatible Storage as the backend storage for performing backup and restore operations in an air-gapped controller environment.

Unlike public S3 services, this setup assumes S3-compatible storage is hosted on a separate Linux server within the same private network, ensuring data locality and offline functionality.

Prerequisites¶

Ensure the following are available before proceeding:

An S3-compatible storage server accessible over the private network
A dedicated bucket in the S3-compatible storage for Velero backups
S3 access credentials (Access Key and Secret Key)
S3 API endpoint: http://<S3_SERVER_IP>:9000
S3 Console UI: http://<S3_SERVER_IP>:9001 (if available)
Network connectivity between the Kubernetes cluster and the S3-compatible storage server

Backup Implementation Procedure¶

Follow the steps below to enable and schedule backup jobs using Velero with any S3-compatible storage.

Note: If backups are already enabled and scheduled, this section can be skipped.

Step 1: Create an S3-Compatible Storage Bucket¶

Skip this step if a backup bucket already exists.

Use your S3-compatible storage client (e.g., MinIO Client mc, AWS CLI, etc.) or the console UI to create the bucket:

# Using MinIO Client (example)
mc alias set mys3 http://<S3_SERVER_IP>:9000 <ACCESS_KEY> <SECRET_KEY>
mc mb mys3/<YOUR_BUCKET_NAME>

# Or access the Console UI at (if available):
http://<S3_SERVER_IP>:9001

Step 2: Verify S3-Compatible Storage Connectivity¶

# List buckets to verify access (example using MinIO Client)
mc ls mys3/

Step 3: Update the `backup_restore` Section in config.yaml¶

Update the config.yaml file as shown below:

backup_restore:
  enabled: true
  restore: false
  schedule: "0 0 * * *"  # Use cron syntax for scheduling (e.g., every day at midnight)
  bucketName: "rafay-core-backup"
  retentionPeriod: "168h0m0s"  # Retain backups for 7 days
  resticEnable: true           # Enable pod volume backups
  snapshotsEnabled: true       # Enable volume snapshots

  # External Blob Storage (S3-Compatible) credentials (base64 encoded)
  externalBlobStorage:
    username: ""  # Base64-encoded Access Key
    password: ""  # Base64-encoded Secret Key
    endpoint: ""  # e.g., http://<S3_SERVER_IP>:9000

Step 4: Enable Backup Support with Velero¶

Run the following command to install controller dependencies with Velero enabled:

sudo radm dependency --config config.yaml

Note: For S3-compatible storage endpoints using self-signed certificates, you may need to patch the backup storage location to skip TLS verification. Run the following command:
kubectl patch backupstoragelocation default -n velero --type merge -p '{"spec":{"config":{"insecureSkipTLSVerify":"true"}}}'

Step 5: Verify Backup Job Execution¶

Use the command below to confirm scheduled backups are being created:

kubectl get backups -n velero

Sample output

NAMESPACE   NAME                                      AGE
velero      velero-rafay-core-backup-20240403100012   22h
velero      velero-rafay-core-backup-20240403110012   21h
velero      velero-rafay-core-backup-20240403120013   20h
velero      velero-rafay-core-backup-20240403130013   19h

Step 6: Check Backup Status¶

Verify backup status and progress:

kubectl describe backup -n velero velero-rafay-core-backup-20240403130013

Sample output (excerpt)

Status:
  Completion Timestamp:  2024-04-04T05:34:10Z
  Expiration:            2024-04-11T05:33:37Z
  Phase:                 Completed
  Progress:
    Items Backed Up:           9037
    Total Items:               9037
  Volume Snapshots Attempted:  13
  Volume Snapshots Completed:  13

Restore Procedure¶

Follow the steps below to restore the controller from an existing backup.

Step 1: Prepare the Controller Package¶

Download and extract the controller tarball
Ensure the config.yaml file is aligned with the previous (backup) configuration
Update the backup_restore section in config.yaml as follows:

backup_restore:
  enabled: true
  restore: true
  bucketName: "rafay-core-backup"
  restoreFolderName: "velero-rafay-core-backup-20250519130015"  # Update with latest backup folder name
  resticEnable: true
  snapshotsEnabled: true

  # External Blob Storage (S3-Compatible) credentials (base64 encoded)
  externalBlobStorage:
    username: ""  # Base64-encoded Access Key
    password: ""  # Base64-encoded Secret Key
    endpoint: ""  # e.g., http://<S3_SERVER_IP>:9000

⚠️ Ensure restoreFolderName matches the backup name to be restored.

Step 2: Initialize the Controller¶

sudo radm init --config config.yaml

Step 3: Deploy the Controller with Restore Enabled¶

sudo radm dependency --config config.yaml
sudo radm application --config config.yaml

After completion, the controller will be restored using the latest Velero backup from the S3-compatible storage.