Backup and restore an XDM instance

Overview

An XDM instance consists of several important parts that should be backed up. During the backup process, XDM should be stopped to avoid inconsistencies. This means that no tasks should be executed and no objects should be created, changed, or deleted.

The backup of an XDM instance consists of two different parts:

  • Configuration files and directories

  • The XDM admin database

For a complete restore of an XDM instance, both parts are required. The configuration files and directories contain information required for the installation itself, task directories, and other important data, such as Icebox generations or mapping table containers.

The admin database contains the data of the XDM configuration objects, such as credentials, database connections, and many other important configurations. For more information see Administration Database Overview.

The admin database is a PostgreSQL database. The backup can be created in different formats using pg_dump.

By default, pg_dump uses COPY statements. This format is usually faster and creates smaller backup files. It is recommended for regular full database backups and complete restores.

Alternatively, the option --inserts can be used. This creates individual INSERT statements for each row. The backup file becomes larger and the restore process is slower, but individual rows are easier to identify, analyze, and restore manually.

This can be useful if only a single database object, for example a task or process definition, has to be inspected or recovered from the backup.

Configuration Files Backup

  • Kubernetes

  • Docker

In Kubernetes, the Helm chart and the related values.yaml file are usually stored in a Git repository. In this case, they do not have to be backed up separately, as long as the repository itself is backed up and the used version is known.

However, configuration files and runtime data that are stored on persistent volumes must be backed up separately. This includes the following directories:

  • xdm-config

  • xdm-backups

  • xdm-data

  • xdm-tasks

  • xdm-mapping

    The exact PVC names and mount paths depend on the Kubernetes installation. They can be checked with the following commands:

    kubectl get pvc --namespace <namespace>
kubectl get pods --namespace <namespace>
kubectl describe pod <xdm-pod-name> --namespace <namespace>

    The recommended way to back up persistent volumes in Kubernetes is to use the backup mechanism provided by the Kubernetes platform, for example volume snapshots or a cluster backup solution.

    An example backup process could look like this:

    flowchart TB subgraph K8S["Kubernetes Cluster"] subgraph APP["Application Namespace: xdm-prod"] DEP["Deployment / StatefulSet"] PVC1["PVC: xdm-config"] PVC2["PVC: xdm-data"] PVC3["PVC: xdm-backups"] PVC4["PVC: xdm-tasks"] PVC5["PVC: xdm-mapping"] DEP --> PVC1 DEP --> PVC2 DEP --> PVC3 DEP --> PVC4 DEP --> PVC5 end subgraph BACKUP["Backup Tool"] TOOL["Velero / Kasten / Trilio"] CSI["CSI Snapshot Integration"] POLICY["Backup Policy: xdm-prod-backup"] POLICY --> TOOL TOOL --> CSI end end subgraph TARGET["External Backup Target"] S3["S3 / MinIO / Azure Blob / GCS"] IMMUTABLE["Immutable Bucket"] ENC["Encryption"] RET["Retention Policies"] S3 --> IMMUTABLE S3 --> ENC S3 --> RET end PVC1 --> CSI PVC2 --> CSI PVC3 --> CSI PVC4 --> CSI PVC5 --> CSI CSI --> S3 POLICY_DETAILS["Schedule: daily at 02:00
    Scope: Namespace xdm-prod
    Retention: 30 days
    Weekly: 12 weeks
    Monthly: 12 months
    Method: CSI Snapshot + Export"] POLICY --> POLICY_DETAILS

In a Docker installation, the configuration files define how and where XDM is started and operated. These files are required to recreate or restore an XDM instance.

The configuration includes configuration files, the license file, JDBC drivers, task directories, XDM icebox generation data, and other runtime data. These files and directories should be included in the file backup process of the server.

The following files and directories should be backed up:

  • xdm-config

  • xdm-backups

  • xdm-data

  • xdm-tasks

  • xdm-mapping

  • docker-compose.yml

  • docker-compose-user.yml (until 3.26.23)

    To back up these configuration files and directories, copy them to another safe location.

    Example:

    mkdir -p ./file-backups/xdm-config-$(date +%Y-%m-%d)
cp -a ./xdm-config ./file-backups/xdm-config-$(date +%Y-%m-%d)/
cp -a ./xdm-backups ./file-backups/xdm-config-$(date +%Y-%m-%d)/
cp -a ./xdm-data ./file-backups/xdm-config-$(date +%Y-%m-%d)/
cp -a ./xdm-tasks ./file-backups/xdm-config-$(date +%Y-%m-%d)/
cp -a ./xdm-mapping ./file-backups/xdm-config-$(date +%Y-%m-%d)/
cp -a ./docker-compose.yml ./file-backups/xdm-config-$(date +%Y-%m-%d)/
cp -a ./docker-compose-user.yml ./file-backups/xdm-config-$(date +%Y-%m-%d)/ 2>/dev/null || true
    The command for docker-compose-user.yml is optional because this file is only used until version 3.26.23.
    The configuration directories xdm-config, xdm-backups, xdm-data, xdm-tasks, xdm-mapping have to be mounted as volume in the docker-compose.yml. Through the mounting the configuration directories are available from the host machine.

    For more information, see volumes and mount points.

Admin Database Backup

The database contains all objects and processes defined in XDM together with the historical information for these objects.

Make sure that XDM is stopped or that no objects are created, changed, or deleted while the pg_dump command is running.
This is only an example of how a backup can be created. This example uses the pg_dump command of the PostgreSQL database. For more information, see the official PostgreSQL pg_dump documentation.

The option --clean creates drop commands to delete existing database objects before they are recreated during restore.

By default, pg_dump uses COPY statements. This format is usually faster and creates smaller backup files. It is recommended for regular full database backups and complete restores.

If individual rows have to be inspected or restored manually, the backup can be created with INSERT statements instead. The option --inserts creates single INSERT statements instead of COPY statements. This makes the backup file larger and slows down the restore process, but it is useful to find, analyze, or restore specific rows manually.

  • Kubernetes

  • Docker

Run the following command to create a database backup using the default COPY format:

kubectl --namespace <namespace> exec -i <postgres-container-name> -- bash -c "export PGPASSWORD='xdm'; pg_dump -U xdm --clean" > xdm-db-backup.sql

To create a backup with INSERT statements, run:

kubectl --namespace <namespace> exec -i <postgres-container-name> -- bash -c "export PGPASSWORD='xdm'; pg_dump -U xdm --clean --inserts" > xdm-db-backup.sql
Verify that the backup file xdm-db-backup.sql has been created on the server where the command was executed.

To prepare a consistent backup, stop XDM and start only the XDM database:

docker compose down
docker compose up -d xdm-db

To create a compressed database backup using the default COPY format, run:

mkdir -p ./admin-backups
docker compose exec -T xdm-db pg_dump -U xdm --clean | gzip > ./admin-backups/xdm-db-$(date +%Y-%m-%d).sql.gz

To create a compressed backup with INSERT statements, run:

mkdir -p ./admin-backups
docker compose exec -T xdm-db pg_dump -U xdm --clean --inserts | gzip > ./admin-backups/xdm-db-$(date +%Y-%m-%d).sql.gz
Verify that the backup file xdm-db-$(date +%Y-%m-%d).sql.gz has been created in the admin-backups directory.

The previous command can be placed into a .sh file. This allows you to create a cron job to automatically create backups with the following example expression:

0 0 * * * /opt/xdm/backup-xdm-db.sh

Configuration Files Restore

This section describes how to restore the XDM configuration files and directories. The admin database is restored separately.

Before restoring files, make sure that XDM is stopped. Existing files and directories may be overwritten during the restore.

The configuration files and directories should match the XDM version that is restored. Restoring files from another version can lead to inconsistent or unsupported configurations.

The following files and directories may have to be restored, depending on the installation type:

  • xdm-config

  • xdm-backups

  • xdm-data

  • xdm-tasks

  • xdm-mapping

  • docker-compose.yml

  • docker-compose-user.yml (until 3.26.23)

  • Kubernetes

  • Docker

For Kubernetes installations, the configuration files and runtime data are usually stored on persistent volumes.

The recommended restore method is to use the restore mechanism provided by the Kubernetes platform, for example volume snapshot restore or a cluster backup solution.

An example restore process could look like this:

flowchart TB subgraph TARGET["External Backup Target"] S3["S3 / MinIO / Azure Blob / GCS"] BACKUPSET["Backup Set
xdm-prod-backup"] META["Kubernetes Resource Metadata"] VOLDATA["PVC / Volume Data"] S3 --> BACKUPSET BACKUPSET --> META BACKUPSET --> VOLDATA end subgraph RESTORETOOL["Backup Tool"] TOOL["Velero / Kasten / Trilio"] RESTOREJOB["Restore Job"] VALIDATE["Validate Backup
Check Namespace / PVC Mapping"] RECREATE["Recreate Kubernetes Resources"] RESTOREPVC["Restore PVCs from Snapshot / Backup"] TOOL --> RESTOREJOB RESTOREJOB --> VALIDATE VALIDATE --> RECREATE VALIDATE --> RESTOREPVC end subgraph K8S["Kubernetes Cluster"] subgraph NS["Target Namespace: xdm-prod / xdm-prod-restore"] DEP["Deployment / StatefulSet"] SVC["Services / ConfigMaps / Secrets"] POD["Restored Pods"] subgraph PVCS["Restored PVCs"] PVC1["PVC xdm-config"] PVC2["PVC xdm-data"] PVC3["PVC xdm-backups"] PVC4["PVC xdm-tasks"] PVC5["PVC xdm-mapping"] end DEP --> POD POD --> PVCS end end META --> TOOL VOLDATA --> TOOL RECREATE --> DEP RECREATE --> SVC RESTOREPVC --> PVCS

For Docker installations, restore the files and directories into the directory where XDM is installed.

Example:

docker compose down

cp -a ./file-backups/<backup-directory>/xdm-config ./
cp -a ./file-backups/<backup-directory>/xdm-backups ./
cp -a ./file-backups/<backup-directory>/xdm-data ./
cp -a ./file-backups/<backup-directory>/xdm-tasks ./
cp -a ./file-backups/<backup-directory>/xdm-mapping ./
cp -a ./file-backups/<backup-directory>/docker-compose.yml ./
cp -a ./file-backups/<backup-directory>/docker-compose-user.yml ./ 2>/dev/null || true
The file docker-compose-user.yml is optional and is only used until version 3.26.23.

After the files have been restored, XDM can be started again or the admin database restore can be performed.

docker compose up -d

Admin Database Restore

  • Kubernetes

  • Docker

Restoring a database backup into XDM may take some time, depending on the size of the database.
During the restore, the XDM application components must be stopped. Only the PostgreSQL database should be running.

Stop the XDM application components. The exact command depends on the Kubernetes installation and the used workload names.

Example:

kubectl scale deployment --all --replicas=0 --namespace <namespace>
Use the following command to identify the relevant XDM pods and workloads: 
kubectl get pods --namespace <namespace>
kubectl get deployments --namespace <namespace>
kubectl get statefulsets --namespace <namespace>

Make sure that the PostgreSQL pod is running:

kubectl get pods --namespace <namespace>

Import the previously created database backup:

cat xdm-db-backup.sql | kubectl --namespace <namespace> exec -i <postgres-pod-name> -- bash -c "export PGPASSWORD='xdm'; psql -U xdm -v ON_ERROR_STOP=1"
The option ON_ERROR_STOP=1 stops the restore if an SQL error occurs.

After the restore has finished successfully, start the XDM application components again.

Example:

kubectl scale deployment --all --replicas=1 --namespace <namespace>
Restoring a database backup into XDM may take some time, depending on the size of the database.
During the restore, the XDM application components must be stopped. Only the PostgreSQL database should be running.

To restore the XDM admin database, stop XDM and start only the XDM database:

docker compose down
docker compose up -d xdm-db

Import the previously created database backup:

gunzip < ./admin-backups/xdm-db-$(date +%Y-%m-%d).sql.gz | docker compose exec -T xdm-db psql -U xdm -v ON_ERROR_STOP=1
The option ON_ERROR_STOP=1 stops the restore if an SQL error occurs.

After the restore has finished successfully, restart the complete XDM environment:

docker compose up -d