Installation
This section describes the installation process of XDM. The XDM deployment is based on Docker containers, which allows for easy installation and management of the XDM components. It is required to have a container runtime environment installed on the host system, such as Docker. For more information about the requirements, see Requirements. For installation instruction on how to install in container orchestration environments, such as Kubernetes, see Kubernetes installation.
Components of an XDM installation
XDM consists of multiple program modules. These modules are located in the following Docker containers:
-
xdm3-ui - An interactive browser-based user interface that allows end users to work with XDM.
-
xdm3-core - The core component of XDM that is responsible to control the workflow, persist XDM objects, etc.
-
xdm3-dataflow - A service that is responsible to execute XDM tasks.
-
xdm3-neo4j- A Neo4j graph database that is used internally to track dependencies between the objects in XDM.
-
xdm3-db - A PostgreSQL database that is used for persistence storage of objects in XDM.
-
xdm3-grafana - The Grafana monitoring service is used to visualize statistics about XDM, especially its task execution.
The containers xdm3-ui, xdm3-core, xdm3-dataflow, xdm3-neo4j and xdm3-grafana are always required. Starting these containers means starting XDM itself.
The container xdm3-db is required if you want to use the PostgreSQL database that ships with XDM for persistent storage of XDM settings and tasks. If you use a separate PostgreSQL database for this purpose, the container xdm3-db is optional.
After installing and configuring XDM, you can start it by using the docker-compose application (see section Starting and Stopping XDM). Using this command will automatically start all required containers.
| To run XDM, it is necessary to install docker and docker-compose on the host server. See requirements for more information. |
Obtaining and installing XDM
Obtaining an installation package
The main installation package of XDM is the file xdm3-installation.zip.
This installation package is designed for systems that are able to download the required Docker
containers over an internet connection. This is referred to as an online installation.
If you intend to install XDM on a system that cannot download the required Docker images automatically,
you also need the file xdm3-images-installation.zip in addition to the main installation package.
This file contains all Docker required containers, and you can register them manually in Docker.
This is referred to as an offline installation.
One or both of the files will be provided to you by UBS Hainer as an electronic download.
XDM Versioning
XDM versions are provided with tags. Tags are simply labels to manage versions.
The XDM docker images are provided under the tag latest or under a specific version tag.
Version tag:latest
The latest tag represents the newest and most recently published release. For this reason it is
recommended that you use the latest tag. An advantage is that, provided your system has online
access to the UBS Hainer Docker repository, the latest version can be downloaded with a simple docker command:
docker-compose pull
No adjustment in the docker-compose.yml or in other files are necessary.
Version tag:3.YY.WW
The specification of a certain release version is also possible. For this a tag with the pattern 3.YY.WW can be used.
This pattern defines the XDM version, the year, and the calendar week in which it was released, e.g. 3.21.17.
In the week before a new release is published, a pre-release is available for all XDM docker images under
such a version tag. The pre-release is still in the internal testing phase during this time, but it can
still be pulled from the UBS Hainer Docker repository and used.
To do this, the tag for the individual XDM components must be set to refer to the preferred release version in docker-compose.yml.
The required steps are similar to those of a regular update, which is described in the section Online installation .
| A pre-release version, which is available one week before the actual release, should only be installed and used on a test installation or in really urgent circumstances. |
Installing XDM
Depending on your preferred method, follow the instructions under the section Online installation or the section Offline installation. Afterwards, proceed to section The main XDM configuration file.
Online installation
If the system on which you are installing XDM has access to the internet, you can let Docker download the required images automatically from the UBS Hainer Docker repository.
-
Contact XDM Support under xdm-helpdesk@ubs-hainer.com and obtain a login for the UBS Hainer Docker repository.
-
Open a command line shell on the system where XDM should be installed.
-
Create a base directory for your XDM installation, then change into that directory using the
cdcommand. -
Download the latest base installation package
-
Copy the file
xdm3-installation.zipinto the base directory. You can use FTP, SCP, or any other file transfer method that you have available, or you can download the file directly into this directory by using tools such aswgetorcurl. -
Unpack the file xdm3-installation.zip by entering the following command:
unzip xdm3-installation.zip
-
Connect your local Docker environment to the UBS Hainer Docker repository by entering the following command on the command line:
docker login docker.ubs-hainer.com -
When prompted, enter the credentials you received from UBS Hainer. The credentials will be saved for your Docker client.
The required Docker images will be downloaded automatically when XDM is started for the first time.
Offline installation
If the system on which you are installing XDM does not have internet access, you can use the packaged images in the file xdm3-image-installation.zip.
-
Download the latest base installation package and the docker images package
-
Open a command line shell on the system where XDM should be installed.
-
Create a base directory for your XDM installation, then change into that directory using the
cdcommand. -
Copy the files
xdm3-installation.zipandxdm3-images-installation.zipinto the base directory. You can use FTP, SCP, or any other file transfer method that you have available. -
Unpack the files
xdm3-installation.zipandxdm3-images-installation.zipby entering the following commands:unzip xdm3-installation.zip unzip xdm3-images-installation.zip
-
Register the Docker images for all XDM components by using the following commands:
chmod 775 load-image.sh ./load-image.sh
All images are tagged
latestafter registration. This must be considered when using the images indocker-compose.yml, for example.
Next steps
After unpacking the XDM installation package and either downloading or registering the Docker containers that belong to XDM, the next steps in the installation process are:
-
Customize the
docker-compose.ymlfile to suit your needs (see section The main XDM configuration file) -
Add the XDM version 3 license key (see License key)
-
Add the necessary JDBC drivers to
xdm-config/jdbc-driver(See JDBC drivers) -
Configure the PostgreSQL database that is used by the XDM core service (optional, see Database configuration)
-
Start XDM (see section Starting and stopping XDM)
The main XDM configuration file
After unpacking the XDM installation package and either downloading or registering the Docker containers that belong to XDM, you must customize the main XDM configuration file.
This file controls the Docker containers of XDM. The name of the main configuration file is docker-compose.yml.
For a production ready installation, you should customize the provided docker-compose file.
It is recommended to configure SSL encryption and an LDAP server.
In addition, to make sure that all XDM settings and tasks persist when XDM is shut down, you should either configure an external PostgreSQL database for persistence, or change the volume configuration in the Docker container
xdm-db so that it points to a persistent storage location on your local file system.
The following sections use the contents of the included sample configuration file to explain the basic structure and syntax of this file.
The purpose of this chapter is to explain the basic structure and syntax of the file
docker-compose.yml so that you can customize it according to the requirements of your environment.
For a detailed explanation of the file format and all available options, please refer to https://docs.docker.com/compose/
and https://docs.docker.com/.
|
Comments and empty lines
In the file docker-compose.yml, lines that start with the pound sign (#) are treated as comments.
Multi-line comments are not supported.
If a comment spans multiple lines, each of these lines must start with the pound sign.
Empty lines and lines that only consist of space characters are ignored.
Sections and parameters
The file docker-compose.yml is divided into multiple sections, each of which may have zero or more parameters and any number of subsections.
Parameters may have a single value, or a list of values.
Lists of values are specified by writing the list items into separate lines and prefixing them with a minus sign followed by a space character.
In the following example, there is one top section and one subsection. The subsection has three parameters. The first parameter takes a single value, and the second and third parameter take lists of values.
top-section:
sub-section:
parameter-1: value-1
parameter-2:
- list-item-a: value-a
- list-item-b: value-b
parameter-3:
- list-item-x: value-x
- list-item-y: value-y
Indentation
The structure of the file docker-compose.yml is determined by indentation.
Indentation is defined as a zero or more space characters at the start of a line.
To maintain portability, it is recommended to avoid tab characters and use spaces instead.
In the sample configuration file, four spaces are used per level of indentation.
Each node must be indented further than its parent node.
All sibling nodes must use the exact same indentation level.
services:
neo4j:
image: docker.ubs-hainer.com/neo4j:4.2
environment:
[...]
- services
-
Top level node. No spaces at the start of the line.
- neo4j
-
One level of indentation. Four spaces at the start of the line. Direct child node of
services. - image
-
Two levels of indentation. Eight spaces at the start of the line. Direct child node of
neo4j. - environment
-
Two levels of indentation. Eight spaces at the start of the line. Direct child node of
neo4jand sibling node ofimage.
Running XDM as a non-root user
By default, the XDM docker containers must be started under root authorization. If you prefer run XDM under a
different user ID, you must repack the docker containers xdm3-ui, xdm3-core and xdm3-dataflow and adjust the file
docker-compose.yml.
Follow these steps to run XDM as a non-root user:
-
Identify the numeric user and group ID on the server where docker-compose runs by executing the following commands:
id -u <user> id -g <group>
-
Change the file
docker-compose-user.ymlas follows. Replace <user-id> and <group-id> with the user ID and group ID that were identified in the previous step:services: [...] web-ui: image: xdm3-ui ports: - 4280:8080 build: [...] args: UID: <user-id> GID: <group-id> [...] core-server: image: xdm3-core build: [...] args: UID: <user-id> GID: <group-id> [...] dataflow-server: image: xdm3-dataflow build: [...] args: UID: <user-id> GID: <group-id> [...]- image
-
To use the locally created images from the above command, adjust the
imageparameter. - UID/GID
-
Add an
argsblock to the build section of the three services to define the UID and GID. - ports
-
Adjust the port configuration, because the default UI port has changed to 8080.
If the web-ui port is defined inside the docker-compose-user.yml, it needs to be removed from thedocker-compose.yml.
-
Run the following command in order to create new images with the desired user and group ID:
docker-compose --file docker-compose-user.yml build
| Before XDM can run, the local directories must be created on the host operating system. For detailed information see section Volumes and mount points. |
In order to receive updates for XDM, you must rerun the command
docker-compose --file docker-compose-user.yml build
|
Start XDM by entering the following command docker-compose --file docker-compose.yml --file docker-compose-user.yml up
|
Adding parameters
To add a parameter to a section in the file docker-compose.yml, open the file in an editor and locate the specified section.
Add a new line after its location.
Use spaces to create an indentation that is one level deeper than the indentation of the parameter name.
Add the new parameter.
If the parameter is part of a list of values, make sure that it is prefixed with a minus sign and a space.
Example
Assume that you want to add the parameter environment_color=green to the section services → web-ui → environment.
The last item, environment, refers to a parameter that takes a list of items as its value.
Therefore, the new parameter becomes one of the items in this list.
If the current contents of the section web-ui are as follows:
services:
[...]
web-ui:
image: docker.ubs-hainer.com/xdm3-ui:latest
ports:
- '4280:80'
environment:
- TZ=Europe/Berlin
- xdm_core_serverUri=http://core-server:8000/api/
depends_on:
- core-server
[...]
Then, after making above changes, the section should look like this:
services:
[...]
web-ui:
image: docker.ubs-hainer.com/xdm3-ui:latest
ports:
- '4280:80'
environment:
- TZ=Europe/Berlin
- xdm_core_serverUri=http://core-server:8000/api/
- environment_color=green
depends_on:
- core-server
[...]
Volumes and mount points
It is important to understand how programs running inside Docker containers access the file system of your operating system.
By default, there is no persistent storage defined in the sample file docker-compose.yml.
All data will be lost when the containers are removed.
You must add folders to the volume mappings if you want the environment to persist.
Applications that run in a Docker container are isolated from the file system of the host operating system.
They use their own virtual file system instead.
Changes to files and directories that are made by an application inside a container are only visible to that application.
These changes will remain if a container is stopped and then started again.
However, if a container is removed from Docker (using the docker container rm command), its file system is reset to its original state.
Docker can mount local directories from the host operating system to mount points inside the virtual file system of a Docker container. Mounting local directories to the virtual file systems has the following advantages:
-
It allows file system changes to persist when the container is removed.
-
It allows sharing the same directory between multiple different containers.
-
It makes it easier to create, maintain and use central configuration files that are located outside the Docker containers.
The local directories of the host operating system are also referred to as host directories.
The directories to mount must be specified in the section volumes: inside the file
docker-compose.yml.
In the sample configuration file, each container has its own set of named volumes.
The syntax for a volume entry is:
- <host-directory>:<virtual-directory>
To mount a volume for read only access, append the suffix :ro to the entry.
services:
[...]
core-server:
[...]
volumes:
- ./xdm-config:/xdm/config:ro
# - xdm-data:/xdm/data
- config mounting
-
This line makes the local directory
./xdm-configappear under/xdm/configinside the application that is running in the containercore-server. The directory is mounted for read only access. - comment lines
-
Comment lines start with a pound sign (#), these lines will then be ignored.
If the host system is configured with SELinux, it might be necessary to configure the
volume bind mount with the label z.
services:
[...]
core-server:
[...]
volumes:
- ./xdm-config:/xdm/config:z
- ./xdm-data:/xdm/data:z
For more information see the Configure the selinux label section in the docker manual.
Location of the XDM configuration directory
After unpacking the XDM installation file into a local directory called xdm, you will get the following directory structure:
xdm
|- docker-compose.yml
|- JDBCSRV.XMIT.BIN
|- README.txt
|- xdm3-jenkins-plugin.hpi
+- xdm-config
+- jdbc-drivers
Additional files and directories may be present if you downloaded and unpacked the Docker images for an offline installation.
In the sample configuration file, the directory xdm-config is shared between the different Docker containers that XDM uses.
Its purpose is to store the license key, additional configuration files, and JDBC driver files (in the sub-directory
jdbc-drivers) that need to be accessed from multiple different modules of XDM.
Therefore, this directory is also referred to in this manual as the XDM configuration directory.
If you choose to use a different directory as the XDM configuration directory, make sure to change all volume entries
in the file docker-compose.yml that use the mount point /xdm/config to the new location inside your local file system.
Additional directories
XDM uses additional directories in its virtual file system which, by default, are not shared with the host system.
It is possible to add entries to the configuration file docker-compose.yml in order to associate these directories with locations in your host file system.
This allows easy data exchange between the host system, and the Docker environment in which XDM runs.
Doing this is optional. You can change the configuration at any point in time. However, if a virtual directory already contains files at the point in time where you associate it with a directory on the host file system, then the contents of the directory on the host file system will overlay the old contents of virtual directory. You must manually copy the existing data from the virtual directory into the new directory on the host file system to ensure that access to this data is still possible.
Data exchange directory
The virtual directory /xdm/data can be used for general data exchange between the host system and XDM. You can use this directory to provide, among other things:
-
Files that contain lists of start values (for example, lists of customer numbers) for row-level tasks
-
Files that contain additional dictionaries for the purpose of searching for PII
-
Helper files that may be referenced by modification methods
To associate a directory on the host file system with this virtual directory, first create a new directory in your host file system.
Then edit the file docker-compose.yml. Locate section services → core-server → volumes and add a volume entry
for the mount point /xdm/data.
Task working directory
The virtual directory /xdm/tasks is used to store files that are used during the
execution of an XDM task. It is also used by the schema browser in task executions
to display the extracted data from the subset extract. You can associate a directory
on your host file system with this virtual directory in order to have access to the
task work files from your host system.
To associate a directory on the host file system with this virtual directory,
first create a new directory in your host file system. Then edit the file
docker-compose.yml. Locate section services → dataflow-server/core-server → volumes
and add a volume entry for the mount point /xdm/tasks.
IceBox backup directory
The virtual directory /xdm/backups is used to store files that contain IceBox
backups created by XDM. You can associate a directory on your host file system
with this virtual directory in order to have access to the IceBox backups from
your host system.
To associate a directory on the host file system with this virtual directory,
first create a new directory in your host file system. Then edit the file
docker-compose.yml. Locate section services → dataflow-server → volumes
and add a volume entry for the mount point /xdm/backups.
Mapping directory
The virtual directory /xdm/mapping is used to store H2 databases for
the mapping table containers the mapping table containers.
You can associate a directory on your host file system
with this virtual directory in order to have access to the h2 database files from your host system.
To associate a directory on the host file system with this virtual directory,
first create a new directory in your host file system. Then edit the file
docker-compose.yml. Locate section services → dataflow-server/core-server → volumes
and add a volume entry for the mount point /xdm/mapping.
Important notes when managing XDM installation
There are several things that must be kept in mind when installing XDM to provide a stable installation and to prevent any type of data loss.
First XDM should be installed into a persistent directory for example /opt/xdm.
It is also important to make sure that this directory is secured outside the server, so there is always a way
to restore the xdm data in case of server failure.
It should also be checked, if the following mounts are applied correctly:
-
/xdm/tasksin Dataflow and Core -
/xdm/datain Dataflow and Core -
/xdm/backupin Dataflow -
/xdm/configin Dataflow and Core -
File Bridge if it is in use
The directory /xdm should definitely be backed up regularly, including the xdm/backup directory, which is really
important.
There should also always be a dump file of the Admin-Database. So the Admin-Database can always be restored in case
of emergency.
Admin-Database
For the Admin-Database it is important to have the database being persisted from the docker container via docker mount. Also make sure that the Admin-Database is backed up correctly and that the backup can be restored successfully. When backing up the database it is really important to take a look into the backup file to make sure that the backup was successful. The backup file usually contains any sort of error messages, therefore we recommend to always take a look into the backup file. Otherwise, there is a risk that the backup is damaged and that the database can not be restored.
The backup should also not be placed into the xdm directory. It should be secured in a separate directory outside the server to make it less susceptible for server failure and data loss.
Starting and stopping XDM
After installing XDM using one of the methods described above and customizing the configuration file, it can be started and stopped using the following procedure:
-
Use the
cdcommand to change into the base XDM directory, i.e., the directory where the filedocker-compose.ymlis located. -
Start XDM by entering the following command:
docker-compose up -d
The parameter
-dcauses XDM to start in the background so that it will keep running when you exit then shell. -
After starting XDM as a background process, you can use the following command to stop XDM.
docker-compose down
Connecting to the XDM UI
After XDM is started, it is possible to connect to the XDM user interface. By default, there is an administrative account with the following credentials:
user: admin
password: default
Section Login on to XDM describes the steps to log on to the XDM user interface.