Optional parameters

Admin database configuration

Database configuration

XDM uses a PostgreSQL database to store information about its configuration. This includes the defined tasks, the history of task executions, your environment configuration, your user configuration, the cached object-containers, etc.

The default configuration starts a PostgreSQL database used for persistent storage.

If you prefer to use a different PostgreSQL system for your as administration database, you need to perform the following steps:

Kubernetes
Docker

Parameters:

database.extern
database.url
database.schema
database.user
database.password or database.secret
database.upgrade

database.sample

database:
  extern: true
  url: jdbc:postgresql://db-host:5432/xdm
  schema: public
  user: xdm
  password: change_me      # or use 'secret' instead
  # secret: admin-db-secret
  upgrade: automatic
  sample: false

Parameters:

admin.db.url
admin.db.schema
admin.db.user
admin.db.password
admin.db.driver

admin.db.dialect

services:
  core-server:
    environment:
      - admin.db.url=jdbc:postgresql://xdm-db:5432/xdm
      - admin.db.schema=public
      - admin.db.user=xdm
      - admin.db.password=change_me
      - admin.db.driver=org.postgresql.Driver
      - admin.db.dialect=org.hibernate.dialect.PostgreSQL81Dialect
  dataflow-server:
    environment:
      - admin.db.url=jdbc:postgresql://xdm-db:5432/xdm
      - admin.db.schema=public
      - admin.db.user=xdm
      - admin.db.password=change_me
      - admin.db.driver=org.postgresql.Driver
      - admin.db.dialect=org.hibernate.dialect.PostgreSQL81Dialect

admin.db.url (Docker) / url (Kubernetes)

Specifies the JDBC URL of the PostgreSQL database that you want to use as administration database.

In Kubernetes, this option is only required if an external database is used. If the internal database is used, the URL is set to the correct value.

admin.db.schema (Docker) / schema (Kubernetes)

Specifies the schema name. This is set to public by default.

admin.db.user (Docker) / user (Kubernetes)

Specifies the username that is used to access the database.

admin.db.password (Docker) / password (Kubernetes)

Specifies the password that is used to access the database.

For docker, the specified parameters must be added to both services core-server and dataflow-server

extern (Kubernbetes only)

Controls if an internal Pod with a PostgreSQL database is started, or if an external database is used. If an external database is used, XDM will not start its own PostgreSQL database.

secret (Kubernetes only)

As an alternative to storing the database credentials as plaintext, a Kubernetes secret can be used. The name of the secret can be chosen freely. The secret must be of type kubernetes.io/basic-auth, or it must contain the keys 'username' and password'.

hostname (Kubernetes only)

Specifies the host name of the PostgreSQL database that you want to use as administration database. This option is only required if an external database is used.

This is only required if the Grafana service is used.

port (Kubernetes only)

Specifies the port of the PostgreSQL database that you want to use as administration database. This option is only required if an external database is used.

This is only required if the Grafana service is used.

name (Kubernetes only)

Specifies the database name of the PostgreSQL database that you want to use as administration database. This option is only required if an external database is used.

This is only required if the Grafana service is used.

upgrade (Kubernetes only)

XDM requires tables and sequences in the specified database. This option controls if the database objects are created automatically by XDM, or if the user needs to create them manually. If the value is set to automatic XDM will create these objects automatically. The specified user ID must have the permissions to create tables and sequences inside the database.

sample (Kubernetes only)

Controls if an additional PostgreSQL database is started that contains sample data. This sample database is used in the XDM tutorials. By default, the sample database is not started.

Creating database tables

By default, XDM automatically creates the administration tables and sequences in the specified database system if they do not exist yet. XDM also automatically migrates its administration tables to the latest version if it is necessary. Every time XDM starts, it will generate the DDL for all tables and sequences that are missing or changed and store it in the file 'admin-db-diff.sql' inside the data directory. Since this file is newly generated at each start, it only contains the latest changes made to the administration database. With this file, users can inspect the changes made to the database compared to the previously installed version.

The automatic creation and migration of the administration database can be disabled, as described in disable automatic database migration. But in this case, the administrator has to migrate the database himself with the generated 'admin-db-diff.sql' file. This file contains the SQL statements executed by the migration system. It is also possible that the database skips some migration steps that XDM would take at startup, which are too complicated to describe as plain SQL statements.

Database user

The user ID used to create these tables and sequences is specified by the parameter admin.db.user. This parameter is located in the file docker-compose.yml in the section services→core-server→environment.

The specified user ID must have the privilege to create tables and sequences inside the database.

Disable automatic database migration

To disable the automatic creation of the administration tables, edit the file docker-compose.yml, and specify the following line:

Kubernetes
Docker

database:
  upgrade: user

services:
  core-server:
    environment:
      - xdm.core.liquibase.enabled=false

After changing this setting, you must restart the XDM core service.

AI Assistance

The AI Assistance service provides support to users by writing scripts like modification methods in XDM, by using a Large Language Model (LLM). It is an optional service that integrates a LLM in XDM.

To use this feature, the AI_ASSISTANCE component must be licensed.

For Kubernetes, the following values have to be defined:

enabled: Controls if an own pod is started with the AI Assistance service. To use the AI assistance feature of XDM, the LLM must be configured in the values.yml file.

XDM can use a Large Language Model (LLM) for different purposes. The LLM configuration is used to specify the LLM provider that should be used.

base-url: Optional override for the spring.ai.openai.base-url to provide a chat-specific base URL.
api-key-secret: Specifies the API key secret that is used to authenticate against the LLM provider. The secret must provide a key named data, that contains the API key.
organization-id: Optionally, you can specify which organization to use for an API request.
project-id: Optionally, you can specify which project to use for an API request.
model: Optionally, you can specify which model to use for an API request. By default gpt-4o-mini is used.

For Docker, a separate ai-assistance service with spring.ai.openai.api-key is required.

Kubernetes
Docker

Parameters:

ai_assistance.enabled

llm.spring.ai.openai.*

ai_assistance:
  enabled: true

llm:
  spring.ai.openai.base-url: https://api.openai.com/v1
  spring.ai.openai.api-key: llm-api-key-secret    # Kubernetes secret
  spring.ai.openai.organization-id: <organization-id>
  spring.ai.openai.chat.options.model: gpt-4o-mini
  spring.ai.openai.project-id: <project-id>

Parameters:

separate ai-assistance service with spring.ai.openai.api-key

services:
  ai-assistance:
    image: docker.ubs-hainer.com/xdm3-ai-assistance:latest
    environment:
      - TZ=Europe/Berlin
      - spring.ai.openai.api-key=<openai-api-key>

Central Search (Elasticsearch)

XDM’s central search enables full‑text search across configuration objects and execution logs. It is particularly suitable for navigating quickly between objects.

You are not limited to the name of the XDM objects. The central search also allows you to search for the description or certain properties of objects. The content provided in the source code of task stage hooks or workflow templates can also be used in the search.

XDM’s central search is realized with Elasticsearch. Elasticsearch is a search engine that is used to search through XDM configuration objects. XDM uses this service to indexes all configuration objects. If the service is enabled a search field is displayed in the XDM UI. You can instantly search through XDM configuration objects and task execution logs.

To make this possible, an additional service has to be added. This service then indexes the XDM objects to provide the actual search function.

To configure this service, the following parameters must be set:

xdm.core.global.search.enabled (Docker)/elasticsearch.enabled (Kubernetes): Activates the central search in XDM.
xdm.core.global.search.port (Docker only): Specifies the port of the server running XDM’s central search.
xdm.core.global.search.hostname (Docker only): Specifies the host name of the server running XDM’s central search. When using docker, this is the name of the service running the elasticsearch.
xdm.core.global.search.password (Docker)/elasticsearch.password (Kubernetes): Specifies the password for the built-in elastic search user.

Kubernetes
Docker

Parameters:

elasticsearch.enabled

elasticsearch.password

elasticsearch:
  enabled: true
  password: xdm

Parameters:

xdm.core.global.search.enabled
xdm.core.global.search.hostname
xdm.core.global.search.port

xdm.core.global.search.password

Additionally, it is necessary to add the volume elasticsearch-volume.

services:
  core-server:
    environment:
      - xdm.core.global.search.enabled=true
      - xdm.core.global.search.hostname=elasticsearch
      - xdm.core.global.search.port=9200
      - xdm.core.global.search.password=xdm

  elasticsearch:
    image: docker.ubs-hainer.com/elasticsearch:7.17.21
    ports:
      - "9200:9200"
    environment:
      - discovery.type=single-node
      - cluster.name=elasticsearch
      - node.name=es1
      - xpack.security.enabled=true
      - ELASTIC_PASSWORD=xdm
    volumes:
      - elasticsearch-volume:/usr/share/elasticsearch/data:rw

Configuration as Code (CasC)

Synchronizes XDM configuration objects with a Git repository.

XDM can be configured to use a Git repository as source. The repository is used to populate the objects of the XDM installation. XDM continuously monitors the Git repository and automatically applies any changes on the Git files to the XDM installation. In this case XDM objects can not only be created via manual clicks in the UI, but also via the code in Git.

Git configurations are written in YAML files. These can be written manually or generated by an export via the UI. The structure of the file tree, within the Git repository, is up to the user. XDM will parse all YAML files recursively over all directories. The user can create a single YAML file for each object, or combine objects within one file. The dependencies between the YAML files will be resolved by XDM before the files are applied to the XDM installation.

CasC enables the operators of an XDM installation to automatically set up a defined set of objects. Manual synchronization of several XDM installations is not necessary. Test and production instances are kept in a consistent state through the Git configuration.

XDM will only track changes to YAML files. If other files are committed to Git, XDM will not process them. If scripts, e.g. for workflows, hooks, environments, etc. are stored in separate files and are not part of the YAML file, you need to make an additional change to the lastChangedDate field in the corresponding YAML file. Otherwise, XDM will not take this change into account.

A complete overview of how to work with CasC can be found in Configuration as Code Overview.

Configuration

To enable to Configuration as Code the following settings must be specified.

xdm.core.environment-id (Docker) / xdm.environment.id

Defines an id or a name of the XDM environment. This id is used to identify environment-specific properties or cascignore files.

xdm.casc.url (Docker) / url (Kubenetes)

Specifies the URL of the Git repository. This repository must be accessible via HTTP or HTTPS.

xdm.casc.user (Docker) / user (Kubenetes)

The user ID used to clone the Git repository. This setting is not required, if the git clone command does not require authentication.

xdm.casc.password (Docker) / password (Kubenetes)

The password of the Git user. This setting is not required, if no user is required, or the user does not require a password for authentication.

xdm.casc.cron (Docker) / cron (Kubenetes)

A cron expression controls the interval at which the Git repository should be checked for changes. More details of the cron syntax can be found under Scheduling.

xdm.casc.owner (Docker) / owner (Kubenetes)

Specifies an XDM user that will be the owner of all created objects in the XDM installation. These include all objects that are created by the Configuration as Code process. The specified username must exist in the used authentication provider. For example, if the XDM installation uses LDAP as authentication provider, the user must exist in the LDAP system.

This feature cannot be used with an Open ID based authentication provider like KeyCloak. These systems do not allow retrieval of the group membership for a specific user. The roles are required to check whether the respective user is allowed to create or overwrite objects in the XDM installation.

The specified user needs permissions for list and for creating all types of objects.

For more information how to create a list access permission, see the section list access permissions.

If the created objects should be shared with other users, appropriate permission must be granted to the set of users or role. The permission(s) can be specified in the YAML file of the respective object.

xdm.casc.path (Docker) / path (Kubenetes)

Specifies the directory path in the Git repository that should be monitored. This setting is optional. If the setting is not specified, XDM will monitor and apply all changes made in the Git repository. If a path is specified, XDM will only monitor and apply changes made in the specified directory.

xdm.casc.branch (Docker) / branch (Kubenetes)

Specifies the branch that should be monitored. XDM will only apply changes that are committed to that specific branch. By default, this is set to master.

xdm.casc.directory (Docker only)

The local directory, in which XDM will clone the Git repository, and continuously monitor it. By default, the directory /xdm/data/casc is used. The /xdm/data directory should be mounted to an external directory, or persistent volume. Otherwise, a full clone of the Git repository is performed each time the core-server is restarted.

secret (Kubernetes only)

If Git authentication is required, a Kubernetes secret can be used as an alternative to storing the credentials as plaintext. The name of the secret can be chosen freely. The secret must be of type kubernetes.io/basic-auth, or it must contain the keys 'username' and password'. If the user does not require a password for authentication, the 'password' key can be empty but must exist.

The configuration will start a scheduled job inside the XDM server. The job will check for new commits every time it is executed. All objects contained in changed files will be imported. The output of the import operation will be listed in the user interface under System Settings / System Configuration / Scheduled Jobs in the Configuration as Code entry.

All configuration objects in the configuration as code repository must reference objects that are itself part of the configuration as code repository. It is not possible to reference objects, that are part of the XDM instance, but not listed inside the configuration as code repository.

If the import process fails, the errors will be listed in the output of the scheduled job Configuration as Code. The scheduled job will be paused automatically and will not run until the scheduled job is resumed. If the issue is due to a broken object definition in the repository, an additional commit may be pushed to the repository to fix the problem. When the scheduled job is resumed all configuration changes of both commits will be processed.

Properties

Properties are defined in files which have to be located in the root directory. Global properties need to be defined in a file named casc.properties. Environment-dependent properties can be put in a separate file which must be named like casc-<ENVIRONMENT_ID>.properties. The later ones will possibly overwrite global properties.

Property definition

Property files have to contain the properties with one property per line and each property will be written as a key=value entry.

Using properties

YML files can use the properties by writing #{param:PROPERTY_NAME}. During the import of the YML files contained in the CasC repository, all occurrences of a property will be replaced by the concrete value of the property.

When using environment-dependent properties, please ensure that all properties are available for all environments. Either by defining all used properties in all environment-dependent property files or by the definition of default values in the global properties file. Another option is to only import these YAML-files on environments where the used property is defined. Exclusion of files is explained in the chapter Filtering of YML files.

Filtering of YML files

When using the repository for different environments, it’s possible that not every XDM-object is needed in every environment.

For instance, environments can be differentiated by databases they use. Therefore, in an environment that only uses Postgres database connections other connection types should not be imported.

This can be achieved by using cascignore-<ENVIRONMENT_ID>-files, inspired by the .gitignore-file mechanism. It is also possible to define global cascignore-files, which will be used on every import. For details how to define the files to ignore, see the Git documentation on gitignore. Typically, absolute paths are defined in the cascignore-file. Such an entry must be prefixed with

<xdm.casc.directory>/<xdm.casc.path>/<Path to file>

Files matching a pattern in a cascignore-file or cascignore-<ENVIRONMENT_ID>-file will not be imported.

Example

Suppose you have the following structure of your Git repository:

Root
|_ Projects
  |_ Git
    |_ Documentation
    |_ Scripts
    |_ XDM
      |_ CasC
        |_ Connection
          |_ Postgres
            |_ Admin DB.yaml
        |_ Credential
          |_ Admin Credential.yaml
        |_ File
          |_ AFile.yaml

The contents of /Projects/Git/ is managed by Git. Then a valid configuration for CasC could be:

xdm.casc.directory=/Projects/Git
xdm.casc.path=/XDM/CasC

With this configuration XDM monitors the directory /Projects/Git/XDM/CasC/ and its sub-directories for changes. In this example you want to exclude the import of the file AFile.yaml for all environments (globally). Then you have to create a file /Projects/Git/XDM/CasC/cascignore with the following contents:

/Projects/Git/XDM/CasC/File/AFile.yaml

Kubernetes
Docker

Parameters:

casc.enabled
casc.url
casc.user
casc.password or casc.secret
casc.cron
casc.path
casc.branch

casc.owner

casc:
  enabled: true
  url: https://git.example.com/xdm-config.git
  user: git-user
  password: git-password
  # secret: xdm-casc-secret
  cron: 0 */5 * * * ?
  path: /casc
  branch: main
  owner: xdm-admin

Parameters:

xdm.core.environment-id
xdm.casc.url
xdm.casc.user
xdm.casc.password
xdm.casc.cron
xdm.casc.path
xdm.casc.branch
xdm.casc.owner

xdm.casc.directory

services:
  core-server:
    volumes:
      - ./xdm-data:/xdm/data
    environment:
      - xdm.core.environment-id=DEV
      - xdm.casc.url=https://git.example.com/xdm-config.git
      - xdm.casc.user=git-user
      - xdm.casc.password=git-password
      - xdm.casc.cron=0 */5 * * * ?
      - xdm.casc.path=/casc
      - xdm.casc.branch=main
      - xdm.casc.owner=xdm-admin
      - xdm.casc.directory=/xdm/data/casc

Certificate Based Authentication

In order to establish an encrypted connection via SSL, it is necessary to install single certificates or a complete certificate-chain. All installed certificates will be used by the core-server, dataflow-server and xdm-runner (batch parts). With the certificates XDM is able to establish an encrypted connection using a certificate to a database or authentication server.

Installation of certificates

The certificates must be located in a directory /xdm/certs/ which is mounted by XDM.

Kubernetes
Docker (local)

The path is automatically mounted when an entry is made under xdm.certificates.

xdm:
  certificates:
    - <secret-name>

Add mount for /xdm/certs/

services:
  core-server:
    volumes:
       - <your-certificate-folder>:/xdm/certs/
  dataflow-server:
    volumes:
       - <your-certificate-folder>:/xdm/certs/

Replace the variable <your-certificate-folder> in the above with the name of the folder that contains your certificates.

Verify imported certificates

The certificates get imported through the entry point script at the startup of the core-server, dataflow-server and xdm-runner. Check the logs of the respective service to verify that the certificates have been imported successfully. Check for the following log entry:

File '<file-name>' successfully installed as certificate '<certificate-alias>'.

Installation of certificates for microservices

Microservices may also use certificates, e.g, when the connection to the Kafka broker requires encryption. The installation for these services is similar to the other services (like the core-server) by using a volume mount for the certificates.

But there are two additional requirements:

You should use a separate host directory for the volume mount. This avoids conflicts because the microservices use a different method to import the certificates.
An additional file is needed in the same folder as the certificates, which is described below.

This file must be named type and must have the content ca-certificates. Finally, in the environment section of the service a new entry SERVICE_BINDING_ROOT must be added. The complete configuration may look like the following:

environment:
  - SERVICE_BINDING_ROOT=/bindings
volumes:
  - <microservice-certificate-folder>:/bindings/certs

As an example, this is a valid content of the folder which is specified by <microservice-certificate-folder>

$ ls -1
ca.crt
type
$ cat type
ca-certificates
$ cat ca.crt
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----

Serving under a Context Path and using Reverse Proxy

Depending on the environment in which XDM is running, it might be desirable to configure a context path under which the XDM user interface can be accessed.

The context path must start with a forward slash but must not end with a forward slash. It is also possible to configure multiple path elements.

If XDM is installed on the machine testdataserver, uses SSL encryption with the default port, and the context path is set to /xdm, then the user interface can be accessed under the following URL:

https://testdataserver/xdm/

Accessing the user interface will only work if a forward slash is present at the end of the URL.

When using openID authentication, the context path must also be set in the OpenID connect authentication configuration.

Allows XDM to be served under a path prefix (e.g. /xdm) behind a reverse proxy.

Kubernetes
Docker

Parameters:

xdm.environment.contextPath

ingress.* (domain, class, TLS, annotations)

xdm:
  environment:
    contextPath: /xdm

ingress:
  enabled: true
  domain: xdm.example.com
  ingressClassName: nginx
  tls:
    - hosts:
        - xdm.example.com
      secretName: xdm-tls

Parameters:

xdm_context_path (web-ui)

Reverse proxy config (e.g. Nginx) outside of XDM

services:
  web-ui:
    environment:
      - xdm_context_path=/xdm
    ports:
      - "443:443"

# Example Nginx (outside of docker-compose.yml):
# location /xdm/ {
#   proxy_pass https://xdm-ui:443/xdm/;
#   proxy_set_header X-Forwarded-Proto https;
#   ...
# }

Data Type Conversion

By default, scripts which use custom parameters of type boolean or number represent them as strings. For example, to compare a boolean custom parameter, users have to use an expression like:

booleanCustomParameter == 'true'

When activated, these custom parameters now are of the corresponding type and can be treated as such in the scripts. The expression above can now be:

booleanCustomParameter == true

Kubernetes
Docker

Parameters:

xdm.core.data-type-conversion

core:
     environment:
         xdm.core.data-type-conversion: true

dataflow:
     environment:
         xdm.core.data-type-conversion: true

Parameters:

xdm.core.data-type-conversion

services:
  core-server:
    environment:
      - xdm.core.data-type-conversion=true

  dataflow-server:
    environment:
      - xdm.core.data-type-conversion=true

Existing scripts which use custom parameters of type boolean or number, may not work properly after data type conversion has been enabled and should be checked in order to maintain functionality.

Docker Memory Settings

This setting is only possible in Docker environments. For Kubernetes environments no setting is necessary and possible.

Configure memory for task executions

In Docker, each XDM task execution runs as a standalone program that is executed within the dataflow docker container. By default, each execution uses a quarter of the available memory from the docker container. For example, if the dataflow container can use up to 2 GB memory, each task execution will have 500 MB by default.

In some situations this setting is not sufficient, for example if a large RLP extract task needs more than a quarter of the available memory. This setting can be changed via the parameter 'ANT_OPTS'. In this example each task execution would have up to 800 MB memory. Alternatively the memory allocation can be set relatively.

Kubernetes
Docker

No settings possible

Parameters:

ANT_OPTS

services:
  dataflow-server:
    environment:
        # print used environment variables in task stage output
        - XDM_SHOW_PROCESS_DEBUG=true
        # setting absolute memory usage
        - ANT_OPTS=-Xmx800m
        # setting relative memory usage
#       - ANT_OPTS=-XX:MaxRAMPercentage=50

Please keep in mind that the memory of the dataflow server is shared between the dataflow server and each task execution. If you assign too much memory for each task executions it might lead to out of memory errors if too many tasks are executed in parallel.

Refer also to the Docker Compose documentation for more syntax definitions and options.

Define the Event Execution Log Cleanup Process Interval

The executionLogRetentionPeriod setting defines a period of time that controls how long an event execution log should be kept after it is automatically deleted. This setting can be configured for specific events.

The cron expression controls the execution of the process that cleans up the expired event execution logs. It specifies what time the process that deletes old event execution logs. All event execution logs older than the specified executionLogRetentionPeriod will be deleted by this process.

By default, the cleanup process runs every day at midnight. The default cron expression is 0 0 0 * * ?.

xdm.event.cleanup-cron=0 0 0 * * ?

Accessible Tables for Task Execution Report SQL statements

The SQL statement of a task execution report is executed against XDM’s internal database, which might be a potential security risk if all available tables are accessible. Therefore, the property xdm.core.execution-report-table-whitelist defines which tables may be accessed.

If the above property is not explicitly set, only the tables task_execution_table_statistics, task_execution_classification_term_data and classification_term can be accessed by default. If it is set to an empty value, it permits access to all internal tables for task execution reports. Otherwise, a comma separated list of table names will define the accessible tables. If a report SQL statement tries to access more than the allowed tables, the execution step Generate reports won’t generate a report, but instead it will write a warning message to the step log.

To override the property’s default value, it must be defined for the dataflow service. For example, if the user should only be able to access the tables task_execution_table_statistics, file and connection the property must set as follows:

Kubernetes
Docker

Parameters:

xdm.core.execution-report-table-whitelist

deployer:
  configuration:
    environmentVariables:
       xdm.core.execution-report-table-whitelist: task_execution_table_statistics,file,connection

Parameters:

xdm.core.execution-report-table-whitelist

services:
  dataflow-server:
    environment:
      - xdm.core.execution-report-table-whitelist=task_execution_table_statistics,file,connection

Gateway Timeouts

Configuring HTTP Timeouts For Long-Running Operations

After triggering some operations manually in XDM, this operations can run for multiple minutes, for example large exports and imports of configuration or test data. These long-running requests may exceed default HTTP or gateway timeouts in load balancers, reverse proxies, or API gateways. In such cases, the client may receive an error like:

HttpErrorResponse 504 Gateway Timeout

even though the underlying XDM process is still running or completes successfully.

To avoid this, you must configure sufficiently high timeouts along the complete path between the browser and the XDM services. As a rule of thumb, configure at least 3600 seconds (10 minutes) for long-running operations.

The concrete configuration depends on your environment. The following examples illustrate typical setups.

Recommended Timeout Values

For productive environments with long-running XDM operations, the following values are usually sufficient:

Connection timeout: 3600 seconds
Read/response timeout: 3600 seconds
Send/write timeout: 3600 seconds

Ensure that all components in the request path use compatible values, for example:

External load balancer or API gateway
Reverse proxy (e.g. Nginx, Ingress controller)
Web server in front of XDM UI (Nginx inside the xdm3-ui image)
Any web application firewall

If one component has a significantly shorter timeout, requests may still fail with 504 Gateway Timeout.

Nginx Reverse Proxy

The following example shows how to configure generous timeouts in a standalone Nginx reverse proxy in front of XDM:

server {
    listen 443 ssl;
    server_name xdm.example.com;

    ssl_certificate     /etc/nginx/certs/xdm.crt;
    ssl_certificate_key /etc/nginx/certs/xdm.key;

    location /xdm/ {
        proxy_pass http://xdm-web-ui:4280/xdm/;

        # Forward relevant headers
        proxy_set_header Host                $host;
        proxy_set_header X-Real-IP           $remote_addr;
        proxy_set_header X-Forwarded-For     $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto   https;
        proxy_set_header X-Forwarded-Scheme  https;

        # Timeouts for long-running XDM requests (e.g. 3600 seconds)
        proxy_connect_timeout   3600s;
        proxy_send_timeout      3600s;
        proxy_read_timeout      3600s;
        send_timeout            3600s;
    }
}

If you use a context path such as /xdm/, ensure that the proxy_pass and the internal XDM configuration (for example xdm.core.ui.serverUri) are aligned with this path.

Kubernetes Ingress (Nginx Ingress Controller)

If you run XDM on Kubernetes with the Nginx Ingress Controller, you can configure the timeouts via annotations on the Ingress resource:

ingress:
  enabled: true
  domain: testdatenmanagement.xdm.de
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"

XDM UI Container Environment

The XDM UI image uses an internal Nginx server. If you build a custom UI image based on xdm3-ui, you can also adjust Nginx settings (including internal timeouts) in /etc/nginx/nginx.conf or a template such as /etc/nginx/conf.d/default.tmpl. The base image already exposes these files and allows you to modify them in a derived Dockerfile:

FROM docker.ubs-hainer.com/xdm3-ui:latest

# Optional: adjust Nginx configuration (including timeouts)
USER root
RUN sed -i 's/proxy_read_timeout 60s;/proxy_read_timeout 3600s;/' /etc/nginx/nginx.conf
    # Add further timeout settings as required
USER xdm

The concrete Nginx configuration inside the UI container may change between releases. Always verify the current configuration in /etc/nginx/nginx.conf and related templates before applying changes.

Troubleshooting 504 Gateway Timeout Errors

If you still see 504 Gateway Timeout errors after increasing timeouts:

Check proxy and gateway logs for indications of which component is closing the connection.
Verify that all components in the chain use compatible timeout values (no shorter timeout in front of a longer one).
In Kubernetes, verify that the Ingress controller and any cloud load balancer do not enforce additional, lower idle timeouts.

Groovy Script references

By default, our used Groovy script engine uses soft references for generated classes. With soft references, Groovy script classes may be garbage collected under memory pressure and re-created when needed. This helps avoid memory leaks, but it may lead to a MissingMethodException when the script is called many times during a long workflow or task execution because the generated classes may be removed by the garbage collector.

With hard references, generated classes stays in memory as long as the engine is alive. While this could improve performance for long-running workflows or task executions, it also increases the risk of higher memory usage and possible OutOfMemoryError.

Kubernetes
Docker

Parameters:

xdm.script.groovy.hardReference

core:
     environment:
         xdm.script.groovy.hardReference: true

dataflow:
     environment:
         xdm.script.groovy.hardReference: true

Parameters:

xdm.script.groovy.hardReference

Additionally, it is necessary to add the volume elasticsearch-volume.

services:
  core:
   environment:
     - xdm.script.groovy.hardReference=true

  dataflow-server:
    environment:
      - xdm.script.groovy.hardReferences=true

Enabling hard references may increase memory consumption significantly, especially with many or large Groovy scripts.

HTTP settings

Changing the TCP port

By default, the XDM user interface is accessible on TCP port 4280, which is the default port for the HTTP protocol.

To change the TCP port number for the XDM user interface, edit the file docker-compose.yml and locate the section services → web-ui → ports. The value of this parameter is a list of port mappings that contains one entry. This entry is a string, enclosed in apostrophes, that contains the external and the internal port numbers, separated by a colon.

The internal port number is 80 and must not be changed. The default value for the external port number is 4280. You can change the external port number to any value that does not cause conflicts with other services that are running on the same system.

Example

To change the TCP port number for the XDM user interface to 4281, change the entry so that it looks like this:

    web-ui:
        [...]
        ports:
            - '4281:80'

Using SSL encryption

Using Self-Signed Certificates

XDM uses the HTTP protocol both for the interactive web interface and for the internal communication between the different modules of the product. In addition to standard HTTP, it is also possible to use the encrypted HTTPS protocol.

It is recommended to use HTTPS in a productive environment. To set up HTTPS, you need to configure a certificate store (key store) for the server. The following example illustrates how to create a PKCS key store using the Java keytool command:

keytool -genkey -alias xdm3 -storetype PKCS12 -keyalg RSA -keysize 2048 -keystore xdm3-keystore.p12 -validity 3650 -ext "SAN:c=IP:127.0.0.1,DNS:localhost,DNS:dataflow-server,DNS:core-server"

After creating the key store edit the docker-compose.yml file and add the following SSL configuration in core-server → environment and dataflow-server → environment. The value of the parameter ssl.key-store-password must be set to the password that you entered when the key store was created.

server.ssl.key-store=/xdm/config/xdm3-keystore.p12
server.ssl.key-store-password=mypassword
server.ssl.keyStoreType=PKCS12
server.ssl.keyAlias=xdm3
spring.cloud.dataflow.client.skip-ssl-validation=true

server.ssl.key-store: The path to the root key store on the execution server.
server.ssl.key-store-password: The key store password specified during key store creation.
server.ssl.keyStoreType: The type of the key store. PKCS12 is the standard format.
server.ssl.keyAlias: If the key store was created with the keytool command, this must be the value of the -alias parameter. If the key store was created with OpenSSL, then this is the value of the -name parameter in the command.
spring.cloud.dataflow.client.skip-ssl-validation: Using this parameter, the dataflow server will accept any (self-signed) SSL certificate.

The generated file xdm3-keystore.p12 must be copied into the mounted volume /xdm/config.

Using HTTPS for internal communication

To enable the HTTPS protocol for the internal communication between the XDM modules, edit the file docker-compose.yml. You must change the protocol from http to https in three locations:

services → core-server → environment → spring.cloud.dataflow.client.serverUri
services → web_ui → environment → xdm_core_serverUri
services → dataflow_server → environment → xdm_core_serverUri

Using HTTPS for the web interface

The internal web server of the UI image is pre-configured to serve HTTP on port 80 and HTTPS on port 443 using a self-signed certificate.

These ports can be configured using the environment variables http_port and https_port.

To enable the HTTPS protocol for the XDM web interface, edit the file docker-compose.yml and make the following changes:

Under services → web-ui → ports, change the internal port number from 80 to 443 (port 443 is the default port for the HTTPS protocol. If desired, you can use a different port number). The internal port number will be different if you changed https_port.

To replace the default certificate with a certificate matching your internal names, make the following change:

Under services → web-ui → volumes, mount the volume that contains the key and certificate file. The mount point must be /xdm/certificates. You may have to add the section volumes if it does not exist yet.

Example

A complete setup for the section web-ui in the file docker-compose.yml might look as follows:

web-ui:
        image: docker.ubs-hainer.com/xdm3-ui
        ports:
            - '4280:443'
        environment:
            - TZ=Europe/Berlin
            - xdm_core_serverUri=http://core-server:8000/api/
            - https_key_file=/xdm/certificates/xdm.key
            - https_certificate_file=/xdm/certificates/xdm.crt
        depends_on:
            - core-server
        volumes:
            - ./certificates:/xdm/certificates

The local directory ./certificates must contain the key file xdm.key and the certificate file xdm.crt.

Session Cookie Settings For HTTP Deployments

In a standard Docker-based installation XDM will be deployed via HTTP. For security reasons the session cookie will be created secure. This will cause an error, because browsers do not deploy non-secure cookies over HTTP connections.

This can be changed by either configure XDM to use HTTPS (see SSL Encryption for configuring tihus), or configure it to create session cookies without the secure flag.

With this setting, XDM will no longer mark the session cookie as Secure and the browser will send it for HTTP requests as well, allowing the login process to complete successfully.

This property should only be used for HTTP deployments. For productive environments, it is strongly recommended to enable HTTPS and remove this setting so that session cookies are always transmitted using transport encryption.

Kubernetes deployments typically do not require this additional property, because HTTPS termination and cookie handling are usually performed by the ingress controller or a reverse proxy in front of XDM. Only configure server.servlet.session.cookie.secure=false in Kubernetes if XDM is intentionally exposed over plain HTTP and the browser refuses to send the session cookie.

To allow HTTP access in a Docker installation, add the following property to the core-server environment in the <docker-compose.yml> file:

Kubernetes
Docker

Parameters:

server.servlet.session.cookie.secure

core:
  environment:
    - name: server.servlet.session.cookie.secure
      value: "false"

Parameters:

server.servlet.session.cookie.secure

services:
  core-server:
    environment:
      - server.servlet.session.cookie.secure=false

Installation of Lookup Tables

If you want to use the masking feature of XDM, you need to install the XDM lookup tables. These lookup tables provide a pre-defined set of first and last names, addresses and banking information. The lookup tables are used by the pre-defined XDM masking methods.

Follow these steps to install the lookup tables:

Download the lookup tables from https://products.ubs-hainer.com/downloads/XDM3/Lookup-Tables-2.3.232/XDM3_Lookup_Tables.zip
Unzip the downloaded file. The ZIP file contains the file h2_default.mv.db.
Create a mapping table container in XDM. To do this, go to the mapping table container list in the user interface and click on Create
Now enter the name XDM Lookup Table for the mapping table container and the variable name lookupTable to use the default configuration of the downloadable configuration examples in the user manual.
The credential required is one with the user sa and an empty password. If you have already created one, select it. Otherwise, create one by pressing the + button.
Now open the object by clicking Create and edit. You can now upload the actual mapping table using the Upload button. To do this, click on Upload and select the previously downloaded and unpacked h2_default.mv.db and upload it. This may take a moment depending on the size of the file and the network speed.
When the upload is complete, save the mapping table container object.

Internal contact

The About page which is accessible via the System menu on the right-hand side of the head bar shows both the version number and the contact details for UBS Hainer support. UBS Hainer is shown as vendor on this page and also per default in prompts in dialogs.

If an internal contact person from the own company is to be contacted first before UBS Hainer support in the event of problems, this can be defined via an environment variable. This contact person is then also linked if there is a prompt in a dialog, for example in the task execution export, to contact them If problems occur.

Kubernetes
Docker

to be determined

...

Properties:

contact_information

services:
  web-ui:
    environment:
       - contact_information=<team-name> <br> <mail-or-link>

The internal contact information supports Markdown for formatting the text.

Logging

The XDM services write log information to the output of the respective docker containers. The detail of the log can be configured using environment parameters.

UI Logging

By default, the xdm-ui container writes information on server events into the docker log. User specific information is written into the browser console on the client. The client log can be configured to be visible on the docker log.

There are three parameters to configure the logging:

browser_log_level: This specifies the log level for the browser console for each individual client. The default is Info.
local_log_level: This specifies the log level for the browser storage for each individual client. The default is Info.
nginx_log_level: This specifies the log level for message written to the docker log from each client. The default is Off.

Possible log levels are:

Off No log messages are displayed

Fatal Messages are displayed for events blocking a workflow.

Error Messages are displayed that cause missing information, but the workflow may continue.

Warn Messages are displayed that indicate possible problems.

Info Messages are displayed informing about the usage of the UI.

Verbose Messages are displayed helping to debug.

All All Messages are displayed.

Every level includes all levels listed above.

Kubernetes
Docker

Not available.

Parameters:

browser_log_level
local_log_level

nginx_log_level

    xdm-ui:
        environment:
            - browser_log_level=Info
            - local_log_level=Info
            - nginx_log_level=Warn

When activating nginx_log_level the log is stored in the path /var/log/nginx. The files are automatically rotated. If you want to persist the logs, consider to mount the path /var/log/nginx to a persistent storage.

Inspecting the log

The log messages can be viewed in the user interface. The browser console can be found in the development tools of the respective browser. The local log is visible inside the interface under Client Log in the User Settings. This shows all messages logged in the local instance of the UI. A collection of all messages connecting to the same web-ui service is visible under Client System Log in the System Settings.

Cross-service logging for task executions

XDM makes it possible to summarize logs of a task execution using the loki service, and it is possible to trace calls and their execution time using the tempo service.

For Loki, these parameters have to be set in the configuration:

JAEGER_AGENT_HOST: Tracing agent host name
JAEGER_ENDPOINT: Tracing HTTP endpoint for sending spans directly to a collector.
JAEGER_SAMPLER_TYPE: Instantiate the type of sampling for the tracer
JAEGER_SAMPLER_PARAM: The corresponding parameter of the JAEGER_SAMPLER_TYPE

If you use the grafana service, it is possible to display the information collected by the two services via the dashboard Logs, Traces, Metrics. The configuration of loki and tempo in graph-store, webservice-extract-source and webservice-apply-sink, is only necessary, if the components are used in the XDM installation.

Kubernetes
Docker

Parameters:

Loki:
- JAEGER_AGENT_HOST
- JAEGER_ENDPOINT
- JAEGER_SAMPLER_TYPE
- JAEGER_SAMPLER_PARAM
- Service port to define Loki URL

Tempo

Service zipkin port to define Tempo zipkin URL

loki:
  environment:
     JAEGER_AGENT_HOST: tempo
     JAEGER_ENDPOINT: http://tempo:14268/api/traces
     JAEGER_SAMPLER_TYPE: const
     JAEGER_SAMPLER_PARAM: 1
  service:
     port: <port>

tempo:
  service:
     zipkinPort: <port>

Parameters:

Loki:
- JAEGER_AGENT_HOST
- JAEGER_ENDPOINT
- JAEGER_SAMPLER_TYPE
- JAEGER_SAMPLER_PARAM
Tempo:
- mount for tempo-data
dataflow-server, graph-store, webservice-extract-source, webservice-apply-sink
- xdm_loki_serverUri
- xdm_tempo_zipkin_serverUri

Grafana:

LOKI_URL

TEMPO_URL

  loki:
    image: docker.ubs-hainer.com/grafana/loki:3.1.0
    environment:
      - JAEGER_AGENT_HOST=tempo
      - JAEGER_ENDPOINT=http://tempo:14268/api/traces
      - JAEGER_SAMPLER_TYPE=const
      - JAEGER_SAMPLER_PARAM=1

  tempo:
    image: docker.ubs-hainer.com/xdm3-tempo:latest
    volumes:
     - ./tempo-data:/tmp/data

  dataflow-server:
    environment:
      - xdm_loki_serverUri=http://loki:3100
      - xdm_tempo_zipkin_serverUri=http://tempo:9411

  grafana:
    environment:
       - LOKI_URL=http://loki:3100
       - TEMPO_URL=http://tempo:3200

  graph-store:
    environment:
      - xdm_loki_serverUri=http://loki:3100
      - xdm_tempo_zipkin_serverUri=http://tempo:9411

  webservice-extract-source:
    environment:
      - xdm_loki_serverUri=http://loki:3100
      - xdm_tempo_zipkin_serverUri=http://tempo:9411

  webservice-apply-sink:
    environment:
      - xdm_loki_serverUri=http://loki:3100
      - xdm_tempo_zipkin_serverUri=http://tempo:9411

Loki configuration

Loki is a log aggregation system developed by Grafana Labs, designed to work seamlessly alongside Prometheus. Unlike traditional log systems, Loki indexes only metadata (labels), not the full log content. Logs are grouped into streams based on these labels.

Configuring a retention period for Loki

Loki allows users to configure a retention period which tells Loki to delete old logs after their age reaches beyond the specified value.

This feature is disabled by default. To enable it, simply add the environment variable and set its value to true

Furthermore, these parameters can be set individually in the configuration when Loki retention period is enabled:

LOKI_RETENTION_PERIOD: The age at when the logs should be deleted. Default is 1 week (1w).
COMPACTION_INTERVAL: Defines, how often compaction/retention is applied. Default is 10 minutes (10m).
RETENTION_DELETE_DELAY: Defines the the delay after which marked chunks will be deleted. Default is 5 minutes (5m).

Kubernetes
Docker

Parameters:

RETENTION_ENABLED
LOKI_RETENTION_PERIOD
COMPACTION_INTERVAL

RETENTION_DELETE_DELAY

loki:
  environment:
    RETENTION_ENABLED: true
    LOKI_RETENTION_PERIOD: "1w"
    COMPACTION_INTERVAL: "10m"
    RETENTION_DELETE_DELAY: "5m"

Parameters:

RETENTION_ENABLED
LOKI_RETENTION_PERIOD
COMPACTION_INTERVAL

RETENTION_DELETE_DELAY

services:
  loki:
    image: docker.ubs-hainer.com/xdm3-loki:latest
    environment:
        - RETENTION_ENABLED=true
        - LOKI_RETENTION_PERIOD=1w
        - COMPACTION_INTERVAL=10m
        - RETENTION_DELETE_DELAY=5m

If the RETENTION_ENABLED variable is set to true, then the LOKI_RETENTION_PERIOD variable is used with its default (1 week) or overwritten value.

If you encounter permission problems while setting up the retention period, you can try the following command within the loki-data folder:

sudo chown -R 10001:10001 .

(make sure to include the . )

This command ensures that the contents of the volume of the loki service can be accessed by changing the user of that volume to the standard loki user (10001).

Other environment variables in the loki service can be configured to set different options for the retention period.

The value of the LOKI_RETENTION_PERIOD should have at least the same time span as the execution token. This token specifies how long a session of an execution will be active. The loki logs should at least be kept for as long as the executions can be run.

When changing the retention period, make sure to use a value that is equal to or higher than 24h. Loki is not able to automatically delete logs newer than 24 hours.

Configuring additional options for Loki

If you wish to include additional configurations for loki, you can add these variables as well:

LOKI_MAX_CONCURRENT_TAIL_REQUESTS: Defines, how many tail requests can be sent to loki simultaneously. Default is 20.
LOKI_MAX_RECV_MSG_SIZE: Defines the maximum size for receiving messages. Default is 8388608 (8MiB)
LOKI_MAX_SEND_MSG_SIZE: Defines the maximum size for sending messages. Default is 8388608 (8MiB)

Kubernetes
Docker

Parameters:

LOKI_MAX_CONCURRENT_TAIL_REQUESTS
LOKI_MAX_RECV_MSG_SIZE

LOKI_MAX_SEND_MSG_SIZE

loki:
  environment:
    LOKI_MAX_CONCURRENT_TAIL_REQUESTS: 20
    LOKI_MAX_RECV_MSG_SIZE: 8388608
    LOKI_MAX_SEND_MSG_SIZE: 8388608

Parameters:

LOKI_MAX_CONCURRENT_TAIL_REQUESTS
LOKI_MAX_RECV_MSG_SIZE

LOKI_MAX_SEND_MSG_SIZE

services:
  loki:
    image: docker.ubs-hainer.com/xdm3-loki:latest
    environment:
      - LOKI_MAX_CONCURRENT_TAIL_REQUESTS=20
      - LOKI_MAX_RECV_MSG_SIZE=8388608
      - LOKI_MAX_SEND_MSG_SIZE=8388608

These variables are optional, meaning if you leave them out, their default value will be used instead.

Docker Service Logging

For all services that are deployed with the XDM installation, the logs are available in the docker logs. For the following services, the log level can be changed as described below:

core-server
dataflow-server
file-sink
generator-source
graph-store
webservice-apply-sink

The default log level of these services is INFO and can be changed by setting environment variables.

One possibility is to set the root log level, which affects all components of the service. This can be done with environment variable logging.level.root. The possible values for the variable are: TRACE, DEBUG, INFO, WARN, ERROR, FATAL, or OFF.

It is also possible to set the log level for individual components. For example, to set the log level of the authentication, you can set the environment variable logging.level.org.springframework.security in the core-server service. If you want to increase the log level of other specific components, you can contact UBS Hainer Support to get more information.

Kubernetes
Docker

Parameters:

logging.level.root

environment:
    logging.level.root: DEBUG

Parameters:

logging.level.root

environment:
  - logging.level.root=DEBUG

Session Logging

XDM persists the logged-in sessions in its administration database. Therefore, it is possible to query information about users and this information also gets archived. The following property defines how long this information is kept. This value defaults to 30 days. When specifying another value, this value has to follow the syntax for a duration.

Kubernetes
Docker

Parameters:

xdm.core.session.keepExpiredSessionsFor

core:
     environment:
         xdm.core.session.keepExpiredSessionsFor=P30D

Parameters:

xdm.core.session.keepExpiredSessionsFor

environment:
 - xdm.core.session.keepExpiredSessionsFor=P30D

Markup Configuration

Overview

To improve system security and prevent the injection and execution of potentially harmful code, the platform now restricts the use of markup or HTML code in announcement texts, data shop descriptions, and the information page by default. This behavior is governed by a system property that allows administrators to enable or disable markup support as needed.

Default Behavior

By default, the system treats any markup or HTML code as plain text. This means that any HTML tags or markdown-like syntax entered in announcement or description fields will not be rendered or interpreted, but instead shown as-is.

This approach helps to prevent:

Cross-site scripting (XSS) vulnerabilities
Misuse of HTML for layout manipulation
Injection of malicious scripts

Enabling Markup Support

To explicitly allow markup or HTML rendering, administrators can set the system property allowMarkdownDescription. Once this property is set to true, the system will permit and render HTML/markup content in:

Announcements
Data Shop descriptions
XDM Information pages

Kubernetes
Docker

Parameters:

allowMarkdownDescription

xdm:
  pui:
    environment:
      allowMarkdownDescription: true

Parameters:

allowMarkdownDescription

services:
web-ui:
  environment:
    - allowMarkdownDescription=true

Security Considerations

When allowMarkdownDescriptions is disabled, the vulnerable rendering library is not used, providing a higher level of protection.

When enabled, ensure only trusted users have the ability to create or edit content containing markup/HTML. Restrict the group of users who can create content — especially announcements — to the absolute minimum necessary.

Always sanitize user input when markup is allowed.

Keep allowMarkdownDescriptions set to false unless HTML/markup rendering is explicitly required.

Document internal policies for using markup if enabling this feature and educate content editors about potential security risks associated with HTML content.

Use Cases

Disabled (default): Secure environments where user input should be strictly treated as text.

Enabled: Internal systems or trusted environments where enriched formatting is needed (e.g., links, bold text, lists).

Mapping Table Containers Storage

Describes the storage and persistence definition for lookup tables and mapping table containers used for masking and anonymization XDM uses a H2‑based internal DBs managed by XDM.

Main properties are configured in the UI (see Mapping Table Container description in the Reference for more information); only storage backend is configured here:

Kubernetes (cloud storage example)
Docker

Assigned to the general storage or persistence configuration, or cloud storage section

storage:
  type: "s3"
  s3:
    bucket: "xdm-mapping"
    region: "eu-central-1"
    endpoint: "s3.eu-central-1.amazonaws.com"
    credentials:
      accessKey: "AKIA..."
      secretKey: "..."

Mapping DBs live under /xdm/mapping (mounted volume)

services:
  core-server:
    volumes:
      - ./xdm-mapping:/xdm/mapping
  dataflow-server:
    volumes:
      - ./xdm-mapping:/xdm/mapping

Override Execution User for Deployment Flexibility

Depending on security policies or platform constraints, it may be necessary to run XDM containers with a different execution user. This page describes where to override the runtime user in Docker Compose and Kubernetes (values.yaml).

For more detailed information on how to set up a non-root user environment, you can refer to the official documentation for both Docker and Kubernetes.

In Docker installations with rootless Docker deamon this setting is required. In all other cases, the setting is optional.

Kubernetes
Docker

For Kubernetes deployments, configure user and group settings using securityContext in values.yaml.

The security context fields are documented here: Security context.

Example:

core:
  securityContext:
    runAsUser: 0
    runAsGroup: 0
    fsGroup: 0

The same pattern can be applied to other services where supported by the chart.

In Docker Compose based installations, by default the XDM containers are running with a non-root user. If you need to change the execution user to root for any reason, add the user attribute to all services that mount a host volume.

core-server:
  user: "root:root"

Make sure, that the host permissions for the volume mounts are compatible with the selected UID/GID.

If you are using a Docker runner provider, which is set via xdm.core.runner.provider=DOCKER, you can also override the execution user for the xdm-runner container in the runner.yaml configuration file. This can be done by setting the following property:

executionUser: 0:0

See the full documentation for the runner.yaml configuration file here: Runner Deployment Configuration.

Configuring Pod Affinity for Kubernetes

In XDM we highly recommend using a cloud storage solution that supports the ReadWriteMany access mode. This gives you the best performance and lets you fully benefit from Kubernetes features such as flexible scheduling, scalability and higher availability.

If you do not have the option to use cloud storage with ReadWriteMany, you must ensure that the core and dataflow pods run on the same node. This significantly reduces both performance and the advantages of Kubernetes and should therefore only be used as a temporary workaround to make XDM runnable until a suitable storage solution is available.

As a transitional solution, you can use podAffinity. One pod is selected by a label, and the other pod is configured to run on the same node as that labeled pod. In the following example, the core pod is placed on the same node as the dataflow pod by using the app.kubernetes.io/name: xdm-dataflow label:

Kubernetes
Docker

core:
  service:
    port: 8000
    type: ClusterIP
  image:
    name: xdm3-core
    version: '{{ .Values.xdm.version }}'
    repository: '{{ .Values.repository }}'
    pullPolicy: IfNotPresent
  affinity:
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app.kubernetes.io/name: xdm-dataflow
          topologyKey: "kubernetes.io/hostname"

Not available.

Notes:

Changing the accessMode of existing PVCs may not be accepted by Kubernetes.
In this case, it may be necessary to:
- stop the affected Deployments/StatefulSets,
- explicitly delete the corresponding PVCs (and, if required, PVs),
- run the Helm upgrade again so that the PVCs are recreated with the correct parameters.
Before doing this, make sure that a valid backup exists if important data is stored on the volume.

Prime UI

The Prime UI can be installed to run in parallel with the current web UI. When enabled, it is available under the /prime/ path on the same host as the current web UI. For example, if your classic web UI is available under xdm.my-company.com, the prime UI ia a available under xdm.my-company.com/prime/ .

Kubernetes
Docker

Parameters:

prime-ui: enabled=true

oauth2: redirect-uri in user management set.

prime-ui:
  enabled: true
userManagement:
  oauth2:
    registration:
      <name>:
        redirect-uri: <redirect-uri>/api/login/oauth2/code/<provider>

Ensure the redirect-uri includes the full path, including everything from /api/… onward. Add /api/login/oauth2/code/<provider> and replace <provider> with the actual provider name, so the final redirect URL is complete.

To enable Prime UI in a Docker installation, update the Prime UI URL, add the Prime UI service, and (if applicable) include the OpenID settings

Parameters:

xdm_prime_ui_url (in web-ui)
xdm_core_serverUri (in prime-ui)
xdm_context_path (in prime-ui)
xdm.core.ui.serverUri (in core-server)

spring.security.oauth2.client.registration.<provider>.redirect-uri (in core-server)

services:
    web-ui:
        environment:
            - xdm_prime_ui_url=http://prime-ui:8888/prime/
    prime-ui:
        image: docker.ubs-hainer.com/xdm3-prime-ui:latest
        environment:
            - xdm_core_serverUri=http://core-server:8000/api/
            - xdm_context_path=prime/
        depends_on:
            - core-server
    core-server:
        environment:
            - xdm.core.ui.serverUri=https://<external-domain-url>/prime/
            - spring.security.oauth2.client.registration.<provider>.redirect-uri=https://<external-domain-url>/api/login/oauth2/code/<provider>

Set <external-domain-url> to the externally reachable URL you have configured for your own domain. When using OpenID, update xdm.core.ui.serverUri and ensure that the redirect-uri contains the full path, including everything from /api/… onward. Add /api/login/oauth2/code/<provider> and replace <provider> with the actual provider name, so the final redirect URL is complete.

If you use OpenID authentication, only one redirect URL can be stored. Therefore, logins initiated from Prime UI will redirect to the current web UI. After login, you can switch back to /prime/ in the browser.

Storage backends for Tasks and Icebox

Storage settings specify where to store task directories, Icebox generations ans Mapping Table Containers. XDM supports storing in a local storage or cloud storage. By default, XDM stores the files on the local file system. For Kubernetes, XDM also supports several cloud storage systems:

Amazon AWS S3 compatible storage
Microsoft Azure blob store
Google Cloud buckets

To enable a cloud storage you must specify one of the options described in the following sections.

Kubernetes (S3 example)
Docker (local)

Parameters:

storage.type ( s3 | azure | gcp )
storage.s3.*
storage.azure.*

storage.gcp.*

storage:
  type: "s3"
  s3:
    bucket: "xdm"
    region: "eu-central-1"
    endpoint: "s3.eu-central-1.amazonaws.com"
    credentials:
      accessKey: "AKIA..."
      secretKey: "..."

Usually Local mount, also S3 Storage or similar solutions possible.

services:
  core-server:
    volumes:
      - ./xdm-tasks:/xdm/tasks
      - ./xdm-backups:/xdm/backups
  dataflow-server:
    volumes:
      - ./xdm-tasks:/xdm/tasks
      - ./xdm-backups:/xdm/backups

Amazon AWS S3 storage

To store the data in Amazon AWS S3 compatible storage, the following settings are available. To enable this type of storage you must specify a value of s3 for the type attribute.

storage:
  type: "s3"
  s3:
    bucket: "xdm"
    region: "eu-central-1"
    pathStyle: "false"
    endpoint: "s3.eu-central-1.amazonaws.com"
    credentials:
      accessKey: "AKIAIOSFODNN7EXAMPLE"
      secretKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

bucket: A S3 bucket is an object container. Objects can’t exist without a bucket. For details about buckets refer to Amazon’s documentation: Bucket overview and naming rules for buckets.
region: Region specifies the geographic location where your data will be stored. See regions and endpoints in the AWS General Reference.
pathStyle: 'pathStyle' specifies whether to use the path-style or virtual-hosted-style access for the S3 API. The default value is 'false', which means that the virtual-hosted-style access will be used. See Accessing Amazon S3 using path-style or virtual-hosted-style URLs.
endpoint: Together with the previous two properties, this property specifies the S3 API endpoint for object manipulation/access. See regions and endpoints in the AWS General Reference.
accessKey: An accessKey is like a username and is used for authentication of requests. See Managing access keys for IAM users.
secretKey: secretKey is like a password and is used together with accessKey to authenticate requests. See Managing access keys for IAM users.

Microsoft Azure blob store

To store the data in a Microsoft Azure blob store, the following settings are available. To enable this type of storage you must specify a value of azure for the type attribute.

Permissions

XDM requires several permissions to access the Azure blob service. The required privileges and resources are listed below. They must be specified in the Shared Access Signature (SAS) token creation dialog:

Type
Allowed services	Blob
Allowed resource types	Container, Object
Allowed permission	Read, Write, Delete, Add, Create

Type

Allowed services

Blob

Allowed resource types

Container, Object

Allowed permission

Read, Write, Delete, Add, Create

XDM will store the data in blob objects stored in a container. It needs access to the container and object resource types.

storage:
  type: "azure"
  azure:
    containerName: "xdm"
    endpoint: "https://xdmtest1.blob.core.windows/..."
    tokenSecret: "my-secret"

containerName: The name of the Azure container where the blobs will be stored. XDM will create the container if it does not already exist.
endpoint: The SAS URL to access the blob service.
tokenSecret: As an alternative to a token a Kubernetes secret can be used. The name of the secret can be chosen freely. The secret must contain a key named token that contains the SAS token.

Google Cloud Bucket

To store the data in a Google Cloud Platform compatible storage, the following settings are available. To enable this type of storage you must specify a value of gcp for the type attribute.

storage:
  type: "gcp"
  gcp:
    bucketName: "sample-bucket"
    projectId: ""
    keySecret: "my-secret"

bucketName: The name of the bucket that stores the data. The bucket will be created if it does not exist, otherwise the existing bucket is used to store files in it.
projectId: The ID of the respective Google Cloud project.
keySecret: A secret name that contains the access key of the service account in JSON format. The content of the secret must be stored with data.json.

Token based authentication configuration

XDM supports the user authentication with an API Token. XDM uses a non-interactive mode for logging in via API Token meaning that the user is not forwarded to the authentication provider’s login page. Instead, the user may log in via the username and the API token directly from XDM.

As a prerequisite for using an API Token within XDM, the corresponding role must be set and defined in the docker compose file. Therefore, the environment variable xdm.core.security.required-token-creation-role must be set to the name of the role which should be able to create and use API Tokens.

If the external user is deactivated or deleted at the authentication provider, the respective XDM user has to be deactivated manually inside XDM. By this, the corresponding API Token will also be deactivated and can not be used for authentication anymore. This will not happen automatically.

To configure the connection via an API Token, you need to edit for docker installations the docker_compose.yml file or for Kubernetes installations the values.yaml file.

Kubernetes
Docker

security:
    requiredTokenCreationRole:<any role>

core-server:
  environment:
    - xdm.core.security.required-token-creation-role=<any role>

Examples

Kubernetes
Docker

To allow users with the role XDM_USER_GROUP to create and use API Tokens, add the following line to the values.yaml file:

security:
    requiredTokenCreationRole:XDM_USER_GROUP

You can as well define multiple roles that are allowed to create and use API Tokens. To allow users with the role XDM_USER_GROUP and XDM_TESTER_GROUP to create and use API Tokens, add the following line to the values.yaml file:

security:
    requiredTokenCreationRole:XDM_USER_GROUP,XDM_TESTER_GROUP

To allow users with the role XDM_USER_GROUP to create and use API Tokens, add the following line to docker-compose.yml in the environment section in the core-server configuration block:

environment:
- xdm.core.security.required-token-creation-role=XDM_USER_GROUP

environment:
- xdm.core.security.required-token-creation-role=XDM_USER_GROUP,XDM_TESTER_GROUP

Further information on how to use API Tokens and the concepts behind it can be found in the Token based authentication concepts.

Override kafka properties

Kafka client properties can be overridden when XDM checks a Kafka topic and when an Entity Copy Task reads or writes Kafka messages. Specify the property by appending its Kafka property name to the appropriate XDM prefix.

Override Kafka client properties for check

When you want to override a Kafka client property for the Check Configuration Action, set the property as an environment variable of the Core Server using the xdm.kafka.admin. prefix.

Kubernetes
Docker

core:
  environment:
    - xdm.kafka.admin.<parameter-name>=<parameter-value>

services:
  core-server:
    environment:
      - xdm.kafka.admin.<parameter-name>=<parameter-value>

Override Kafka client properties for copy

When you want to override a Kafka client property for an Entity Copy Task, set the property as an environment variable of the Runner using the xdm.kafka.consumer. or xdm.kafka.producer. prefix, depending on whether the task reads or writes messages. Furthermore, it can be set as a task property using a Task Stage Hook before the copy stage.

Kubernetes
Docker

deployer:
  configuration:
    environmentVariables:
      - xdm.kafka.consumer.<parameter-name>=<parameter-value>
      - xdm.kafka.producer.<parameter-name>=<parameter-value>

For example, the following environment variables change the maximum number of records returned by the Consumer and the compression type used by the Producer:

deployer:
  configuration:
    environmentVariables:
      - xdm.kafka.consumer.max.poll.records=1000
      - xdm.kafka.producer.compression.type=gzip

services:
  dataflow-server:
    environment:
      - xdm.kafka.consumer.<parameter-name>=<parameter-value>
      - xdm.kafka.producer.<parameter-name>=<parameter-value>

For example, the following environment variables change the maximum number of records returned by the Consumer and the compression type used by the Producer:

services:
  dataflow-server:
    environment:
      - xdm.kafka.consumer.max.poll.records=1000
      - xdm.kafka.producer.compression.type=gzip

When you want to set it for an individual task execution, add a Task Stage Hook before the copy stage and add the property to the task’s properties map.

Groovy
JavaScript
Ant

properties.put('xdm.kafka.consumer.<parameter-name>', '<parameter-value>')
properties.put('xdm.kafka.producer.<parameter-name>', '<parameter-value>')

properties.put('xdm.kafka.consumer.<parameter-name>', '<parameter-value>');
properties.put('xdm.kafka.producer.<parameter-name>', '<parameter-value>');

<property name="properties.xdm.kafka.consumer.<parameter-name>" value="<parameter-value>"/>
<property name="properties.xdm.kafka.producer.<parameter-name>" value="<parameter-value>"/>

The value set by a Task Stage Hook takes precedence over the corresponding Runner environment variable. Both take precedence over the value set by XDM.

Admin client properties cannot be changed by a Task Stage Hook because the Kafka topic check is performed by the Core Server outside a task execution.

Properties set by XDM

XDM sets the following Kafka client properties automatically. Some properties are only set when the corresponding authentication or encryption option is enabled on the Kafka Topic. They can be overridden by using the prefixes described above.

Kafka property Client Value set by XDM

Kafka property	Client	Value set by XDM
`bootstrap.servers`	Admin, Consumer, Producer	URL configured on the Kafka Topic
`group.id`	Consumer	Consumer group configured for the task or Kafka Topic; otherwise, an XDM-generated consumer group
`enable.auto.commit`	Consumer	`false`
`auto.offset.reset`	Consumer	`none`
`key.deserializer`	Consumer	`org.apache.kafka.common.serialization.ByteArrayDeserializer`
`value.deserializer`	Consumer	`org.apache.kafka.common.serialization.StringDeserializer` for JSON or `org.apache.kafka.common.serialization.ByteArrayDeserializer` for Avro
`request.timeout.ms`	Producer	`30000`
`delivery.timeout.ms`	Producer	`120000`
`max.block.ms`	Producer	`120000`
`acks`	Producer	`all`
`enable.idempotence`	Producer	`true`
`max.in.flight.requests.per.connection`	Producer	`5`
`linger.ms`	Producer	`5`
`key.serializer`	Producer	`org.apache.kafka.common.serialization.ByteArraySerializer`
`value.serializer`	Producer	`org.apache.kafka.common.serialization.ByteArraySerializer`
`security.protocol`	Admin, Consumer, Producer	`PLAINTEXT`, `SSL`, `SASL_PLAINTEXT`, or `SASL_SSL`, depending on the Kafka Topic configuration
`sasl.mechanism`	Admin, Consumer, Producer	`OAUTHBEARER` when OAuth authentication is enabled
`sasl.jaas.config`	Admin, Consumer, Producer	OAuth login module and the credentials configured on the Kafka Topic
`sasl.login.callback.handler.class`	Admin, Consumer, Producer	`org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginCallbackHandler` when OAuth authentication is enabled
`sasl.oauthbearer.token.endpoint.url`	Admin, Consumer, Producer	Token endpoint configured on the Kafka Topic when OAuth authentication is enabled
`ssl.endpoint.identification.algorithm`	Admin, Consumer, Producer	Empty when hostname verification is] disabled on the Kafka Topic

bootstrap.servers

Admin, Consumer, Producer

URL configured on the Kafka Topic

group.id

Consumer

Consumer group configured for the task or Kafka Topic; otherwise, an XDM-generated consumer group

enable.auto.commit

Consumer

false

auto.offset.reset

Consumer

none

key.deserializer

Consumer

org.apache.kafka.common.serialization.ByteArrayDeserializer

value.deserializer

Consumer

org.apache.kafka.common.serialization.StringDeserializer for JSON or org.apache.kafka.common.serialization.ByteArrayDeserializer for Avro

request.timeout.ms

Producer

30000

delivery.timeout.ms

Producer

120000

max.block.ms

Producer

120000

acks

Producer

all

enable.idempotence

Producer

true

max.in.flight.requests.per.connection

Producer

5

linger.ms

Producer

5

key.serializer

Producer

org.apache.kafka.common.serialization.ByteArraySerializer

value.serializer

Producer

org.apache.kafka.common.serialization.ByteArraySerializer

security.protocol

Admin, Consumer, Producer

PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL, depending on the Kafka Topic configuration

sasl.mechanism

Admin, Consumer, Producer

OAUTHBEARER when OAuth authentication is enabled

sasl.jaas.config

Admin, Consumer, Producer

OAuth login module and the credentials configured on the Kafka Topic

sasl.login.callback.handler.class

Admin, Consumer, Producer

org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginCallbackHandler when OAuth authentication is enabled

sasl.oauthbearer.token.endpoint.url

Admin, Consumer, Producer

Token endpoint configured on the Kafka Topic when OAuth authentication is enabled

ssl.endpoint.identification.algorithm

Admin, Consumer, Producer

Empty when hostname verification is] disabled on the Kafka Topic

Overriding properties that XDM automatically set, can prevent the entity copy tasks from working correctly.

Graph Store

The Graph Store service is a component of XDM that manages the storage and retrieval of graph data. It is responsible for handling graph-related operations, such as storing nodes and edges, executing graph queries, and maintaining the integrity of the graph data for Entity Copy Task Templates or Generation Task Template.

Overwriting additional Java options for the Graph Store service

When you want to overwrite additional Java options for the Graph Store service, set the property as an environment variable of the Graph Store service using the JAVA_TOOL_OPTIONS variable.

Kubernetes
Docker

graph_store:
    environment:
      JAVA_TOOL_OPTIONS: "-XX:MaxDirectMemorySize=512M"

For Kubernetes the MaxDirectMemorySize is set default to 512M

graph-store:
    environment:
        - JAVA_TOOL_OPTIONS=-XX:MaxDirectMemorySize=512M

Optional parameters

Admin database configuration

Database configuration

Creating database tables

Database user

Disable automatic database migration

AI Assistance

Central Search (Elasticsearch)

Configuration as Code (CasC)

Configuration

Properties

Property definition

Using properties

Filtering of YML files

Example

Certificate Based Authentication

Installation of certificates

Verify imported certificates

Installation of certificates for microservices

Serving under a Context Path and using Reverse Proxy

Data Type Conversion

Docker Memory Settings

Configure memory for task executions

Define the Event Execution Log Cleanup Process Interval

Accessible Tables for Task Execution Report SQL statements

Gateway Timeouts

Configuring HTTP Timeouts For Long-Running Operations

Recommended Timeout Values

Nginx Reverse Proxy

Kubernetes Ingress (Nginx Ingress Controller)

XDM UI Container Environment

Troubleshooting 504 Gateway Timeout Errors

Groovy Script references

HTTP settings

Changing the TCP port

Using SSL encryption

Using Self-Signed Certificates

Using HTTPS for internal communication

Using HTTPS for the web interface

Example

Session Cookie Settings For HTTP Deployments

Installation of Lookup Tables

Internal contact

Logging

UI Logging

Inspecting the log

Cross-service logging for task executions

Loki configuration

Configuring a retention period for Loki

Configuring additional options for Loki

Docker Service Logging

Session Logging

Markup Configuration

Overview

Default Behavior

Enabling Markup Support

Security Considerations

Use Cases

Mapping Table Containers Storage

Override Execution User for Deployment Flexibility

Configuring Pod Affinity for Kubernetes

Prime UI

Storage backends for Tasks and Icebox

Amazon AWS S3 storage

Microsoft Azure blob store

Permissions

Google Cloud Bucket

Tags

Token based authentication configuration

Examples

Override kafka properties

Override Kafka client properties for check

Override Kafka client properties for copy

Properties set by XDM

Graph Store

Overwriting additional Java options for the Graph Store service