Configuration

By default, XDM automatically creates the administration tables and sequences in the specified database system if they do not exist yet. XDM also automatically migrates its administration tables to the latest version if it is necessary. Every time XDM starts, it will generate the DDL for all tables and sequences that are missing or changed and store it in the file 'admin-db-diff.sql' inside the data directory. Since this file is newly generated at each start, it only contains the latest changes made to the administration database. With this file, users can inspect the changes made to the database compared to the previously installed version.

The automatic creation and migration of the administration database can be disabled, as described in disable automatic database migration. But in this case, the administrator has to migrate the database himself with the generated 'admin-db-diff.sql' file. This file contains the SQL statements executed by the migration system. It is also possible that the database skips some migration steps that XDM would take at startup, which are too complicated to describe as plain SQL statements.

For being able to use the JDBC driver for Google Big Query that was mentioned above, it is necessary to add some additional dependencies. You can either add them directly into the driver directory or build a super JAR that consists of the actual JDBC driver for Google Big Query and all other necessary dependencies. The latter is recommended. The dependencies that have to be included are listed below.

The directory where XDM searches for JDBC drivers can be configured with the property xdm.core.jdbc-driver-dir. The default directory is /xdm/config/jdbc-drivers/ which is used when no other value is specified. When changing the default, it must be specified in the environment sections of both, core and dataflow server. The following configuration shows the default setting.

The property xdm.core.registered-jdbc-driver specifies a comma separated list of additional JDBC driver names to load. This property can only be used in the environment section of the dataflow server and the default is an empty list. IBM’s AS400 and IMS drivers are always implicitly added to this list. Specifying driver names is only necessary if the driver can not be automatically loaded, as it is the case for the two IBM drivers. The following line shows how a driver with the fully qualified name org.vendor.jdbc.someDriverName would be added to the list.

The certificates must be located in a directory /xdm/certs/ which is mounted by XDM.

The certificates get imported through the entry point script at the startup of the core-server, dataflow-server and xdm-runner. Check the logs of the respective service to verify that the certificates have been imported successfully. Check for the following log entry:

Microservices like KafkaSource and KafkaSink may also use certificates, e.g, when the connection to the Kafka broker requires encryption. The installation for these services is similar to the other services (like the core-server) by using a volume mount for the certificates.

But there are two additional requirements:

This file must be named type and must have the content ca-certificates. Finally, in the environment section of the service a new entry SERVICE_BINDING_ROOT must be added. The complete configuration may look like the following:

As an example, this is a valid content of the folder which is specified by <microservice-certificate-folder>

XDM can use an LDAP server to authenticate users. The LDAP server controls which users exist and how they have to authenticate. The LDAP server also controls what roles a user is a member of.

To configure the connection to LDAP, you need to edit the file docker_compose.yml and add the following parameters to the section services→core-server:

Optionally, you can also define the following group search properties:

Details on the use of these properties can be found in the Spring LDAP documentation.

To test your settings, you can use the ldapsearch utility. This utility is available in all Microsoft Windows Enterprise versions, and under Linux.

Use this command:

This command will establish a connection to the specified LDAP and retrieve information about the user specified under <user_name_to_lookup>.

XDM offers a built-in user management system that allows you to maintain the usernames, passwords, and roles in a plain text file. Each line represents a separate user and must have the following format:

XDM supports the user authentication with an external OpenID Connect provider. XDM forwards login requests to the configured OpenID Connect provider and the user needs to log in at that system. After a successful login the OpenID system redirects to XDM. These settings must be configured with options that are described in this section.

As a prerequisite for using an OpenID provider with XDM, some information must be set and defined in the provider. Here the corresponding client to be used for the authentication within XDM must be defined and configured.

To configure the connection to an OpenID Connect provider, you need to edit the file docker_compose.yml. Add the following lines to environment in the core-server configuration block.

The client’s registered redirect URI that the authorization server redirects the end-user’s user-agent to, after the end-user has authenticated and authorized access to the client. Typically, this is set to <base-url>/api/login/oauth2/code/<provider-name> where <base-url> is the base URL of your XDM installation.

The Spring Security properties are prefixed with spring.security.oauth2.client.registration, followed by the client name and the name of the client property. The client name represents the name specified in the OpenID Connect provider.

The provider has to embed the user role names inside the access token resource_access or realm_access claims within a list of strings named roles. When using the resource_access, claim the entity roles must be embedded in a sub-entry whose name is defined by the value of the azp entity.

Microsoft’s Azure Active Directory deploys the roles within the ID-token.

Each role name has to be written in uppercase prefixed with ROLE_. This prefix is a requirement of the spring security framework and won’t be visible to users who will see only the role name without this prefix. If the provider doesn’t define a role for a user, no default role is specified.

Example access token content with the definition of three roles:

The redirect URI must be a valid link to which a browser can redirect after a successful login.

Example: https://xdm-ui:4280

Example configuration for the identity and access management client Keycloak:

To set the XDM credential panel to be visible initially if an OpenID Provider is configured for an XDM installation, add the following line in the section services→web-ui→environment:

LDAP and OpenID authenticators provide additional user information upon successful login. This information can be used to adapt the username to a more meaningful value. By default, the 'givenName' property is used, but the following properties can also be used:

Obviously, not all possible properties will provide a better username. Furthermore, it depends on the authentication provider which properties are available and have a value.

When another property instead of the default should be used, add the key xdm.user.full-name-property in the section services→core-server→environment and specify the property name. The following line will change the displayed username to the user’s family name:

Specifies a role to which the usage of XDM will be limited. This property enables the administrator to define a list of roles, which will restrict the access to XDM. In order to use XDM, an authorized user has to be a member of one of these roles. By default, all authorized users are able to use XDM.

This setting controls the message that is displayed if a user tried to log in, that is not a member of the roles specified by the property xdm.required-user-role. By default, the following message will be displayed: Missing one of the following roles: <roles set in xdm.required-user-role>

Specifies the role, of which the members will be treated as data shop users. These users don’t need full access to all functions of XDM and will receive a customized UI with which they can more easily order test data and see the results of their orders. When specifying multiple purchaser roles, then use a comma separated list.

The system property xdm.core.purchaser.role must be specified in the docker-compose.yml file of the XDM installation. Please add the line in the section services→core-server→environment and specify the role of the data shop users.

All users in this role are only able to use the web interface with reduced functionality. See description of purchaser layout for more details and recommended permission settings for data shop purchaser users.

Specifies a role, that will be able to see and use the pre-defined matchers, comparable fields, and modification methods, that XDM ships with. By default, access to these pre-defined entities is restricted to the system, and only users with administrative permissions are able to see, use and change them.

Users that are a member of the specified role will be able to see and use the pre-defined matchers, comparable fields, and modification methods, however they will not be able to change them. These objects can only be changed by users with administrative permissions.

To allow read access from a different role, customize the docker-compose.yml file and specify the following parameter in the services→core-server→environment section:

By default, the XDM user interface is accessible on TCP port 4280, which is the default port for the HTTP protocol.

To change the TCP port number for the XDM user interface, edit the file docker-compose.yml and locate the section services → web-ui → ports. The value of this parameter is a list of port mappings that contains one entry. This entry is a string, enclosed in apostrophes, that contains the external and the internal port numbers, separated by a colon.

The internal port number is 80 and must not be changed. The default value for the external port number is 4280. You can change the external port number to any value that does not cause conflicts with other services that are running on the same system.

Example

To change the TCP port number for the XDM user interface to 4281, change the entry so that it looks like this:

Sets the environment name for the XDM installation. Each XDM installation can have a specific environment name that is displayed in the UI. This makes it easier to differentiate between different XDM installations, for example for a production and test environment.

To set the environment name for an XDM installation, edit the file docker-compose.yml. In the section services→web-ui→environment, add the following line:

Set the primary color of the XDM UI. It is recommended to use a different coloring for the different XDM installations. This setting can have the following options:

The columns above are predefined colors working fine with other colors and texts in the UI.

Alternatively, you can use a valid CSS color definition, see Color palette and Color syntax for more information. Please consider that the text for the menu entries is always white, so the color should be chosen accordingly. For color names from CSS that are equal to the predefined colors, the predefined colors will be used. Use RGB or hex code color definitions for custom colors.

Additionally, the secondary column, which is used for the background of the side menu footer, can be set using CSS color definitions with the variable secondary_environment_color. The default secondary color is black.

To set the environment color for an XDM installation, edit the file docker-compose.yml. In the section services→web-ui→environment, add the following line:

The About page which is accessible via the System menu on the right-hand side of the head bar shows both the version number and the contact details for UBS Hainer support. UBS Hainer is shown as vendor on this page and also per default in prompts in dialogs.

If an internal contact person from the own company is to be contacted first before UBS Hainer support in the event of problems, this can be defined via an environment variable. This contact person is then also linked if there is a prompt in a dialog, for example in the task execution export, to contact them If problems occur.

To set the internal contact information for an XDM installation, edit the file docker-compose.yml. In the section services→web-ui→environment, add the following line:

The internal contact information supports Markdown to layout the text.

Specifies the secret, with which the JWT token is generated and encrypted.

To change the secret for JSON Web Tokens that XDM uses, edit the file docker-compose.yml, add the following parameters to the section services → core-server → environment:

After changing the secret, you must restart the XDM core service for the change to take effect. After restarting the core service, all generated JWT tokens will be encrypted with the new secret.

Time can be specified in XDM as a duration or as a period. A duration is used when the user has to specify a shorter time range in the size of minutes, hours or days. Periods define longer time ranges in days, months or years. Both are based on the ISO 8601 format.

The Dataflow Server periodically checks whether a task is still running. The waiting time between these periods can be set using the following parameter:

The parameter <seconds> specifies how many seconds the process waits between the checks. By default, this parameter is set to 5 seconds.

Every time before an execution starts, a pre-execution check will be executed to determine whether the task is correctly configured or not. This check can be deactivated for tasks started from workflows with the following boolean parameter:

Each XDM task execution runs as a standalone program that is executed within the dataflow docker container. By default, each execution uses a quarter of the available memory from the docker container. For example, if the dataflow container can use up to 2 GB memory, each task execution will have 500 MB by default.

In some situations this setting is not sufficient, for example if a large RLP extract task needs more than a quarter of the available memory. This setting can be changed via the parameter 'ANT_OPTS' in the docker-compose.yml file using the following environment variable in the dataflow-server section:

In this example each task execution would have up to 800 MB memory.

Alternatively the memory allocation can be set relatively:

By default, the xdm-ui container writes information on server events into the docker log. User specific information is written into the browser console on the client. The client log can be configured to be visible on the docker log.

There are three parameters to configure the logging:

Possible log levels are:

Off No log messages are displayed

Fatal Messages are displayed for events blocking a workflow.

Error Messages are displayed that cause missing information, but the workflow may continue.

Warn Messages are displayed that indicate possible problems.

Info Messages are displayed informing about the usage of the UI.

Verbose Message are displayed helping to debug.

All All Messages are displayed.

Every level includes all levels listed above.

XDM makes it possible to summarize logs of a task execution using the loki service and it is possible to trace calls and their execution time using the tempo service.

Both services are configured in the docker-compose.yaml file under the loki and tempo sections.

After the two services have been set, the following properties must be set in the environment section of dataflow-server, graph-store, webservice-extract-source and webservice-apply-sink:

If you use the grafana service, it is possible to display the information collected by the two services via the dashboard Logs, Traces, Metrics. To do this, the following properties must be specified in the environment section of the grafana service.

For all services that are deployed with the XDM installation, the logs are available in the docker logs. For the following services, the log level can be changed as described below:

The default log level of these services is INFO and can be changed by setting environment variables.

One possibility is to set the root log level, which affects all components of the service. This can be done with environment variable logging.level.root. The possible values for the variable are: TRACE, DEBUG, INFO, WARN, ERROR, FATAL, or OFF.

It is also possible to set the log level for individual components. For example, to set the log level of the authentication, you can set the environment variable logging.level.org.springframework.security in the core-server service. If you want to increase the log level of other specific components, you can contact UBS Hainer Support to get more information.

The way to set this variable depends on your installation method of XDM.

To enable to Configuration as Code the following settings must be specified in the docker-compose.yml file. The settings must be specified in the environment section of the core-server:

The specified user needs permissions for list and for creating all types of objects.

For more information how to create a list access permission, see the section list access permissions.

The configuration will start a scheduled job inside the XDM server. The job will check for new commits every time it is executed. All objects contained in changed files will be imported. The output of the import operation will be listed in the user interface under System Settings / System Configuration / Scheduled Jobs in the Configuration as Code entry.

All configuration objects in the configuration as code repository must reference objects that are itself part of the configuration as code repository. It is not possible to reference objects, that are part of the XDM instance, but not listed inside the configuration as code repository.

If the import process fails, the errors will be listed in the output of the scheduled job Configuration as Code. The scheduled job will be paused automatically and will not run until the scheduled job is resumed. If the issue is due to a broken object definition in the repository, an additional commit may be pushed to the repository to fix the problem. When the scheduled job is resumed all configuration changes of both commits will be processed.

Properties are defined in files which have to be located in the root directory. Global properties need to be defined in a file named casc.properties. Environment-dependent properties can be put in a separate file which must be named like casc-<ENVIRONMENT_ID>.properties. The later ones will possibly overwrite global properties.

When using the repository for different environments, it’s possible that not every XDM-object is needed in every environment.

For instance, environments can be differentiated by databases they use. Therefore, in an environment that only uses Postgres database connections other connection types should not be imported.

This can be achieved by using cascignore-<ENVIRONMENT_ID>-files, inspired by the .gitignore-file mechanism. It is also possible to define global cascignore-files, which will be used on every import. For details how to define the files to ignore, see the Git documentation on gitignore. Typically, absolute paths are defined in the cascignore-file. Such an entry must be prefixed with

Files matching a pattern in a cascignore-file or cascignore-<ENVIRONMENT_ID>-file will not be imported.

Optionally, it is possible to configure the values for jsonData yourself for the built-in data source of the administration database. By default, the sslmode is set to disabled and the postgresVersion to version 12. More details how to configure a PostgresQL data source can be found here.

To configure your own jsonData values, the following must be added to the environment section of Grafana

Since XDM provides a link for reaching Grafana and both services share the user management, XDM needs to know how to reach Grafana. Therefore, the following settings are needed in the environment section of the core-server.

The following example shows the relevant part of the default docker-compose.yml file. Here, XDM can be accessed under the context path xdm, so the Grafana configuration also has to include this context path. Without using a context path, the URL would become http://report-dashboard:3000/reports/.

There will be a link from the user interface to the Grafana dashboard. Therefore, the following settings are needed in the environment section of the web-ui.

The following example shows the relevant part of the default docker-compose.yml file.

In XDM Grafana is running as a docker container, so it is not possible to customize Grafana via the grafana.ini file like described in the documentation, because the grafana.ini file is packed in the docker image and can not directly be customized. But each of the properties from the grafana.ini file can be set a environment variable in the docker-compose.yaml file or the helm charts of the Kubernetes installation.

This works like the following schema shows:

Details:

Loki allows users to configure a retention period which tells Loki to delete old logs after their age reaches beyond the specified value. This value can be set within the docker-compose file in the 'loki' service.

This feature is disabled by default. To enable it, simply add the environment variable and set its value to true:

Other environment variables in the loki service can be configured to set different options for the retention period:

If you wish to include additional configurations for loki, you can add these variables as well:

These variables are optional, meaning if you leave them out, their default value will be used instead.

To improve system security and prevent the injection and execution of potentially harmful code, the platform now restricts the use of markup or HTML code in announcement texts, data shop descriptions, and the information page by default. This behavior is governed by a system property that allows administrators to enable or disable markup support as needed.

By default, the system treats any markup or HTML code as plain text. This means that any HTML tags or markdown-like syntax entered in announcement or description fields will not be rendered or interpreted, but instead shown as-is.

This approach helps prevent:

To explicitly allow markup or HTML rendering, administrators can set the following system property:

This feature is disabled by default. To enable it, simply add the environment variable (unless this parameter is already listed) and set its value to true.

Once this property is set to true, the system will permit and render HTML/markup content in:

When allowMarkdownDescriptions is disabled, the vulnerable rendering library is not used, providing a higher level of protection.

When enabled, ensure only trusted users have the ability to create or edit content containing markup/HTML.

Always sanitize user input when markup is allowed.

Disabled (default): Secure environments where user input should be strictly treated as text.

Enabled: Internal systems or trusted environments where enriched formatting is needed (e.g., links, bold text, lists).

Keep allowMarkdownDescriptions set to false unless HTML/markup rendering is explicitly required.