Changelog

The Entity Copy Task now supports Entity Modification Methods, allowing the users to adjust the data during the copy process. These Entity Modification Methods are linked to a Domain Entity with the use of an Entity Modification Rule. The documentation has been extended with additional examples and usage notes to demonstrate how to utilize modification methods in Entity Copy Tasks effectively.

This new feature may only become available once the epic XDEV-2000 - Support for Kafka Topics has been fully completed. Information on the current development of epics can be found at epic overview.

Increased the maximum credential password length from 512 characters to 1MB to allow storing and using long tokens (e.g., JWT) directly in credentials and accessing them in Stage Hooks, avoiding insecure storage in config files. Existing credentials continued to work unchanged.

The new clear and organized display of parameters makes managing embedded objects intuitive through a simple and user-friendly interface:

A common application would be, for example: A user is working on a data shop with various configuration parameters. The new feature allows the user to easily add, edit, and delete parameters without needing to manually adjust the underlying data structure. If a parameter is deleted, the change is instantly reflected in the parent system, which updates the data accordingly.

Known issue: When creating a new form parameter in a data shop that uses mapped properties referencing another XDM object (e.g., targetEnvironment), the "Possible Values" drop-down appears empty, although it should display the XDM objects.

Fixed an issue where the report generation appeared to be stalled. The stage was marked complete in the logs but still shown as running in the UI. Updated stage status handling to reflect ongoing processing until reports are fully generated.

Fixed an issue where Domain Entities created via JSON import included an empty endpoints array, causing Entity Copy Tasks using the entity to fail with an internal server error. The import logic was updated to no longer generate the endpoints array when empty, allowing tasks to complete successfully.

Registering a new Icebox generation became increasingly slow as more generations accumulated. For example, with ~8,000 generations on a task template, the registration step could take several minutes.

Root cause: the registration step retrieved the full list of existing generations to calculate the next generation number. Fix: the step now fetches only the latest generation and derives the next number from it.

This change was implemented in a central/shared code path and may improve performance in other areas as well (e.g., workflows creating new tasks from task templates when many tasks already exist).

To enhance the performance of XDM overall, modifications have been made to the underlying database schema. The way in which object parameters were stored in the SingleValueParameter and MultiValueParameter tables has changed. Parameters are now stored directly as JSONB columns on the respective objects. This significantly reduces the number of database queries required to resolve parameters, resulting in shorter loading times and improved performance for various XDM actions, including the pre-execution check, RebuildGraph and other operations involving intensive parameter access.

The SingleValueParameter and MultiValueParameter tables will remain in place for the time being. The parameter tables linking the objects to the parameters are also retained. Existing data will be migrated to the new JSONB format automatically when the application is started, ensuring the original dataset remains available during the transition period.

The Neo4j-based XDM relationship store has been replaced with a simple Postgres table. Existing relationships will be created automatically in the new table when the core-server is started for the first time. Referencing behavior remains the same – objects cannot be deleted as long as dependent relationships exist.

The Neo4j container and the associated settings are therefore obsolete. In Docker setups, the Neo4j service should be removed from docker-compose.yaml, and the following environment variables should be deleted from the core-server and dataflow-server configuration.

No further adjustments are necessary for Kubernetes installations, provided that the current Helm Chart is used – the Neo4j container has already been removed there.

Fixed a bug where the Exclude Rule for column A caused modified data to be incorrectly inserted into column B instead of being excluded as intended. The Modification Rule now correctly ignores changes when column A is excluded, ensuring that data for column A is ignored and remains excluded without affecting other columns.

Trying to delete a Permission Recipient with user default permissions caused a 500 error. The deletion process was updated to ensure that it completes successfully without errors.

Fixed an issue where automatically generated keys from Db2 were not correctly transferred to dependent columns in PostgreSQL during RLP copy. The fix ensured that the 'Generated Always' columns were included in the generated columns list, allowing the generated keys to be accurately stored in the referenced table, thus maintaining the integrity of data relationships.

When Groovy Scripts were called many times during a very long workflow or a task execution, it could happen that a MissingMethodException was thrown after some time, causing the execution to fail. This was caused by a setting in the Groovy Script engine, that used soft references for generated classes. Therefore, classes could be removed by the garbage collector, which caused the exception. To prevent this issue, you can now set this new environment variable in the dataflow container:

xdm.script.groovy.hardReferences=true

Further information about this setting can be found in the configuration of Groovy script references section of the user manual.

When using a RLP Task with fill mode MERGE and a user-defined primary key on a nullable column, the comparison with NULL values always generated an INSERT instead of a MERGE SQL statement. User-defined primary keys now have a new flag markNullValuesIdentical which defines whether two NULL values are equal or not. The MERGE ON clause was updated to correctly compare NULL values. When comparing nullable columns, a cast to the correct datatype was added because NULL values have no valid datatype.

The Entity Copy Task now supports Entity Modification Methods, allowing the users to adjust the data during the copy process. These Entity Modification Methods are linked to a Domain Entity with the use of an Entity Modification Rule. The documentation has been extended with additional examples and usage notes to demonstrate how to utilize modification methods in Entity Copy Tasks effectively.

This new feature may only become available once the epic XDEV-2000 - Support for Kafka Topics has been fully completed. Information on the current development of epics can be found at epic overview.

When Groovy Scripts were called many times during a very long workflow or a task execution, it could happen that a MissingMethodException was thrown after some time, causing the execution to fail. This was caused by a setting in the Groovy Script engine, that used soft references for generated classes. Therefore, classes could be removed by the garbage collector, which caused the exception. To prevent this issue, you can now set this new environment variable in the dataflow container:

xdm.script.groovy.hardReferences=true

Further information about this setting can be found in the configuration of Groovy script references section of the user manual.

A new guide has been added that describes a practical two step approach for cloning a complete or almost complete schema. The approach separates structural alignment from data transfer by combining a structure compare task with a native table copy task. It can be found here.

Modification Methods in NTC (Native Table Copy) and CTC (Compatibility Table Copy) tasks were updated to expose all source columns (including dropped ones) and any additional SELECT-only columns, improving flexibility for complex modifications and migrations. A new guide has been added to describe how to utilize this new functionality. It can be found here.

This new feature may only become available once the epic XDEV-2013 - Batch Row Refactoring has been fully completed. Information on the current development of epics can be found at epic overview.

An alpha release of the Prime UI is now available alongside the current web UI. It can be enabled for Docker-based installations or Kubernetes deployments by following the instructions in Installation and Kubernetes. This allows customers to try the new Prime UI in parallel with the existing interface.

Fixed an issue for entity copy tasks that copy from Kafka topics. When multiple Kafka messages shared the same key, only a single message was copied.

Fixed Kafka Copy Task issues where some logs were missing in stage logs/exports and where offset gaps in a topic caused kafka-source to abort or copy messages incompletely. Also improved Error and Exception Handling.

Fixed Kafka Copy Task issues where some logs were missing in stage logs/exports and where offset gaps in a topic caused kafka-source to abort or copy messages incompletely. Also improved Error and Exception Handling.

The modification methods in the Configuration Object Repository are now provided as Groovy examples.

The JavaScript versions of these methods can still be used in XDM, but we recommend using the Groovy methods instead.

A new guide has been added that describes how to add a mapping table to XDM that acts as a shuffling table for data from an existing table. It can be found here.

A list of all system parameters was added to the custom parameter reference. The names of the system parameters can not be used as variable names for custom parameters.

Access to columns via index-based data has been replaced by access via column names. This ensures correct mapping and stable, error-free copies and changes in selection, exclude, mapping, modification and reduction scenarios, even when the order or number of columns changes. This change only affects the product code. Customer-defined scripts remain unaffected and do not require any adjustments.

This new feature may only become available once the epic XDEV-2013 - Batch Row Refactoring has been fully completed. Information on the current development of epics can be found at epic overview.

Added an import endpoint to Domain Entities that generated a configuration script from an uploaded JSON. Attributes were created for each top-level key, and lists/mappings are set to OBJECT. This enabled faster entity creation from example JSON without manual attribute setup.

More details can be found in the Domain Entity - Import List Action section of the user manual.

This new feature may only become available once the epic XDEV-2000 - Support for Kafka Topics has been fully completed. Information on the current development of epics can be found at epic overview.

Removed support for SSH private keys from Credentials because the private key was not implemented and therefore was not used in XDM. The following fields: Certificate based authentication, Key Format and Private Key, was removed to prevent clear-text display and export, and to eliminate any risk of unauthorized access or data leakage.

Fixed an issue where tasks ran extremely slowly when Trace logging was enabled, causing steps to appear stuck due to excessive log output. Logging was adjusted to reduce Trace verbosity, restoring normal step execution time. Internal HTTP payloads will no longer be logged.

Fixed an issue for entity copy tasks that copy from Kafka topics. When multiple Kafka messages shared the same key, only a single message was copied.

A complete overview of how to work with Configuration as Code (CasC) has been added to the Working with XDM section of the user manual. It can be found here.

Implemented two new system properties, xdm.core.accessTokenRolesClaim and xdm.core.idTokenRolesClaim, to control from which claim or ID token the user roles are extracted. See User roles for details.

An issue has been fixed where unquoted columns in CHECK constraints were causing case mismatches when executing generated DDL in a cross-copy from Db2 z/OS to Postgres. DDL generation has been updated to ensure case consistency by quoting columns in CHECK constraints.

A problem has been fixed where the displayed step duration during task execution was misleading when many log lines were written. This was characterized by the fact that the duration of a running step gradually increasing and then suddenly becoming shorter once the execution step was complete. The duration of each task step is now displayed correctly, and the total duration of all steps matches the actual execution time again.

Sensitive information, such as environment variables and database details, is no longer displayed in execution logs.

Enhanced logging for entity copy tasks that copy from or to Kafka topics was implemented, providing users with regular updates on the copy process. The total number of processed messages is displayed every minute.

This new feature may only become available once the epic XDEV-2000 - Support for Kafka Topics has been fully completed. Information on the current development of epics can be found at epic overview.

A problem has been fixed where not all log messages were displayed in the execution log. This occurred with log messages that were output in quick succession. Now all log messages are displayed.

Fixed an issue where data could not be copied with Fill mode INSERT due to the presence of a Generated Always column, causing the task to fail with the error No value specified for parameter 1. See here for details how the Generated Always columns are handled during data copying.

Fixed an issue where columns in the schema browser and column picker appeared in an unsorted order.

Fixed an issue where the DDL generation generated wrong DDL for Oracle columns with defaults in some situations. In particularly 'DEFAULT USER' was incorrectly replaced with 'SESSION_USER'.

Fixed an issue where bulk delete option showed no feedback and the delete dialog remained open when delete permissions were missing for one or more selected items. The fix ensured that the permission message is displayed and the delete button is disabled in the delete dialog.

Fixed a bug where parsing timestamps in RLP tasks with multiple threads resulted in NumberFormatExceptions due to the non-thread-safe nature of SimpleDateFormat. The implementation was updated to use a thread-safe alternative, ensuring that timestamp parsing now executes without errors across multiple threads, maintaining stable performance regardless of thread count.

Implemented a filtering feature in the Entity Copy Task for Kafka topics sources to allow users to control the volume of data copied. Users can now specify partition, offset, and timestamp criteria. If only a topic is provided, all messages from all partitions are copied. When a partition is specified, only messages from that partition are copied. Setting an offset or timestamp will only copy messages greater or equal to the given values. The filters are combined using a logical AND to refine the data selection.

This new feature may only become available once the epic XDEV-2000 - Support for Kafka Topics has been fully completed. Information on the current development of epics can be found at epic overview.

Fixed an issue where NULL values in Kafka payloads were not being copied to the target topic. The bug caused the process to fail with a 'NOT_FOUND: Database not found' error when attempting to write empty values. The update ensures that payloads containing NULL values are now accurately copied, preserving the NULL entries in the target topic without triggering errors.

Exporting an environment to a zip file or checking its configuration failed when a Data Relation Rule was used in the Application Model, because the script language was not read correctly.

Fixed an issue where a login error message in German („Login fehlgeschlagen“) was displayed instead of an English error message („Login failed“) on an English installation. Fixing ensured that the English error message will be displayed in the English environment.

Implemented support for Partitioned Tables in DDL generation for PostgreSQL. The DDL generation now accurately reflects the necessary syntax for PostgreSQL, allowing for seamless transfer of Partitioned Tables from Db2 z/OS.

Fixed an issue where ALTER statements were not processed correctly by the File Bridge server, resulting in an error message stating that the value ALTER is not listed as a constant. The code was updated to ensure that the ALTER statement is now successfully executed and applied to the table without any errors. You need to update the File Bridge server and driver to the current version to use this fix.

Fixed a bug where the original data type of a form parameter was changed from Collection to String when using scripts in the Data Shop, particularly with Collections. The fix ensured that the original data type remains unchanged, allowing methods like collection.add() to function correctly. Additionally, if no default value is set, an empty list is now passed, preventing errors when adding entries to the Collection.

Disabled reporting in Tempo to prevent the transmission of anonymous usage data to Grafana. This change was made to eliminate excessive warnings in the logs when there is no internet access, indicating failed attempts to connect to Grafana.org. The configuration option reporting_enabled was set to false, ensuring that reporting is consistently disabled.

Implemented the ability to copy message headers, keys, and optional timestamps from incoming Kafka messages to outgoing messages. This change was made to ensure that metadata from the original messages is preserved for further processing. The functionality includes options to enable or disable header copying and timestamp retention, enhancing flexibility in message handling.

This new feature may only become available once the epic XDEV-2000 - Support for Kafka Topics has been fully completed. Information on the current development of epics can be found at epic overview.

Implemented a feature that allowed users to select the authentication method within the Kafka Topic object. Users can now choose between 'None' and 'OAuth Bearer Token'. When 'OAuth Bearer Token' is selected, a mandatory 'TokenEndpoint' field appears. Beside that an option was added to support SSL encryption for Kafka connections.

This new feature may only become available once the epic XDEV-2000 - Support for Kafka Topics has been fully completed. Information on the current development of epics can be found at epic overview.

Analyzed the transaction behavior in the RLP task using FillMode Merge to identify the causes of the Broken PIPE error. This change aimed to ensure stable operation of XDM with the HIT driver and MERGE, addressing transaction handling and commit processes that were causing lock issues during large data transfers. With this change the evaluation of the used merge apply units was improved and additional log messages were added to analyze the selection of the merge apply units.

Fixed an issue where the Application Model names were not displayed in the Source Objects Report. The column for Application Models remained empty for table based tasks, because this information is not available in the runtime environment. Therefore, the column is no longer displayed in reports of table based task executions.

Fixed a bug where NUMERIC columns without length and scale in PostgreSQL caused the CompareEvaluator to throw a NumberFormatException.

Fixed a bug where leading or trailing spaces in usernames during login resulted in the creation of multiple permission recipients for the same user. Existing permissions recipients with leading or training spaces are not merged automatically. This needs to happen via the transfer ownership functionality and a manual delete of the duplicate permission recipients.

Sensitive information, such as environment variables and database details, is no longer displayed in execution logs.

A guide has been added that describes how to use a stage hook and iterative execution of this hook to check the relationships within an application model and identify missing relationships in order to optimize data selection in the application model.

The guide can be found here.

A valid license key is now required to control the 'Data Generation' option. Consequently, the feature toggle in the Core Server configuration is no longer relevant. The 'Store' option, which enabled users to access the Graph Store, has also been transferred to the license key and removed as a feature toggle.

This new feature may only become available once the epic XDEV-1988 - Data generation has been fully completed. Information on the current development of epics can be found at epic overview.

Fixed a bug where the init script removed default values of custom parameters during the order process. The issue occurred when any code or comments were present in the init script, resulting in empty fields instead of displaying default values. The fix ensures that default values remain intact unless explicitly referenced in the init script.

Workflows with overwritten custom parameter caused a problem when they were accessed in a script. The process of overwriting a custom parameter has also changed an internal property of the workflow. With this fix, overwriting custom parameter works as expected. Furthermore, the wrongly changed internal property was corrected for workflows with overwritten parameter.

There was a bug that sets the script language of a modification rule to Groovy when this rule uses a legacy modification method. Now, a modification rule, which uses a legacy modification method, can only be written in JavaScript. Therefore, on all existing such rules, the script language was set to JavaScript. Also, when importing modification sets from older YAML exports, this behavior is enforced.

When exporting a task execution that contains huge data directories (e.g. exporting a compatibility task execution between Stage 5 and Stage 6), the core server could crash and no execution export was possible. Now, the data directory is never restored into a temporary directory inside the core container.

Fixed an issue where exporting more than 100 objects resulted in a 414 error (Request-URI Too Large). A clear error message is now provided for failed exports, and the bulk actions now only include the relative context path in the 'url' parameter to reduce header length.

Added a new object type "Kafka Topic" in XDM to allow users to flexibly add and utilize Kafka topics as sources or targets for copy processes.

This new feature may only become available once the epic XDEV-2000 - Support for Kafka Topics has been fully completed. Information on the current development of epics can be found at epic overview.

Implemented a new system property, xdm.core.execution-report-table-whitelist, to control access to specific tables in the XDM database for TaskExecutionReport SQL statements. This change was made to protect sensitive data from unauthorized access in Task Execution Reports. The property ensures that if not set, only a minimal set of tables is accessible. See configuration of accessible tables for task execution reports for details.

Implemented a new microservice to write Kafka topics. The service uses the data within the Entity Records that was previously collected by the Kafka Source microservice.

This new feature may only become available once the epic XDEV-2000 - Support for Kafka Topics has been fully completed. Information on the current development of epics can be found at epic overview.

Introduced a configuration option (allowMarkdownDescription) that, when set to false, blocks the insertion and rendering of markup/HTML in Announcements, Data Shop Descriptions, and Information Pages. The default remains true, so existing behavior does not change unless explicitly configured. This option is intended for security-hardened environments to reduce attack surface (e.g., XSS, layout manipulation, script injection); when disabled, rendering is bypassed and the HTML parsing library is not used.

Fixed an issue where Grafana statistics displayed incorrect numbers and excessive entries. The Top Executed Objects dashboard now shows accurate execution counts, and the Top 10 Executions dashboard is limited to a maximum of 10 entries. Adjustments were made to ensure statistics align with executed tasks and workflows, and descriptions in the Home dashboard were updated to clarify that all diagrams include deleted executions.

Fixed a bug where users without appropriate permissions could use the REST-API endpoint. All endpoints now correctly check for the required permissions and return a 403 Forbidden status when permissions are lacking.

Fixed a bug where RLP tasks with Performance Supervision would fail at the end of Stage 6 due to an ArrayOutOfBound exception while collecting statistics. The issue occurred after data writing was completed. The code was updated to encapsulate the Performance Supervision process, ensuring that any errors during statistics collection now produce a warning instead of causing a task failure.

Fixed a bug where renaming a connection that is used in a To Icebox Task caused the corresponding From Icebox Task to fail during the Extract Reports step due to the connection not being found. The step will now display a warning message that the connection is missing, and the task will continue executing.

Fixed an issue where a text field was displayed instead of a drop-down list when adding a new form parameter. The drop-down list for possible values is now correctly shown during the creation process, allowing users to select from predefined values. This change ensures that the functionality remains consistent after saving the data shop.

Fixed an issue where parameters with identical names could be added to a modification method without triggering an error. Now, when attempting to add parameters with duplicate names, an error message is displayed immediately, preventing the saving of the modification method. This change ensures that users cannot save modifications with duplicate parameter names, enhancing data integrity.

Fixed a bug where the log view would jump to the beginning upon reloading when the Follow-Log mode was disabled. The log now retains its current position during data refreshes in Non-Follow-Log mode, ensuring a stable user experience. The Follow-Log mode automatically activates when the user scrolls to the end of the log, allowing for seamless data loading.

Implemented a new AI assistance feature in the code editor to provide automated code suggestions. The current code and prompt are sent to a microservice that interacts with a language model, returning suggestions.

This new feature may only become available once the epic XDEV-2008 - XDM AI Assistent has been fully completed. Information on the current development of epics can be found at epic overview.

Fixed a NullPointer Exception that occurred during column name mapping in GENERIC mode. The issue arose when attempting to map columns between source and target. The code has been updated to ensure the correct processing and mapping of column names, enabling correct column mapping and error-free copying of data.

Fixed a bug where the filter fields did not match the scheduled task list. All relevant fields, except the option for 'Output' are now available in the filter. The filter options now correctly correspond to the columns in the scheduled task list. Additionally, users can now filter the list by 'Executable'.

Fixed an issue where the default value for date parameters in data shops was not being saved, resulting in the field remaining empty after saving. The change ensured that the default value can now be successfully set, remains visible in both the template and order views, and is correctly displayed in the order form after saving. Setting a data type value is now only possible using the calendar.

Fixed a bug where parsing timestamps in RLP tasks with multiple threads resulted in NumberFormatExceptions due to the non-thread-safe nature of SimpleDateFormat. The implementation was updated to use a thread-safe alternative, ensuring that timestamp parsing now executes without errors across multiple threads, maintaining stable performance regardless of thread count.

Fixed a bug where the task continued running despite an exception being thrown in the init() method. The task now correctly aborts when a ModificationException or RuntimeException occurs. The exception is logged as a SEVERE message, and no warning is displayed on the interface when the task is aborted, ensuring proper error handling and user experience.

The problem was that NULL values in nullable indexes were always treated as the same value. This resulted in rows being discarded from the process. NULL values will now be treated as different in the processing, so reflecting the rules given by the underlying index.

Fixed a NullPointer Exception that occurred during column name mapping in GENERIC mode. The issue arose when attempting to map columns between source and target. The code has been updated to ensure the correct processing and mapping of column names, enabling correct column mapping and error-free copying of data.

Fixed a bug where parsing timestamps in RLP tasks with multiple threads resulted in NumberFormatExceptions due to the non-thread-safe nature of SimpleDateFormat. The implementation was updated to use a thread-safe alternative, ensuring that timestamp parsing now executes without errors across multiple threads, maintaining stable performance regardless of thread count.