Modification rule

Modification rules change the content of tables while they are being processed.

Concepts and more information

Properties

The table below documents the available properties for modification rules. The 'name' column displays the property name as it can be used in Groovy and Java Scripts.

Name

Type

Default

Description

Active

active

Boolean

true

Specifies whether the rule is active and therefore used, or inactive and therefore ignored, when a task is executed.

Adaption Script

adaptionScript

String

/* * This function adjusts the data array before executing the actual modification method. / / def convertTo() { } */

/* * This function adjusts the data array after execution of the actual modification method. / / def convertFrom() { } */

The adaption script controls how a data values are changed before and after the invocation of the respective modification method.

More details can be found here here

Automatic datatype conversion per row

automaticDatatypeConversionPerRow

Boolean

false

XDM converts data types after modification so that they fit into the target table. Usually, for performance reasons, only the first row of each table is checked to determine whether such a conversion is necessary.

In situations where the first row of a table would fit into the target without conversion, but further rows might not, you can set this property to true so that the check is performed for every row. Doing this reduces performance, so only use this property if absolutely necessary, or if only a small number of rows is expected.

Such situations can occur when the return value of a user written modification method is not compatible with the data type of the target column. An alternative to using this property is to change the modification method to use the correct data types.

Column pattern

columnPattern

String

%

Specifies a selection pattern which is used to match the names of columns. The rule will be applied to the matching columns. If you use patterns for the table schema or table name, then the rule will be applied to all columns which match the column pattern in all matching tables.

Condition Script

conditionScript

String

/* * This function should return true or false and determines whether * the modification method is executed. / / def condition() { return true } */

This script decides whether the modification method should be applied for a row or not. The script is called for all selected rows of the table on which the modification rules is defined.

More details can be found here here

Description

description

String

n/a

An optional description for this object. The description can contain multiple lines to give more context on the configured object. The description is not used in a technical context.

Libraries

libraries

File

n/a

A list of library files that can be accessed in the script code. When editing the classification term usage, all XDM files of the specified script language are made available for selection. The content of the selected XDM files is automatically added to the script context. All methods, variables, and classes in the selected files can be accessed in the format script.

Modification method

modificationMethod

ModificationMethod

n/a

Specifies the modification method to be used by this rule. The method is selected from a list of available modification methods.

Modification propagation

modificationPropagation

Boolean

false

Specifies whether the modification should be propagated to dependent tables. If a primary key has been changed using the modification rule and this option is enabled, the changes to the primary key are also applied in the child tables that reference this key. Modification propagation is only used if:

  • the selected modification method is a column based method,

  • the scope is Target, and

  • tables are related through either a foreign key in the database, or a data apply rule.

Propagation is used, if a column modification rule modifies the parent column of a foreign key relationship. In this case, it is necessary to modify the related child columns in the same way to preserve the relationship. To preserve the relationship, XDM propagates the modification rule to all referencing columns in child tables.

The propagation is applied on database foreign key constraints as well as XDM data apply rules.
When a rule is propagated, XDM will execute the same rule on the child table. If the result of the modification method is not the same for two executions (for example, a random number), then the relationship will be broken in the target environment.
The propagation is not used if the modification is row-based. To change keys when using modification with row-based methods, the rules must be defined on all tables containing the foreign key.

+ XDM propagates modification rules with scope Target only if the flag Modification propagation is enabled and either a database foreign key, or a data apply rule exists between the target tables.

For table copy tasks, the only possibility to propagate modification is to use a database foreign key, since the task type does not support data apply rules.

+ For modification rules with scope Source the modified data will not be propagated.

Modification propagation can be enabled/disabled individually for each modification rule.

Name pattern

namePattern

String

%

Specifies a selection pattern which is used to match the name of a database table.

Parameters

parameters

List

n/a

The parameters are loaded dynamically from the selected modification method and can be set explicitly for this rule. Furthermore, for each existing parameter of the modification method, another parameter is created with the name <methodParameterName>ValueType. For example, an additional parameter SeedValueType is created for the parameter Seed. This specifies what the associated parameter refers to.

The generated value type parameter has two possible values:

Column

Specifies that the associated parameter is a column of the selected table and takes the value from it for the modification.

Constant

Indicates that the value of the used modification is a constant. If this is selected, any possible value can be specified. Furthermore, the stored values of the modification method parameter can be used.

Schema pattern

schemaPattern

String

%

Specifies a selection pattern which is used to match the schema of a database table.

Scope

scope

Scope

SOURCE

The scope can be set to either Source or Target. The scope specifies whether the rule will be applied to a source object or a target object.

Script language

scriptLanguage

Enum

GROOVY

This script language specifies in which language all scripts of the object are written. Currently the following languages are supported:

Groovy (GROOVY)

specifies that the code is written in Groovy.

JavaScript (JS)

specifies that the code is a JavaScript.

Tags

tags

Tag

n/a

Contains the tags that apply to this object. These tags can be used in the search to find objects quickly and effortlessly.

Actions

The available actions are described below. Some actions apply to the list, while others are specific to selected modification rules.

List Actions

The following actions are available on the modification rules list. If the action is disabled a tooltip will provide the exact reason for the deactivation. The required permissions are described in detail for each action.

  • Bulk Delete

  • Create

  • Export CSV

  • Import CSV

  • List History

  • Ordering

Delete the selected objects.

The following options are available:

Cascade

Recursively delete depending objects.

When using cascade, dependent objects are deleted first also with cascade enabled. Thus, a cascade deletion is a recursive function that deeply searches for dependent objects and deletes them first. There is only a confirmation for the first object. The dependent objects are deleted without confirmation but only when the user has the DELETE permission.

This feature is only available in development mode. More information about development mode can be found in the chapter User Settings. It should be used with caution.

An object in the result list can have two different states, these are:

DELETED

The object could be deleted.

NOT_DELETED

The object could be not deleted. This may be because the executing person does not have a delete permission on the object or the object is still referenced by others. A detailed reason can be determined with the help of the error message. If the object is still in use, these objects are also displayed.

The following permissions are required on the list:

  • READ

  • WRITE

Creates a new object in the current list. Depending on the object type either a popup dialog is shown for the most important settings, or the complete object is shown in edit mode. The dialog provides the option to create the object and remain in the current list or to switch to the newly created object in edit mode to perform further changes.

The following permissions are required on the list:

  • CREATE

Exports the current list in CSV format. This will start a download operation for your browser.

The following permissions are required on the list:

  • READ

Creates new objects in the list from a CSV file. The format must comply with the format produced by the export. All imported objects will be added to the list. The import terminates with an error message if an object with the same name already exists and Replace rules is set to false.

Replace rules

The Replace rules option determines whether a rule is appended or replaced. If set to true, all current rules will be replaced with the new rules, otherwise the new rules are appended to the existing rules.

The following permissions are required on the list:

  • WRITE

The history list tracks all modifications made to objects within it. A new record is added each time an object is created, edited, or deleted. A record indicates who made the change, which object was affected, and when the change was made.

For more information about the concept of the history refer to the history concepts.

The following permissions are required on the list:

  • READ

Opens a new view on the list that allows the user to specify the ordering of the elements using drag and drop. Confirm the new ordering with Save Changes.

The following permissions are required on the list:

  • READ

  • WRITE

Object Actions

The following actions are available on specific modification rules. In order to execute the action, the user must possess the necessary permissions for the object. The permissions required for each action are described individually. If the user does not have these permissions, the action will be disabled and the tooltip will provide the exact reason for the deactivation.

  • Check

  • Copy

  • Delete

  • Edit

  • Object History

This action validates the object and its dependencies, reporting configuration errors that could cause issues during task or workflow execution. The validation will cascade through the child objects of the checked objects and objects referenced by them.

For instance, if an installed application of an environment is checked, the check will process the application model, the specified version, the connection, modification sets, and involved modification methods. If an object has rules, all active rules will be checked. The modeling connection and version, including their modification sets and methods, will also be checked. Deactivated objects will not be included in recursive checks, but can be checked individually if the check is executed on the object itself.

Checks often require additional information from the context of the objects being checked, such as necessary connections or custom parameter values. The check will gather information from the objects being checked and use it to perform checks on child objects. Any required additional information must be provided before the check begins. The check queries the user to provide these missing information.

Database object checks

For all rules which reference database objects such as tables, columns, etc, the check verifies that the those objects exist in the database system. If a connection can be inferred from the context, then this connection is used. If no connection is available in the context, it must be specified before the check is executed.

Connection checks

For objects which configure access to external systems, such as connections or storage locations, the configuration check verifies that access can be established using the given credentials. Furthermore, additional operations on database connections are performed to check whether the credential user has the necessary authorization to access relevant database objects. In particular, the credential user’s permission to read source tables and write to target tables is verified. Similarly, for storage locations the check verifies that the credential user has permission to write to the working directory.

Code checks

For all entities containing code segments, such as modification methods or condition scripts, the syntax for the code is checked. This does not check, however, whether at run time all necessary variables are likely to be available.

The following permissions are required:

  • READ

The copy action creates an identical copy of the object. A new entry is created in the object list and all properties in the new object are set identical to the copied object.

The following permissions are required:

  • READ

  • WRITE

Delete the object. If the object is still used by another entity, an error message is displayed, and the object is not deleted. The delete operation must be confirmed in a separate popup.

The following options are available:

Cascade

Recursively delete depending objects.

When using cascade, dependent objects are deleted first also with cascade enabled. Thus, a cascade deletion is a recursive function that deeply searches for dependent objects and deletes them first. There is only a confirmation for the first object. The dependent objects are deleted without confirmation but only when the user has the DELETE permission.

This feature is only available in development mode. More information about development mode can be found in the chapter User Settings. It should be used with caution.

The following permissions are required:

  • READ

  • WRITE

Opens the current entity in edit mode.

The following permissions are required:

  • READ

  • WRITE

The history displays all changes made to the respective XDM object, including any changes made to its rules.

Each change record includes information about the operation performed (e.g. CREATE, UPDATE, DELETE), the timestamp, and the user responsible for the change.

For more information about the concept of the history refer to the history concepts.

The following permissions are required:

  • READ

Usage Details

The order of the modification rules influences the result of modification. If two or more rules modify the same value, all rules are nevertheless applied in ascending order. If modification rules are defined on multiple modification sets, the rule defined in a set more specific for the task execution will be applied later. For example, a rule in a modification set of the task will be applied after a modification rule defined in a set on the connection.

When using modification for a column, you should ensure that the data can be copied to the target table. For example, when shrinking data using modification, in order to avoid compatibility errors resulting from truncation, ensure that the Toleration Mode is set to Allow truncation.
Examples

Typical scenarios for using modification rules are:

  1. You want to replace personally identifiable data in a column of the source table with random data. Assume you have a column LASTNAME, which contains a person’s last name. To replace the data you can add a modification rule for LASTNAME and associate it with a modification method that generates a random string. A modification method that generates a random string typically takes at least one parameter defining the maximum length of the resulting string. You can set this parameter to the maximum length of the column. The resulting random string will never exceed the set length. After the copy, the target table will contain random strings in the column that contains last names in the source environment.

  2. You want to generate data for a column that only exists in the target environment, such as an e-mail address, based on the corresponding person’s first and last names. You can add a modification rule to the modification set used by the task template that matches the table and column that contains the e-mail address in the target environment by setting the option Scope to Target. This modification method could compose a seemingly valid e-mail address by combining the column values for the person’s first and last names with the suffix @example.com (or similar).