XDM3 configuration examples
Matcher
Modification methods
Read the chapter modification method for more information about modification methods. Information about the concept of modification methods can be found in the modify data chapter of Working with XDM.
Some modification methods need the lookup tables to be uploaded in your XDM environment as mapping table XDM Lookup Table.
Read the chapter Installation of lookup tables
for more information.
|
Auto increment
- Description
-
This method generates a new value each time it is called. Internally it uses a database SEQUENCE that is responsible to provided incrementing values.
- Input type
-
COLUMN
- Parameters
| Display name | Description | Variable name | Data type | Default value | Possible values |
|---|---|---|---|---|---|
Sequence Name |
Specifies the name of the sequence. |
sequenceName |
STRING |
AUTO_INCREMENT |
- |
IncrementBy |
Specifies the step size of the sequence. Negative values will lead to a down counting sequence. |
incrementBy |
STRING |
- |
- |
Start Value |
Specifies the value the sequence is starting with. |
startValue |
STRING |
- |
- |
Min Value |
Specifies the lower limit of the sequence. This will be used as start value or at cycling. |
minValue |
STRING |
- |
- |
Max Value |
Specifies the maximum value for the sequence. This will be used as start value for down counting sequences. |
maxValue |
STRING |
- |
- |
Cycle |
Specifies whether the sequence cycles at the borders or causing an error. |
cycle |
BOOLEAN |
FALSE |
- |
Drop Sequence |
Specifies if the sequence is dropped at the end of the modification method finishes. |
dropSequence |
BOOLEAN |
FALSE |
- |
Bank identifier code
- Description
-
Generates a bank identifier code (also known as ISO 9362 or SWIFT code) based on the seed parameter. 'RAND' will generate a random value. All other values are used as they are for the seed. The country parameter specifies the language of origin for the first name ('AT', 'CH', 'DE', 'NL'). It is possible to specify one to all of them separated by a comma. If more than one value is specified one of them is used based on the seed value. 'RAND' can be used to use a random value. This method requires the installation of the lookup tables.
- Input type
-
COLUMN
- Parameters
| Display name | Description | Variable name | Data type | Default value | Possible values |
|---|---|---|---|---|---|
Seed |
Column that is used to calculate the hash. |
seed |
STRING |
- |
- |
Country |
Specifies the origin of the bank identifier code. |
country |
STRING |
- |
- |
Bank name
- Description
-
Generates a bank name based on the seed parameter. 'RAND' will generate a random value. All other values are used as they are for the seed. The country parameter specifies the language of origin for the first name ('AT', 'CH', 'DE', 'NL'). It is possible to specify one to all of them separated by a comma. If more than one value is specified one of them is used based on the seed value. 'RAND' can be used to use a random value. This method requires the installation of the lookup tables.
- Input type
-
COLUMN
- Parameters
| Display name | Description | Variable name | Data type | Default value | Possible values |
|---|---|---|---|---|---|
Seed |
Column that is used to calculate the hash. |
seed |
STRING |
- |
- |
Country |
Specifies the origin of the bank name. |
country |
STRING |
- |
- |
Canadian social insurance number
- Description
-
Generates a Canadian social insurance number (SIN) based on the seed parameter. The generated SIN is returned as a string. Leading zeroes are preserved. 'RAND' will generate a random value. All other values are used as they are for the seed.
- Input type
-
COLUMN
- Parameters
| Display name | Description | Variable name | Data type | Default value | Possible values |
|---|---|---|---|---|---|
Seed |
Specify a value that is used to calculate the hash. |
seed |
STRING |
- |
- |
Delimiter |
The delimiter character used to separate the groups of digits. May be empty. |
delimiter |
STRING |
- |
- |
City
- Description
-
Generates a city based on the seed parameter. 'RAND' will generate a random value. All other values are used as they are for the seed. The country parameter specifies the language of origin for the first name ('US', 'DE', 'FR', 'IT', 'NL', 'AT', 'BE', 'CA', 'CH', 'UK'). It is possible to specify one to all of them separated by a comma. If more than one value is specified one of them is used based on the seed value. 'RAND' can be used to use a random value. This method requires the installation of the lookup tables.
- Input type
-
COLUMN
- Parameters
| Display name | Description | Variable name | Data type | Default value | Possible values |
|---|---|---|---|---|---|
Seed |
Column that is used to calculate the hash. |
seed |
STRING |
- |
- |
Country |
Specifies the origin of the city. |
country |
STRING |
- |
- |
Credit card number
- Description
-
Generates a credit card number. The first parameter should be an existing credit card number. The second parameter specifies how many digits should be copied without modification from the original number.
- Input type
-
COLUMN
- Parameters
| Display name | Description | Variable name | Data type | Default value | Possible values |
|---|---|---|---|---|---|
Seed |
The original credit card number. |
seed |
STRING |
- |
- |
Retain |
The number of digits to retain from the original number. |
retain |
NUMBER |
- |
- |
Czech birthday number
- Description
-
Generates a Czech birthday number based on the birthday and sex parameter. The generated number is returned as a string. 'RAND' will generate a random value. If the value is a date it will be used as it is. All other values will be hashed.
- Input type
-
COLUMN
- Parameters
| Display name | Description | Variable name | Data type | Default value | Possible values |
|---|---|---|---|---|---|
Birthday |
Specify a value that is used to calculate the hash. |
birthday |
STRING |
- |
- |
Sex |
Specifies the gender of the first name. |
sex |
STRING |
- |
- |
Delimiter |
The delimiter character used to separate the groups of digits. May be empty. |
delimiter |
STRING |
/ |
- |
E Mail
- Description
-
Generates an email address based on the seed parameter 'RAND' will generate a random value. All other values are used as they are for the seed. The country parameter specifies the language of origin for the first name ('EN','DE','FR','IT', or 'NL'). It is possible to specify one to all of them separated by a comma. If more than one value is specified one of them is used based on the seed value. 'RAND' can be used to use a random value. The sex parameter specifies the sex (‘M’ for male, ‘F’ for female). It is possible to specify on or both separated by comma. If more than one value is specified one of them is used based on the seed value. 'RAND' can be used to use a random value. The domain parameter specifies the domain name for the email address. If the domain parameter is omitted, it defaults to “example.com”.
- Input type
-
COLUMN
- Parameters
| Display name | Description | Variable name | Data type | Default value | Possible values |
|---|---|---|---|---|---|
Seed |
Specifies the value that is used to calculate the hash. |
seed |
STRING |
- |
- |
Country |
Specifies the origin of the email. |
country |
STRING |
- |
- |
Sex |
Specifies the gender of the first name. |
sex |
STRING |
- |
- |
Domain |
Domain that is used to generate e-mail addresses. |
domain |
STRING |
example.com |
- |
First name
- Description
-
Generates a first name based on the seed parameter. 'RAND' will generate a random value. All other values are used as they are for the seed. The country parameter specifies the language of origin for the first name ('EN','DE','FR','IT', or 'NL'). It is possible to specify one to all of them separated by a comma. If more than one value is specified one of them is used based on the seed value. 'RAND' can be used to use a random value. The sex parameter specifies the sex (‘M’ for male, ‘F’ for female). It is possible to specify on or both separated by comma. If more than one value is specified one of them is used based on the seed value. 'RAND' can be used to use a random value. This method requires the installation of the lookup tables.
- Input type
-
COLUMN
- Parameters
| Display name | Description | Variable name | Data type | Default value | Possible values |
|---|---|---|---|---|---|
Seed |
Specifies the value that is used to calculate the hash. |
seed |
STRING |
- |
- |
Country |
Specifies the origin of the first name. |
country |
STRING |
- |
- |
Sex |
Specifies the gender of the first name. |
sex |
STRING |
- |
- |
IMEI
- Description
-
Generates an IMEI based on an existing IMEI and the seed parameter.'RAND' will generate a random value. All other values are used as they are for the seed.
- Input type
-
COLUMN
- Parameters
| Display name | Description | Variable name | Data type | Default value | Possible values |
|---|---|---|---|---|---|
Seed |
Specifies the value that is used to calculate the hash. |
seed |
STRING |
- |
- |
International bank account number
- Description
-
Generates an international bank account number (IBAN) based on the seed parameter. 'RAND' will generate a random value. All other values are used as they are for the seed. The country parameter specifies the language of origin for the first name ('AT', 'CH', 'DE', 'NL'). It is possible to specify one to all of them separated by a comma. If more than one value is specified, one of them is used based on the seed value. 'RAND' can be used to use a random value. This method requires the installation of the lookup tables.
- Input type
-
COLUMN
- Parameters
| Display name | Description | Variable name | Data type | Default value | Possible values |
|---|---|---|---|---|---|
Seed |
Column that is used to calculate the hash. |
seed |
STRING |
- |
- |
Country |
Specifies the origin of the international bank account number. |
country |
STRING |
- |
- |
Last name
- Description
-
Generates a last name based on the seed parameter. 'RAND' will generate a random value. All other values are used as they are for the seed. The country parameter specifies the language of origin for the first name ('EN','DE','FR','IT', or 'NL'). It is possible to specify one to all of them separated by a comma. If more than one value is specified one of them is used based on the seed value. 'RAND' can be used to use a random value. This method requires the installation of the lookup tables.
- Input type
-
COLUMN
- Parameters
| Display name | Description | Variable name | Data type | Default value | Possible values |
|---|---|---|---|---|---|
Seed |
Specifies the value that is used to calculate the hash. |
seed |
STRING |
- |
- |
Country |
Specifies the origin of the last name. |
country |
STRING |
- |
- |
Latitude
- Description
-
Generates the latitude component of geographical coordinates based on the seed parameter. 'RAND' will generate a random value. All other values are used as they are for the seed. The country parameter specifies the language of origin for the first name ('US', 'DE', 'FR', 'IT', 'NL', 'AT', 'BE', 'CA', 'CH', 'UK'). It is possible to specify one to all of them separated by a comma. If more than one value is specified one of them is used based on the seed value. 'RAND' can be used to use a random value. This method requires the installation of the lookup tables.
- Input type
-
COLUMN
- Parameters
| Display name | Description | Variable name | Data type | Default value | Possible values |
|---|---|---|---|---|---|
Seed |
Column that is used to calculate the hash. |
seed |
STRING |
- |
- |
Country |
Specifies the origin of the latitude. |
country |
STRING |
- |
- |
Longitude
- Description
-
Generates the longitude component of geographical coordinates based on the seed parameter. 'RAND' will generate a random value. All other values are used as they are for the seed. The country parameter specifies the language of origin for the first name ('US', 'DE', 'FR', 'IT', 'NL', 'AT', 'BE', 'CA', 'CH', 'UK'). It is possible to specify one to all of them separated by a comma. If more than one value is specified one of them is used based on the seed value. 'RAND' can be used to use a random value. This method requires the installation of the lookup tables.
- Input type
-
COLUMN
- Parameters
| Display name | Description | Variable name | Data type | Default value | Possible values |
|---|---|---|---|---|---|
Seed |
Column that is used to calculate the hash. |
seed |
STRING |
- |
- |
Country |
Specifies the origin of the longitude. |
country |
STRING |
- |
- |
Modify time data
- Description
-
Change a time data value by modifying it or generate or set a new value. Input can be a date, time, timestamp or string column. This method internally parses each of the input values to a timestamp and then allows to modify each part of this timestamp separately. For each of the parts (year, month, day, hour, minute, second, fractions) one of the following values can be specified: * <value> → Set the specified value
-
+<value> → to add the specified amount to this part
-
-<value> → to subtract a specified amount from this part
-
NOW → take the value of the current timestamp
-
NOW+<value> → to add a given amount to the current timestamps part and use it
-
NOW-<value> → to subtract a given amount from the current timestamps part and use it
-
SEED(<min>:<max>) → Generate a new value for this part based on a given seed. The seed can be a column name of the table to use the columns value. Otherwise the seed value will be used as it is. The part in brackets is optional and can limit the range of the generated value. <min> → specifies the low limit for the generated value. <max> → specifies the high limit for the generated value. Both <min> and <max> are optional. If not specified the default value for the limit will be used. The seed value has to be specified in parameter seed.
-
+SEED(<min>:<max>) → Generate a new value like SEED(<min>:<max>) but this value will be added to the current value of the part.
-
-SEED(<min>:<max>) → Generate a new value like SEED(<min>:<max>) but this value will be subtracted from the current value of the part.
-
RAND(<min>:<max>) → Generate a new random value. The part in brackets is optional and can limit the range of the generated value. <min> → specifies the low limit for the generated value. <max> → specifies the high limit for the generated value. Both <min> and <max> are optional. If not specified the default value for the limit will be used. The default limits depend on the part of the timestamp.
-
+RAND(<min>:<max>) → Generate a new value like RAND(<min>:<max>) but this value will be added to the current value of the part.
-
-RAND(<min>:<max>) → Generate a new value like RAND(<min>:<max>) but this value will be subtracted from the current value of the part.
-
- Input type
-
COLUMN
- Parameters
| Display name | Description | Variable name | Data type | Default value | Possible values |
|---|---|---|---|---|---|
Year |
The rule or value that will be used to build a new year value for the given time data. The possible rules are described in the methods descriptions. |
year |
STRING |
- |
- |
Month |
The rule or value that will be used to build a new month value for the given time data. The possible rules are described in the methods descriptions. |
month |
STRING |
- |
- |
Day |
The rule or value that will be used to build a new day value for the given time data. The possible rules are described in the methods descriptions. |
day |
STRING |
- |
- |
Hour |
The rule or value that will be used to build a new hour value for the given time data. The possible rules are described in the methods descriptions. |
hour |
STRING |
- |
- |
Minute |
The rule or value that will be used to build a new minute value for the given time data. The possible rules are described in the methods descriptions. |
minute |
STRING |
- |
- |
Second |
The rule or value that will be used to build a new second value for the given time data. The possible rules are described in the methods descriptions. |
second |
STRING |
- |
- |
Fraction |
The rule or value that will be used to build a new fraction value for the given time data. The possible rules are described in the methods descriptions. |
fraction |
STRING |
- |
- |
Seed |
Specify the seed which can be used to generate new values for the time data parts. If a column name is specified the value of this column will be used as seed. |
seed |
STRING |
- |
- |
FormatPattern |
When the input value is a string this pattern will be used to parse the value of this string into a timestamp and to format the output into the same format. |
formatPattern |
STRING |
yyyy-MM-dd-HH.mm.ss |
- |
FractionPattern |
When the input value is a string this pattern will be used to parse the value of this string into the fractions part of the timestamp and to format the output into the same format. |
fractionPattern |
STRING |
.SSSSSSSSSSSSZ |
- |
IgnoreNullInputs |
Specify if null value inputs should be ignored. If set to false new values are generated for null inputs. |
ignoreNull |
BOOLEAN |
true |
- |
National bank identifier code
- Description
-
Generates a national bank identifier code (also known as Bankleitzahl in Austria and Germany, and as Instituts-Identifikation or Bankclearing-Nummer in Switzerland) based on the seed parameter. 'RAND' will generate a random value. All other values are used as they are for the seed. The country parameter specifies the language of origin for the first name ('AT', 'CH', 'DE', 'NL'). It is possible to specify one to all of them separated by a comma. If more than one value is specified one of them is used based on the seed value. 'RAND' can be used to use a random value. This method requires the installation of the lookup tables.
- Input type
-
COLUMN
- Parameters
| Display name | Description | Variable name | Data type | Default value | Possible values |
|---|---|---|---|---|---|
Seed |
Column that is used to calculate the hash. |
seed |
STRING |
- |
- |
Country |
Specifies the origin of the national bank identifier code. |
country |
STRING |
- |
- |
Pattern
- Description
-
Generates a sequence of letters and digits according to a pattern. The letter A is replaced by a letter between A and Z. The digit 9 is replaced by a digit between 0 and 9. The letter N is replaced by either a letter or a digit. The letter X is replaced by a hexadecimal digit (0 through F). All other characters are preserved as they are.
- Input type
-
COLUMN
- Parameters
| Display name | Description | Variable name | Data type | Default value | Possible values |
|---|---|---|---|---|---|
Seed |
Specifies the value that is used to calculate the hash. |
seed |
STRING |
- |
- |
Pattern |
The pattern string based upon which the resulting sequence of letters and digits will be calculated. |
pattern |
STRING |
- |
- |
Postal Code
- Description
-
Generates a postal code based on the seed parameter. 'RAND' will generate a random value. All other values are used as they are for the seed. The country parameter specifies the language of origin for the first name ('US', 'DE', 'FR', 'IT', 'NL', 'AT', 'BE', 'CA', 'CH', 'UK'). It is possible to specify one to all of them separated by a comma. If more than one value is specified one of them is used based on the seed value. 'RAND' can be used to use a random value. This method requires the installation of the lookup tables.
- Input type
-
COLUMN
- Parameters
| Display name | Description | Variable name | Data type | Default value | Possible values |
|---|---|---|---|---|---|
Seed |
Column that is used to calculate the hash. |
seed |
STRING |
- |
- |
Country |
Specifies the origin of the postal code. |
country |
STRING |
- |
- |
Set Value
- Description
-
Sets the specified value to the column.
- Input type
-
COLUMN
- Parameters
| Display name | Description | Variable name | Data type | Default value | Possible values |
|---|---|---|---|---|---|
New Value |
The new value to be set. |
newValue |
STRING |
- |
- |
State
- Description
-
Generates an administrative country subdivision name based on the seed parameter. 'RAND' will generate a random value. All other values are used as they are for the seed. The country parameter specifies the language of origin for the first name ('US', 'DE', 'FR', 'IT', 'NL', 'AT', 'BE', 'CA', 'CH', 'UK'). It is possible to specify one to all of them separated by a comma. If more than one value is specified one of them is used based on the seed value. 'RAND' can be used to use a random value. This method requires the installation of the lookup tables.
- Input type
-
COLUMN
- Parameters
| Display name | Description | Variable name | Data type | Default value | Possible values |
|---|---|---|---|---|---|
Seed |
Column that is used to calculate the hash. |
seed |
STRING |
- |
- |
Country |
Specifies the origin of the state name. |
country |
STRING |
- |
- |
Street
- Description
-
Generates a street name based on the seed parameter. 'RAND' will generate a random value. All other values are used as they are for the seed. The country parameter specifies the language of origin for the first name ('US', 'DE', 'FR', 'IT', 'NL', 'AT', 'BE', 'CA', 'CH', 'UK'). It is possible to specify one to all of them separated by a comma. If more than one value is specified one of them is used based on the seed value. 'RAND' can be used to use a random value. This method requires the installation of the lookup tables.
- Input type
-
COLUMN
- Parameters
| Display name | Description | Variable name | Data type | Default value | Possible values |
|---|---|---|---|---|---|
Seed |
Column that is used to calculate the hash. |
seed |
STRING |
- |
- |
Country |
Specifies the origin of the street name. |
country |
STRING |
- |
- |
Swedish id
- Description
-
Generates a Swedish id based on the birthday parameter. The generated ID is returned as a string. 'RAND' will generate a random value. If the value is a date it will be used as is. All other values will be hashed.
- Input type
-
COLUMN
- Parameters
| Display name | Description | Variable name | Data type | Default value | Possible values |
|---|---|---|---|---|---|
Birthday |
Specify a value that is used to calculate the hash. |
birthday |
STRING |
- |
- |
Delimiter |
The delimiter character used to separate the groups of digits. May be empty. |
delimiter |
STRING |
- |
- |
U.S. social security number
- Description
-
Generates a U.S. social security number (SSN) based on the first parameter. The generated SSN is returned as a string. Leading zeroes are preserved. 'RAND' will generate a random value. All other values are used as they are for the seed.
- Input type
-
COLUMN
- Parameters
| Display name | Description | Variable name | Data type | Default value | Possible values |
|---|---|---|---|---|---|
Seed |
Specify a value that is used to calculate the hash. |
seed |
STRING |
- |
- |
Delimiter |
The delimiter character used to separate the groups of digits. May be empty. |
delimiter |
STRING |
- |
- |
UUID
- Description
-
Generates a version 4 universally unique identifier (UUID) based on the seed parameter. 'RAND' will generate a random value. All other values are used as they are for the seed.
- Input type
-
COLUMN
- Parameters
| Display name | Description | Variable name | Data type | Default value | Possible values |
|---|---|---|---|---|---|
Seed |
Specifies the value that is used to calculate the hash. |
seed |
STRING |
- |
- |
Set value with a modification method for an additional column in a target table which is not nullable
Example name |
Set value for additional not nullable column in target |
Version |
3.22.39 |
Download link |
- Description
-
This method deals with a situation where a target table has one or more columns that do not exist in the source table. If such a column is neither nullable nor has a database defined default value, then copying from the source table to the target would fail. The example shows how to use modification to fill an additional target column with a dummy email address.
- Code Type
-
Java script
Restart Sequence
Example name |
Restart Sequence |
Version |
3.22.41 |
Download link |
- Description
-
This modification method reads the highest value of the selected column of the table and adds 1 to this value. Then the selected column of the target table restarts with this value. On this way it is avoided that the current sequence number will ever be reached what would cause loss of data. You have to select the primary key-column in the linked modification set for the correct usage.
Trim Strings
Example name |
Trim Strings |
Version |
3.22.41 |
Download link |
- Description
-
This modification method eliminates leading and trailing spaces from the value to be modified.
German tax identification number/TIN
Example name |
German tax identification number/TIN |
Version |
3.23.15 |
Download link |
- Description
-
This modification method works column based. The input is the value of the current column, which is used to generate a number looking like the german tax identification number, which has to fulfill the following requirements:
-
The first number must not be 0
-
In the first 10 digits there is one number double or triple
-
If there are 3 same numbers at the position 1 to 10 those double numbers could never be consecutive
-
Information about how the check digit is generated and the criteria of a valid number: www.zfa.deutsche-rentenversicherung-bund.de (PDF)
Background information about the number: www.bzst.de
"Zulagennummer"/Allowance number
Example name |
Zulagennummer/Allowance number |
Version |
3.23.13 |
Download link |
- Description
-
Builds a "Zulagennummer" (allowance number) based on the following input parameters:
-
number of the insurer (place 1-2)
-
date of birth of the insured person (place 3-8)
-
family name of the insured person (place 9)
-
gender, 00-49 male, 50-99 female (place 10-11)
-
check digit (place 12)
-
The allowance number is equal to the german social security number and is e. g. important to determine the pension income of an employee, to manage health insurance, as well as other issues concerning social security. The number is normally assigned to an employee via the first paycheck. Some employees like public officials, judges and soldiers have to request their allowance number manually. The allowance number is used to determine the increment of the German Riester pension (german subsidized private pension).
- Parameters
Display name |
Description |
Variable name |
Data type |
Default value |
Possible values |
Area Number |
Number of the insurer |
areaNumber |
STRING |
- |
- |
Birth Date |
Birth date in the format |
birthName |
STRING |
- |
- |
Last Name |
Last name at the time of the application |
lastName |
STRING |
- |
- |
Gender |
00-49 for male, 50-99 for female |
gender |
STRING |
- |
- |
Task execution reports
Read the chapter task execution report for more information about task execution reports.
Classification term report
- Description
-
The above example shows what data was copied during the execution of a task based on classification terms. This report shows the name and values of the copied classification terms.
Summary Report
Example name |
Summary Report |
Version |
3.24.15 |
Download link |
- Description
-
The above example shows how much data was extracted, applied, deleted, discarded, or modified when copying from source to target.
Classification Term Detail Report
Example name |
Classification Term Detail Report |
Version |
3.23.15 |
Download link |
- Description
-
This report shows which data was copied during a task execution based on classification terms. In comparison to the classification term report two columns have been added in this report. Through this you can also see the name of the task template and the task that used the classification term. With these names the task type can be identified. Through the task type it can be determined whether the classification term value has been written, read or deleted.
So for example it can be determined that row level delete task
Ahas deleted a classification term my term which has the value 5 and a row level processor taskBhas written a classification term your term which has the value 6.
Task stage hooks
Read the chapter xre:Reference/TaskStageHook.adoc#TaskStageHook[task stage hook] for more information about task stage hooks. Information about the concept of a task stage hook can be found in the task stage hooks chapter of Working with XDM.
Using column property rules
Example name |
Column Property Rules |
Version |
3.23.27 |
Download link |
- Description
-
This stage hook enables the usage of a column property rule which is set to specific tables or columns. This task stage hook sets the ability to copy
VARCHARcolumns as binary with less degradation. In the aforementioned situation, the binary data was stored as aVARCHARin the database and was henceforth seen as aSTRINGin the internal xdm processes.
| Table schema, table name, and column name have to be specified explicitly in the task stage hook. |
This hook is used in the Pre Hook phase of Stage1 where the object structure is fetched and leads to a different internal treatment for the specified column.
- Code Type
-
Groovy
Delete expired tasks
- Description
-
This task stage hook deletes tasks of the same task template that exceed the specified retention count or retention date. By default, the hook only deletes these tasks that have been implicitly created either by the execution of a data shop, or while executing a task template. Both actions implicitly create a new task in the background. To avoid that the list of created tasks grows and grows you can automatically delete tasks that have not been executed for some time. This hook uses an internal API. It is maintained by UBS Hainer
- Code Type
-
GROOVY
- Parameters
| Display name | Description | Data type | Default value |
|---|---|---|---|
taskRetentionPeriod |
Specifies a period of time after tasks are deleted. |
STRING |
- |
taskRetentionCount |
The number of tasks that should be kept. If more than the specified amount of tasks exists the oldest executed tasks will be deleted. A value of zero defines that no tasks will be deleted. |
NUMBER |
- |
implicitOnly |
Specifies that only implicit tasks should be dropped, or even manually created tasks. Implicit tasks are created by a data shop execution, or by the Jenkins plugin. |
BOOLEAN |
true |
Delete expired workflows
- Description
-
This task stage hook deletes workflows of the same workflow template that exceed the specified retention count or retention date. By default the hook only deletes these workflows that have been implicitly created either by the execution of a data shop, or while execution a workflow template. Both actions implicitly create a new workflow in the background. To avoid that the list of created workflows grows you can automatically delete workflows that have been executed for some time. This hook uses an internal API. It is maintained by UBS Hainer
- Code Type
-
GROOVY
- Parameters
| Display name | Description | Data type | Default value |
|---|---|---|---|
workflowRetentionPeriod |
Specifies a period of time after workflows are deleted. |
STRING |
- |
workflowRetentionCount |
The number of workflows that should be kept. If more than the specified amount of workflows exists the oldest executed workflows will be deleted. A value of zero defines that no workflows will be deleted. |
NUMBER |
- |
implicitOnly |
Specifies that only implicit workflows should be dropped, or even manually created workflows. Implicit workflows are created by a data shop, or by the Jenkins plugin. |
BOOLEAN |
true |
Delete icebox generation
- Description
-
This can be called in a From IceBox task and deletes the generation that has been restored by the task. The generation is deleted by calling the REST API of the core server. It includes the deletion of the entry in the user interface and the data of the generation This hook uses an internal API. It is maintained by UBS Hainer
- Code Type
-
Java script
Extract icebox generation data
Example name |
Extract icebox generation properties |
Version |
3.23.41 |
Download link |
- Description
-
This example can be used to extract all icebox generation parameters and copy it into the task’s runtime properties. The parameter names have the prefix
ig_. These parameters can be used withctx.getTaskRuntimeProperty(…), for example in modification methods.
JavaScript in Ant Task
Example name |
JavaScript in Ant Task |
Version |
3.22.29 |
Download link |
- Description
-
When using an ant stage hook, it might be useful to execute a JavaScript statement to execute a change. In these case, you can simply include JavaScript in the ant task as this executable example.
Property Print
Example name |
Property Print |
Version |
3.22.33 |
Download link |
- Description
-
To see all the properties which are actually available in the property-files, you can use this stage hook. Within the log file of the stage hook the properties are printed.
Print found start keys with stage hook
Example name |
Start keys print |
Version |
3.23.17 |
Download link |
- Description
-
Row level processing tasks uses start keys to identify the database records that are to be copied during the task run. These keys can be viewed after the task export under
files/startValues.csvIt is possible that keys are specified that do not exist in the database. This hook prints the keys which were found in the database at the start of the task run. The result can be viewed in the task execution where the task stage hook is used. This can be used to analyze e.g why perhaps no data was copied, or to process start keys in another task stage.
Send E-Mail
Example name |
Send Email |
Version |
3.23.13 |
Download link |
- Description
-
Using this stage hook in tasks or workflows, you can send e-mails that can include information about this task or workflow. It is also possible to attach files to the mail. Therefore the file must be stored in the directory
xdm-data. To attach a file, the complete filename has to be written into the parameterattachmentFile. If this parameter is empty no file is attached.
It is possible to use wildcards in the parameter attachmentFile, but it is not possible to attach several files
by comma separation.
|
If no toEmail address is specified, the requested-by user is used in a data shop, otherwise the started-by user is used.
- IMPORTANT
-
The placeholder
@xdm.dein line 18 of thesend-email.yamlhas to be replaced by the domain of the company. - Code Type
-
ANT
- Parameter list
-
- mailPort
-
Specifies the mail port number.
Parameter name:
mailPort
Type:
Number
Default value:
25
- mailHost
-
Specifies the host name of the smtp server.
Parameter name:
mailHost
Type:
String
Default value:
-
- toEmail
-
Specifies the mail address of the receiver.
Parameter name:
toEmail
Type:
String
Default value:
-
- fromEmail
-
Specifies the mail address of the sender.
Parameter name:
fromEmail
Type:
String
Default value:
- message
-
Specifies the message to send in the mail.
Parameter name:
message
Type:
String
Default value:
XDM Task Execution
- subject
-
Specifies the subject of the mail.
Parameter name:
subject
Type:
String
Default value:
XDM Task Execution
- attachmentFile
-
The complete file name of the file that should be attached has to be specified.
Parameter name:
attachmentFile
Type:
String
Default value:
-
Create Mapping Rule
| Example name | Create Mapping Rule |
|---|---|
Version |
3.23.45 |
Download link |
- Description
-
Sometimes in RLP tasks the properties of source and target have to be changed manually e.g. when table names or column names differ. To change the names, a mapping rule is needed. This mapping rule can be created with the help of this stage hook. Every mapping described in the chapter "Mapping Rules" can be used except
objectType = "ANY"andfieldType = "SCHEMA"because the schema is handled by the installed application.
The generated mapping rule is applied for all installed applications.
- parameter list
| Parameter name | Type | Default value |
|---|---|---|
objectType |
String |
TABLE |
sourceSelectionPattern |
String |
% |
targetValue |
String |
- |
fieldType |
String |
NAME |
fieldSelectionMode |
String |
Simple |
Read a CSV File from Server
Example name |
Read a CSV File from Server |
Version |
3.22.43 |
Download link |
- Description
-
This stage hook reads the content of a csv file from the server where XDM is installed on and prints it into the log of the hook. Thereby specific values can be identified much easier. The file is an external file which has to be stored in the xdm-data directory. This directory is mounted to the internal path
/xdm/datawithin the docker container. The file is not a XDM internal file.
Read the content of XDM-Files and build a pool table from them
Example name |
Read the content of XDM-Files and build a pool table from them |
Version |
3.22.45 |
Download link |
- Description
-
If you have files with values to lookup in a modification method you can read these files into a h2-table to get quick access to these values within the modification method. Therefore, you only have to create a csv file for each column of the table you need and separate each value with a semicolon. The first file is taken to create the first column of the lookup-table, the second file is taken to create the second column and so on.
This stage hook reads the content of each file and creates a column of the pool table from it. You have to pass each csv file as a parameter to the stage hook. You have to use a connection to connect to the database the pool table is part of. In general any type of connection can be used for this.
In this example CustomerDB is the name of the h2-database, and it’s identical to the name mentioned in the stage hook. In general the name of can be chosen freely. Finally, you can see the finished pool table in the schema browser of the connection.
Read a CSV File and built a table in file bridge from it
Example name |
Read a CSV File and built a table in file bridge from it |
Version |
3.22.49 |
Download link |
- Description
-
The file bridge server is an interface between a CSV file and JDBC. It enables SQL access to data stored in files outside of a relational database system. The column names and data types of the CSV file columns are defined by using create statements for the file bridge server and the file bridge server interprets them as tables and makes them usable via JDBC.
This stage hook builds the CREATE statements from a CSV input file. For this, names and data types
of the columns must be defined in the input file. The first line of the input file contains the column names,
the second line the data type of each column. Possible data types are:
-
CHAR -
VARCHAR -
INTEGER -
SMALLINT -
BIGINT -
FLOAT -
DOUBLE -
BINARY -
DECIMAL -
DATE -
TIME -
TIMESTAMPIf the datatype is
CHAR,VARCHARorDECIMAL, the length of the value has to be set in normal brackets after the datatype definition. All other lines of the input file contain data.
The stage hook extracts the column names and data types of the input file to define and execute CREATE statements
for the file bridge server to define table spaces and tables for the file. Furthermore, it removes the data type
definitions from the input file and write all other lines of the input file in a new output file.
The output file will be used as data source in the file bridge server.
For this, the output folder has to be available to the dataflow-server where the hook runs.
It is possible to process several input files with this hook. The stage hook processes all CSV files stored in the input folder. Furthermore, it processes also all files, which are stored in sub-folders of the input folder.
The table name in the file bridge server is defined as the file name in the input folder.
To avoid duplicate file names, the table name of files in sub-folders contains the sub-folder name and the file name,
concatenated by an underscore (_).
- Parameter list
-
- charDel
-
The character delimiter for
CHARandVARCHARdata types.
Parameter name:
charDel
Type:
String
Default value:
"
- colDel
-
The column delimiter to split your columns in the input CSV file.
Parameter name:
colDel
Type:
String
Default value:
;
- encoding
-
The format in which the csv files are encoded.
Parameter name:
encoding
Type:
String
Default value:
UTF-8
- header
-
Specifies if there are column headings defined in the output file or not.
Parameter name:
header
Type:
Boolean
Default value:
true
- inputFolderName
-
Defines the internal folder mounted in the docker container where the input file is stored. The directory to this path has to be specified in the docker-compose.yml.
Parameter name:
inputFolderName
Type:
String
Default value:
No default set
- jdbcServerConnection
-
Defines the file bridge connection in XDM to get access to the file bridge server.
Parameter name:
jdbcServerConnection
Type:
Connection
Default value:
No default set
- outputFolderName
-
Defines the internal folder mounted in the docker container where the hook writes the output file. The directory to this path has to be specified in the docker-compose.yml.
Parameter name:
outputFolderName
Type:
String
Default value:
No default set
- quoteStrings
-
Indicates that values in string columns are enclosed by the character specified as
charDel.Parameter name:
quoteStrings
Type:
String
Default value:
QUOTE STRINGS
- schemaName
-
The name of the schema in which the table should be located.
Parameter name:
schemaName
Type:
String
Default value:
No default set
Call REST API
Example name |
Call REST API |
Version |
3.23.7 |
Download link |
- Description
-
With XDM it is possible to call any REST API of other applications using a stage hook. In this example, a task in a second XDM environment should be executed via REST API call. The specific execution URL and the URL of the XDM environment have to be given as a parameter.
The stage hook calls the REST API of the XDM environment and executes the task based on the execution URL. Each 10 seconds the stage hook checks the status of the executing task and prints the start time, the execution URL and the current status into the log. When the task execution is finished the end time is also printed in the log. If the task execution aborts the stage hook takes the error message and prints it into the log also.
For more information how the XDM REST API works, see here.
- Parameter list
-
- taskUrl
-
The explicit URL of the task execution you want to execute
Parameter name:
taskUrl
Type:
String
Default value:
-
- restApi
-
The URL of the REST API you want to access to. The server and port have to be included in the URL. The URL has to end with
/api. For examplehttp://<server>:<port>/apiParameter name:
restApi
Type:
String
Default value:
-
Workflows
Read the chapter workflow for more information about workflows. Information about the concept of a task stage hook can be found in the workflows chapter of Working with XDM.
Copy with icebox and delete target
- Description
-
This example workflow calls a to icebox task to save an environment. Then it starts a from icebox task to select the icebox generation to restore. The selection criteria from this generation is given to a delete task to clear the target environment. In the next step a from icebox task is used to clear remaining data in the target environment before the load from the first from icebox task restores the data to the target environment.
Delete object container cache directory
- Description
-
This executable workflow executes a task stage hook which deletes the entire object container cache stored in the database. The cache is stored in the table
batch_object_cachewithin the schema defined by${taskStageHook.xdmSchema}.
| Using this workflow may lead to inconsistencies if tasks using the object container retention period are currently running and possibly writing to the object container table while another task deletes entries from this table using this hook. Please be aware of such scenarios, especially when scheduling this workflow. |
IceBox to cloud migrator
- Description
-
This workflow migrates existing IceBox generations to an AWS S3 cloud storage.
The workflow requires that the following configuration in the XDM properties have been configured:
-
admin.db.url
-
admin.db.user
-
admin.db.password
-
admin.db.schema
-
xdm.blob.store.s3.bucket
-
xdm.blob.store.s3.credentials.accessKey
-
xdm.blob.store.s3.credentials.secretKey
-
xdm.blob.store.s3.region.static
-
xdm.blob.store.s3.endpoint
-
Mapping table container and H2 database migrators
- Description
-
The workflow Migrate Mapping Table Container migrates existing Mapping Table Container to use an actual version of the H2 database.
The workflow just iterates over the Mapping Table Container list and calls for each container the migration hook. The hook will download the database from the Mapping Table Container, migrate the database and will upload it again. Please make sure that the executing user has the appropriate permissions to read and write the mapping table container.
Furthermore, there are two more workflows Migrate H2 files in /xdm/mapping and Migrate H2 databases in /xdm/data.
These will migrate H2 databases found in the directories /xdm/mapping and /xdm/data.
When the databases were created with a user-defined user and password, the user and password must be changed in the workflow.