Rules in XDM

Rules in XDM

XDM tasks and other objects can contain one or more different sets of rules, depending on the task type or object:

Modification rules are bundled in modification sets. All general information for rules are valid for modification rules in sets. The sets can be added to tasks and further objects, as shown below.

The rule sets are evaluated every time an XDM task is executed.

Using rules

Most task types have rules to specify the behavior of the task. For example, the selection of tables to be copied or renamed are represented by rules in a task.

Not every task type uses every possible rule set. For example, a table copy to icebox task only makes a copy of source data into flat files. It does not know into which target environment the data will eventually be restored, and whether the target tables will be renamed or not. Therefore, such a task does not have mapping rules.

Table Using Rules for different task types and objects shows which task type uses which rules.

Table 1. Using Rules for different task types and objects
Task type Selection Exclude Mapping Modification Sets Reduction Data Relation Data Apply Extract data Column Exclude Ignore Difference Column Property Column Specification

Native table copy templates

Native table to icebox task templates

Native table from icebox task templates

Compatibility table copy templates

Compatibility table to icebox task templates

Compatibility table from icebox task templates

Structure compare task templates

Structure to icebox task templates

Structure compare with icebox task templates

Row level processor task templates

Row level to icebox task templates

Row level from icebox task templates

Row level delete task templates

PII finder task templates

Test data indexing task templates

()[1]

Connections

Environments

Installed applications

Application models

Application model versions

For tasks, the same rules are available as for task templates.

Every rule can be deactivated. In this case, the rule is omitted when the task is executed.

Using regular expressions in a rule’s selection pattern

When using selection patterns in rules, XDM only provides % and _ as wild card symbols. However, in some cases it is more useful to work with regular expressions for the schema, table, or column pattern. This snippet shows how regular expressions can be used in selection patterns.

Rules in which regular expressions can be used

Regular expressions ca be used in the following parameters:

Rule Type Possible use of regular expressions

Mapping Rule

Source Selection Pattern

Modification Rule

Schema Pattern, Name Pattern, Column Pattern

Reduction Rule

Schema Pattern, Name Pattern, Column Pattern

Column Exclude Rule

Schema Pattern, Name Pattern, Column Pattern

Column Specification Rule

Schema Pattern, Name Pattern, Column Pattern

Extract Data Rule

Schema Pattern, Name Pattern, Column Pattern

User Defined Primary Key

Schema Pattern, Name Pattern

How to use regular expressions in XDM

XDM interprets most patterns internally as Java regular expressions. Hence, these patterns can also be specified as regular expression in the XDM UI.

When a regular expression is used in a pattern, it must be surrounded with /. For example, the expression (FIRST|LAST)NAME must be written as /(FIRST|LAST)NAME/. The / at the beginning and at the end will be trimmed automatically by XDM. They will not be interpreted as part of the regular expression.

It is not possible to use the wildcards % and _ within a regular expression. Therefore .* instead of the wildcard % and . instead of the wildcard _ must be used.

A short example

The following example describes how this works with a modification rule applied to an EMPLOYEE table.

CREATE TABLE TEST.EMPLOYEE (
    ID INTEGER NOT NULL PRIMARY KEY,
    FIRSTNAME VARCHAR(20),
    LASTNAME VARCHAR(20),
    COMPANYNAME VARCHAR(20));

Some sample data for the table:

INSERT INTO TEST.EMPLOYEE (ID, FIRSTNAME, LASTNAME, COMPANYNAME)
	VALUES (1, 'Peter', 'Müller', 'Deutsche Bahn');
INSERT INTO TEST.EMPLOYEE (ID, FIRSTNAME, LASTNAME, COMPANYNAME)
	VALUES (2, 'Hans', 'Schneider', 'Stadt Lauterbach');
INSERT INTO TEST.EMPLOYEE (ID, FIRSTNAME, LASTNAME, COMPANYNAME)
	VALUES (3, 'Petra', 'Schneider', 'Stadt Lauterbach');
INSERT INTO TEST.EMPLOYEE (ID, FIRSTNAME, LASTNAME, COMPANYNAME)
	VALUES (4, 'Fritz', 'Mayer', 'Deutsche Post');
INSERT INTO TEST.EMPLOYEE (ID, FIRSTNAME, LASTNAME, COMPANYNAME)
	VALUES (5, 'Stefan', 'Schmidt', 'Land Hessen');

The resulting EMPLOYEE table looks like this:

ID FIRSTNAME LASTNAME COMPANYNAME

1

Peter

Müller

Deutsche Bahn

2

Hans

Schneider

Stadt Lauterbach

3

Petra

Schneider

Stadt Lauterbach

4

Fritz

Mayer

Deutsche Post

5

Stefan

Schmidt

Land Hessen

This table should be modified by XDM in such a way that FIRSTNAME and LASTNAME are exchanged with random strings.

Of course, it is possible to define two modification rules which apply the same method. However, we can avoid defining two rules by using a regular expression to reference the columns to be modified.

A suitable regular expression for this is /(FIRST|LAST)NAME/ which has to be set as column name for the rule. The / at the beginning and at the end denotes a regular expression.

When executing the task, the following notification appears in the Stage 5 log:

[extract] 12/20/18 2:25 PM INFO Apply modification rule: Random/string on TEST.EMPLOYEE.FIRSTNAME
[extract] 12/20/18 2:25 PM INFO Apply modification rule: Random/string on TEST.EMPLOYEE.LASTNAME

Afterwards, the data in the target environment looks like this:

ID FIRSTNAME LASTNAME COMPANYNAME

1

uasnazsybxeyrlwcipec

sdlnanzkmchacdipaegt

Deutsche Bahn

2

cssllysqmvzmpfhiwqlu

xazjde

Stadt Lauterbach

3

fjwphwajawmzuwfykzjm

tljoqkbodhbimmevxlsa

Stadt Lauterbach

4

srjgezcdaktpsxvcpyjd

ktoxrfkfwwdbgdceucqv

Deutsche Post

5

hndzweodoyepktcjmyui

uvwirlmckxfkxzadhfcv

Land Hessen
1. Only simple reduction rules starting with a WHERE clause are supported.