Scripts

AttributeGenerator

code

A script to generate a single value for an attribute. These scripts are used in EntityGenerator scripts to generate values for attributes.

Parameter

Type

Optional

Mutable

Description

api

API

NO

NO

Provides access to lists of all XDM objects.

Functions

The following table shows the functions that must be declared in the script and their purpose. The functions have access to all variables described in the above parameter list.

Name

Description

Return value

Return value description

init()

This function is called once before the first call of the generateValue function. It can be used to initialize variables or to perform other setup tasks.

n/a

Function is of type void. It does not return any value.

generateAttribute()

This function is called multiple times to generate a new value.

Object

The generated value must be returned.

getMatchScore()

This function is called once to return the score of this attribute generator for the usability of an attribute on an entity.

Integer

The score for the suitability on the given attribute must be returned. Higher values indicate a more suitable attribute generator.

Examples

Example: Using mapping table container

This example demonstrates a scenario where the value is read from a mapping table container for english last names. The mapping table container will be used with variable name 'mtc' and is also used in the tutorial.

The names loaded from the table of the mapping table container are based on two keys, ID and country. The ID is located in the 'KEY_00' column and the country is in the 'KEY_01' column.

  • Groovy

import de.ubs.xdm3.batch.modification.CompositeKey
import de.ubs.xdm3.batch.modification.HashUtils
import de.ubs.xdm3.batch.modification.MappingTableUtils

def mappingTableUtils = null;
def TABLE_NAME = null;
def country = null;
def map = null;
def numberOfValues = 0;

void init() {
    // Creating an instance of MappingTableUtils using the provided mapping table container, which was defined with the variable name 'mtc'
    mappingTableUtils = new MappingTableUtils(lookupTable);
    TABLE_NAME = "LAST_NAMES";
    country = "EN";

    // Retrieves the count of entries in the "LAST_NAMES" table for the column "KEY_01" (country)
    map = mappingTableUtils.count(TABLE_NAME, "KEY_01");

    // Retrieves the number of entries fpr the current country
    numberOfValues = map.get(new CompositeKey(country))
}

def generateAttribute(EntityInstance entity, FieldDefinition field) {
    def rand = (Math.random() * 1000000000) as int;
    def hash = HashUtils.hashInt(rand);
    def row = (Integer) (hash % numberOfValues) + 1;

    // Fetches the value from the table based on the row ID and the country
    return mappingTableUtils.get(TABLE_NAME, row, country)[0];
}
Example: Using already generated values

The following examples illustrates how to use already generated values in the entity instance to generate the corresponding email address for the employee.

  • Groovy

def generateAttribute(EntityInstance entity, FieldDefinition field) {
     def firstName = entity.attributes['first_name']
     def lastName = entity.attributes['last_name']
     return firstName.toLowerCase() + "." + lastName.toLowerCase() + "@mycompany.de"
}
Example: Using getMatchScore function

The following examples illustrate how to implement the getMatchScore function to return the suitability for a type of entity. It will return a score of 100 if the attribute name is 'email', because this attribute generator is implemented to generate an email address from first and last name and 0 otherwise.

  • Groovy

int getMatchScore(FieldDefinition attribute, EntityGeneratorDefinition entityGeneratorDefinition, Map parameters) {
     if(attribute.name.toLowerCase() == "email") {
         return 100;
     }
     return 0;
}

ClassificationTermUsage

formatScript

A script to format the values of a classification term. This script can be used, if the classification term values are stored in a technical format, and the user is familiar with a human readable format.

Return value: The script must return a string with the formatted value of the classification term.

Return type: String

Parameter

Type

Optional

Mutable

Description

api

API

NO

NO

Provides access to lists of all XDM objects.

data

Row Data Map

NO

NO

A map that contains the values of the classification term. The keys are the column names of the classification term usage.

dataList

Row Data List

NO

NO

The list of classification term values in the same order, as the columns of the classification term usage are specified.

Example

The employee number is a 5 to 6 digit number. This example transforms the employee number to a 6-digit representation and precedes each number with 'E-'.

The table below illustrates the input values of the script and the corresponding output.

Diagram
  • Groovy

  • Java Script

return 'E-' + data['emp_no'].longValue().toString().padLeft(6, '0')
return 'E-' + java.lang.String.format("%06d", data['emp_no'].longValue())

DataRelationRule

baseTableConditionScript

This script determines whether the data relation rule should be considered for a specific row from the base table.

Return value: The result of the script decides if the data extract process follows the relation to the referenced table or not.

Return type: Boolean

Parameter

Type

Optional

Mutable

Description

api

API

NO

NO

Provides access to lists of all XDM objects.

data

Row Data Map

NO

NO

A map that containing the values of the current row of the base table for each key column.

scriptCtx

ScriptContext

NO

NO

A class with common utility methods

dataRelationRule

DataRelationRule

NO

NO

The data relation rule the script was defined on.

task

Task

NO

NO

The task of the current execution.

Example

This example demonstrates how to retrieve the value of a custom parameter using a function from the scriptCtx object. The custom parameter, which contains the employee number being copied, is specified on the task template. The script verifies whether the value in the emp_no column of the base table matches the value of the custom parameter. XDM only follows the relation between the base and referenced tables of the data relation rule for rows where the employee number matches the custom parameter value.

The tables below demonstrate that the relationship between employees and dept_emp is only processed when the custom parameter is 10002. For all other rows, XDM will only copy the rows of the employees table.

Diagram
  • Groovy

  • Java Script

data['emp_no'] == scriptCtx.getTaskRuntimeProperty('employee')
data['emp_no'] == scriptCtx.getTaskRuntimeProperty('employee')

lookupScript

This script is used to transform the extracted values from the base table to join them with the referenced table. The script is necessary when the data in the base table and the referenced table cannot be directly compared, and the values in the base table need to be adjusted to match the values in the referenced table.

Return value: The script must return an array with exactly one value for each column in the key of the referenced table.

Return type: Object[]

Parameter

Type

Optional

Mutable

Description

api

API

NO

NO

Provides access to lists of all XDM objects.

data

Row Data Map

NO

NO

A map that containing the values of the current row of the base table for each key column.

scriptCtx

ScriptContext

NO

NO

A class with common utility methods

dataRelationRule

DataRelationRule

NO

NO

The data relation rule the script was defined on.

task

Task

NO

NO

The task of the current execution.

Examples

Example: Prefix values

The following example illustrates a situation where the values in the base table are stored in a different format than in the referenced table. In the dept_emp table, the values of emp_no are always padded to 6 digits with a prefix of 'E'.

Diagram

The script converts input values to the format of the referenced table. It adds the prefix 'E' to emp_no values from the base table and pads employee numbers to 6 characters.

  • Groovy

  • Java Script

['E' + data['emp_no'].longValue().toString().padLeft(6, '0')].toArray();
['E' + java.lang.String.format("%06d", data['emp_no'].longValue())]
Example: Different number of target columns

This example demonstrates a scenario where the values in the base table are divided into two columns in the referenced table. In the dept_emp table, the employee number is split between columns emp_no1 and emp_no2. The first two characters of the employee number are stored in the emp_no1 column, while the remaining characters are stored in the emp_no2 column.

Diagram

The script must return an array that matches the columns of the referenced table. It splits input employee numbers at position 2 into two pieces and returns them according to the referenced table definition.

  • Groovy

  • Java Script

var empNoStr = data['emp_no'].longValue().toString()
[empNoStr.substring(0, 2), empNoStr.substring(2)].toArray()
var empNoStr = data['emp_no'].longValue().toString()
var array = [empNoStr.substring(0, 2), empNoStr.substring(2)]
array

DataShop

initOrderScript

This script is called when a data shop order or request form is first displayed. The script can set values for form parameters, enable or disable fields, hide fields from display, or control the list of options displayed in a drop-down box. It will be executed with the technical user, if one has been specified, or with the logged-in user.

Parameter

Type

Optional

Mutable

Description

api

API

NO

NO

Provides access to lists of all XDM objects.

dataShop

DataShop

NO

YES

The data shop and it’s custom parameters.

meta

MetaMap

NO

YES

A map that contains the meta information for each form parameter

user

User

NO

NO

Provides information about the current user that requests the data shop.

Examples

Example: Disable a field and preset its value

The example script sets the copy_department field to false, which is a boolean type and appears unchecked when a user purchases the data shop. The copy_manager field is disabled in combination with the copy_department field. The copy_manager field should only be editable when the copy_department checkbox is checked.

  • Groovy

  • Java Script

meta['copy_manager'].updatable = false;
dataShop.copy_manager = false;
meta['copy_manager'].updatable = false;
dataShop.copy_manager = false;
Example: Remove an entry from option list

This example script assumes a data shop with a custom parameter called employees. This parameter is of type collection that allows the selection of multiple employees to be copied.

The script removes a specific employee from the selection list by identifying the position of the employee with ID 10001 and removing that entry from the list. As a result, employee 10001 will not be displayed in the order form of the data shop.

To avoid completely removing an entry, it can be disabled instead. This will display the entry in the order form, but the user will not be able to select it. To disable a specific entry, set the enabled flag to false.

  • Groovy

  • Java Script

var options = meta['employees'].options
var index = options.find { e -> e.value == '10001' }
options.remove(index)
var index = 0;
var options = meta['employees'].options
for(i in options) {
    if(options[i].value == '10001') {
        index = i;
        break;
    }
}
options.remove(index)

revalidationScript

A script that is evaluated each time the user enters a value in the data shop form.It can set values for form parameters, enable or disable fields, it can hide fields from being displayed, or it can control the list of options that will be displayed in a drop down box. The script will be executed with the technical user, if one has been specified, or with the logged-in user.

Parameter

Type

Optional

Mutable

Description

api

API

NO

NO

Provides access to lists of all XDM objects.

dataShop

DataShop

NO

YES

The data shop and it’s custom parameters.

meta

MetaMap

NO

YES

A map that contains the meta information for each form parameter

user

User

NO

NO

Provides information about the current user that requests the data shop.

propertyName

String

NO

NO

The name of the property that has changed by the last user interaction.

Example

The revalidation script example implements the following workflow: If the boolean field copy_department is:

  • false: it should not be possible to select to copy the department manager.

  • true: the purchaser also has the option to copy the department manager.

The script updates the state in the meta variable to control whether the checkbox of the copy_department field is editable or not. If copy_department is false, it also resets the value of copy_manager to false.

  • Groovy

  • Java Script

meta['copy_manager'].updatable = dataShop.copy_department;
if(dataShop.copy_department == "false") {
    dataShop.copy_manager = false;
}
meta['copy_manager'].updatable = dataShop.copy_department;
if(dataShop.copy_department == "false") {
    dataShop.copy_manager = false;
}

validationScript

This script can validate the inputs of a data shop request and is called after the user ordered the data shop. The script has access to all parameters that user entered in the form and can allow or refuse the execution. It will be executed with the technical user, if one has been specified, or with the logged-in user.

Parameter

Type

Optional

Mutable

Description

api

API

NO

NO

Provides access to lists of all XDM objects.

dataShop

DataShop

NO

YES

The data shop and it’s custom parameters.

user

User

NO

NO

Provides information about the current user that requests the data shop.

startAt

Timestamp

YES

YES

The scheduled execution time of the data shop request.

Examples

An employee number is always composed of 5 or 6 digits. In the following example, we assume a data shop where the purchaser can enter an employee number. To ensure that only valid employee numbers are processed, the following script checks the validity of the entered number. If the number is invalid, the script indicates an error in the respective field of the data shop form by throwing a script validation exception.

  • Groovy

  • Java Script

def employeeNumber = dataShop.employee;
if (!(employeeNumber ==~ /[0-9]{5,6}/)) {
    def message = "Employee number ${employeeNumber} is not valid."
    throw new ScriptValidationException(message, 'employee');
}
var employeeNumber = dataShop.employee;
if (!employeeNumber.match('^[0-9]{5,6}$')) {
    var message = 'Employee number ' + employeeNumber + ' is not valid.';
    throw new ScriptValidationException(message, 'employee');
}

Orders with certain parameters can only be executed in the evening. During the day, an error message is displayed indicating that the order must either be scheduled or adjusted. In the following example, the data shop order is cancelled with a message if the employee number is 10001 and the time is before 20:00. If a scheduled time was entered when the data shop was ordered, this time will be taken into account during the check.

  • Groovy

import java.time.LocalDateTime;
import java.time.LocalTime;

LocalDateTime date = LocalDateTime.now();
if (startAt != null) {
    date = startAt;
}

LocalTime earliestExecutionTime = LocalTime.of(20, 00);
if (dataShop.employee.equals('10001') && date.toLocalTime().isBefore(earliestExecutionTime)) {
    throw new ScriptValidationException("This configuration is only to be executed at the end of the day.")
}

DomainEntity

configurationScript

A script is executed when the corresponding domain entity is created or updated. The script creates the specified attributes, as domain attribute entities. Additionally, the script can be configured how the response should be pre-processed by the Rest API, should this be necessary.

Parameter

Type

Optional

Mutable

Description

api

API

NO

NO

Provides access to lists of all XDM objects.

Functions

The following table shows the functions that must be declared in the script and their purpose. The functions have access to all variables described in the above parameter list.

Name

Description

Return value

Return value description

displayName()

This function is called to determine the display name of the entity. The function receives a map of attributes and should return the display name.

String

The display name of the entity.

preProcessResponse()

This function called for each REST invocation to prepare it for further processing.

String

Valid JSON for further processing

create()

The create function is called when a new entity is to be created in a target REST web service. The function is responsible for issuing the appropriate HTTP commands to create the provided entity. The function is optional. By default, XDM issues a HTTP POST to create the entity. The data parameter contains the entity data. The url parameter provides the target URL based on the domain modeling.

Object

The response of the create function. This can be a string, a map, or a list. It depends on the result of the respective POST endpoint. The result is used to update internal references of other entities

update()

The update function is called when an existing entity is to be updated in a target REST web service. The function is responsible for issuing the appropriate HTTP commands to update the provided entity. The function is optional. By default, XDM issues a HTTP PUT to update the entity. The data parameter contains the entity data. The url parameter provides the target URL based on the domain modeling.

Object

The response of the create function. This can be a string, a map, or a list. It depends on the result of the respective PUT endpoint. The result is used to update internal references of other entities.

delete()

The delete function must delete the provided entity in a a target web service. If the function is not present, XDM issues a HTTP DELETE on the entity URL. The data parameter contains the entity data. The url parameter provides the target URL based on the domain modeling.

n/a

n/a

search()

The search function is called to search for an entity in a target REST web service. The function is responsible for issuing the appropriate HTTP commands to search for the provided entity. The function needs to query the target webservice to obtain the respective entity. The function is responsible to identify the target entity, based on application specific fields and webservice specific search criteria. The data parameter contains the entity data. The url parameter provides the target URL based on the domain modeling.

Object

The function must return the matching entity of the target service.

Example

Example: Example of a domain entity configuration script

This example provides a template for a configuration script that maps a domain entity to a webservice response.

  • Groovy

endpoints = [
    find: [
        path: "/department/1/employee/1",
    ],
    list: [
        path: "department/1/employee",
        paging: [
            pageParameter: "page",
            sizeParameter: "size",
            pageSize: 20
        ]
    ]
]

// The attributes variable assigned the attributes of the webservice response to the domain entity.
attributes = [
    [
        name:   "Name",
        description: "The name of the employee",
        index:  false,
        key:    false,
        path:   "name",
        type:   "STRING"
    ],
    [
        name:   "Email",
        description: "The email address of the employee",
        index:  false,
        key:    false,
        path:   "email",
        type:   "STRING"
    ],
    [
        name:   "Hire Date",
        description: "The date the employee was hired",
        index:  false,
        key:    false,
        path:   "hireDate",
        type:   "TIMESTAMP"
    ],
    [
        name:   "department",
        description: "The department the employee belongs to",
        index:  false,
        key:    false,
        path:   "department",
        type:   "STRING"
    ],
    [
        name:   "Self",
        description: "The link to the employee",
        index:  false,
        key:    true,
        path:   "_links.self.herf",
        type:   "STRING"
    ]
];

def displayName(Map<String, String> attributes) {
    return attributes.getOrDefault("name", "UNKNOWN")
}

def preProcessResponse(response) {
  // Drop the first line
  return response.split('\n').drop(1).join('\n')
}

EntityGenerator

code

A script to generate values for all attributes of an entity.

Parameter

Type

Optional

Mutable

Description

api

API

NO

NO

Provides access to lists of all XDM objects.

Functions

The following table shows the functions that must be declared in the script and their purpose. The functions have access to all variables described in the above parameter list.

Name

Description

Return value

Return value description

init()

This function is called once before the first call of the generateValue function. It can be used to initialize variables or to perform other setup tasks.

n/a

Function is of type void. It does not return any value.

generateEntity()

This function is called multiple times to generate a new entity.

Object

The generated value must be returned. The returned value must be an instance of EntityInstance. Use the buildEntity() method to create a instance.

Examples

Example: Using custom generators

The following example generates an employee entity and defines custom generator for attributes. In the entityGenerationConfig following attribute generators are defined:

  • For the attribute gender the genderGenerator will be used.

  • For the attribute first_name the nameGenerator will be used and set the parameter nameType to FIRST_NAMES that is used to get the firstName from an mapping table container table called FIRST_NAMES. So it is possible to use the same generator for different attributes only by changing the parameters.

  • For the attribute last_name the nameGenerator will be used and set the parameter nameType to LAST_NAMES that is used to get the lastName from an mapping table container table called LAST_NAMES.

  • For the attribute birth_date the dateGenerator will be used and set the parameters minValue and maxValue to generate a date between 1964-01-01 and 2007-01-01.

  • For the attribute emp_no the numberGenerator will be used and set the parameter initialValue to the value of the parameter empNo.

This example generates the entity to the following entity definition:

attributes = [
    [ name           : "first_name",
      type           : "STRING",
      path           : "first_name",
      length         : 14
    ],
    [ name           : "last_name",
      type           : "STRING",
      path           : "last_name",
      length         : 16
    ],
    [ name           : "gender",
      type           : "STRING",
      path           : "gender",
      length         : 1
    ],
    [ name           : "emp_no",
      type           : "NUMBER",
    ],
    [ name           : "birth_date",
      type           : "DATE",
      path           : "birth_date"
    ]
]
  • Groovy

def entityGenerationConfiguration = [:];

void init() {
    entityGenerationConfiguration = [
        "gender": processorFactory.createAttributeGenerator(processor, "genderGenerator"),
        "first_name": processorFactory.createAttributeGenerator(processor, "nameGenerator", [nameType: "FIRST_NAMES"]),
        "last_name": processorFactory.createAttributeGenerator(processor, "nameGenerator", [nameType: "LAST_NAMES"]),
        "birth_date": processorFactory.createAttributeGenerator(processor, "dateGenerator", [minValue: "1964-01-01", maxValue: "2007-01-01"]),
        "emp_no": processorFactory.createAttributeGenerator(processor, "numberGenerator", [initialValue: parameters.empNo]),
    ];
}

EntityInstance generateEntity() {
    return buildEntity(entityGenerationConfiguration);
}
Example: With LLM

The following example generates an employee entity with the help of large language model. The llm(n, instructions, attributeInstructions) method is used to generate the entity. The method takes three parameters:

  • n: the number of entity instances to generate with one LLM call.

  • instructions: Additional instructions added to the prompt to guide the LLM.

  • attributeInstructions: Additional instructions for each attribute to guide the LLM. In the entityGenerationConfig following attribute generators are defined:

In the example we instruct llm to generate 20 employees with the instruction "Generate employees.". Furthermore we provide additional instructions for each attribute to guide the LLM:

  • For the first_name attribute we instruct the LLM to "Generate spanish first name".

  • For the last_name attribute we instruct the LLM to "Generate swedish last name".

  • For the emp_no attribute we instruct the LLM to "Generate employee number between 1 and 1000000".

  • For the birth_date attribute we instruct the LLM to "Generate birth date between 1964-01-01 and 2007-01-01".

  • For gender we do not provide specific instructions, so the LLM will use the description of the attribute in entity modelling.

This example generates the entity to the following entity definition:

attributes = [
    [ name           : "first_name",
      type           : "STRING",
      path           : "first_name",
      length         : 14
    ],
    [ name           : "last_name",
      type           : "STRING",
      path           : "last_name",
      length         : 16
    ],
    [ name           : "gender",
      type           : "STRING",
      path           : "gender",
      length         : 1,
      description    : "The gender of the employee. Possible values are M, F."
    ],
    [ name           : "emp_no",
      type           : "NUMBER"
    ],
    [ name           : "birth_date",
      type           : "DATE",
      path           : "birth_date"
    ]
]
  • Groovy

  EntityInstance generateEntity() {
    return llm(1, "Generate employees.", ["first_name":"Generate spanish first name",
                                           "last_name":"Generate swedish last name",
                                           "emp_no":"Generate employee number between 1 and 1000000",
                                           "birth_date":"Generate birth date between 1964-01-01 and 2007-01-01"
                                           ]);
  }

EntityModificationMethodCodeDefinition

code

The script for the modification of entities.

Parameter

Type

Optional

Mutable

Description

api

API

NO

NO

Provides access to lists of all XDM objects.

Functions

The following table shows the functions that must be declared in the script and their purpose. The functions have access to all variables described in the above parameter list.

Name

Description

Return value

Return value description

apply()

Contains all code of the entity modification method that will be executed to modify a single entity.The apply method has one parameter: a map of all attributes of the entity converted according to the specified attribute type.

n/a

n/a

init()

This optional function is executed before the first modification of an attribute.It can be used to initialize resources like global variables.

n/a

n/a

close()

This optional function is executed after the last modification of an attribute.Its purpose may be to clean any previously allocated resources.

n/a

n/a

Examples

Example: Modify sensitive data of employees

This example is based on a Kafka topic that contains employee data. An example payload of a Kafka message could look like this:

{
    "emp_no": "10002",
    "birth_date": "1984-06-02",
    "first_name": "Bezalel",
    "last_name": "Simmel",
    "gender": "F",
    "hire_date": "2005-11-21",
    "email": "bezalel.simmel@example.com",
    "weekly_work_hours": "40.0",
}

The script modifies the first name, last name, and email address of every employee by replacing all characters except the first one with asterisks. Every value of the payload can be modified in the apply method by accessing the data map.

  • Groovy

def void apply(Map<String,Object> data) {
    def fn = data.get("first_name")
    def ln = data.get("last_name")
    data.put("first_name", fn[0] + "***")
    data.put("last_name",  ln[0] + "***")
    data.put("email", "${fn[0]}${ln[0]}***@example.com")
}
Example: Modify Kafka messages with structured values

For Kafka messages with structured values, the script can be used to modify the values of the payload as well, but it is important to note that the structure of the payload must not be changed. Here is an example of an attribute with a structured value:

"items": [
            { "item1": "A1", "qty": 2 },
            { "item2": "B7", "qty": 1 }
          ]

In this example, a data.get("items") call would return the payload. In this case, it would return:

[
    { "item1": "A1", "qty": 2 },
    { "item2": "B7", "qty": 1 }
]

It is possible to modify the payload like a string and write it back to the data map. The following example shows how to set all quantities to 0 by replacing the values of the qty attributes with a regular expression.

  • Groovy

def void apply(Map<String,Object> data) {
    def items = data.get("items")
    def output = items.replaceAll(/"qty"\s*:\s*\d+/, '"qty": 0')
    data.put("items", output)
}

EntityModificationRule

activationScript

This script dynamically controls if the entity modification rule is active or inactive for a task to run.

Return value: The script must return either true or false, depending on whether or not the entity modification rule should be used during a task execution.

Return type: Boolean

Parameter

Type

Optional

Mutable

Description

api

API

NO

NO

Provides access to lists of all XDM objects.

entity

DomainEntity

NO

NO

The domain entity used in the rule.

user

User

NO

NO

Provides information about the current user that executes this script.

Functions

The following table shows the functions that must be declared in the script and their purpose. The functions have access to all variables described in the above parameter list.

Name

Description

Return value

Return value description

isActive()

This function determines whether the entity modification rule is active or not. The function must return either true or false, depending on whether or not the entity modification rule should be used during a task execution.

Boolean

The method must return true, that the entity modification rule is active and should be used.

Example

Assuming that the custom parameter employee contains the employee number to be copied from source to target, the employee with the ID '10001' must be masked due to the sensitive data it contains.

The activation script dynamically decides whether the modification rules should be applied during the copy process or not. The modification rule is only used if the employee with the number '10001' is being copied. All other employees will not be masked by the current modification rule.

  • Groovy

def isActive () {return entity.employee == 10001}

EntityScenarioGenerator

code

A script to generate values for multiple entities of a test scenario.

Parameter

Type

Optional

Mutable

Description

api

API

NO

NO

Provides access to lists of all XDM objects.

Functions

The following table shows the functions that must be declared in the script and their purpose. The functions have access to all variables described in the above parameter list.

Name

Description

Return value

Return value description

init()

This function is called once before the first call of the generateValue function. It can be used to initialize variables or to perform other setup tasks.

n/a

Function is of type void. It does not return any value.

generateScenario()

This function is called one time to generate all entities for the scenario. Completed scenarios are handled by the flush() method

n/a

Function is of type void. It does not return any value.

Example

Example: Using custom entity generators

The following example uses three entity generators to generate the entities employee, salary and title. First the entity employee is generated with the parameter empNo. Then the entity salary is generated with the parameter emp, which is the employee entity generated before. Finally the entity title is generated with the parameter emp, which is the employee entity generated before.

This example generates the entity to the following entity definition:

// employee
attributes = [
    [ name           : "first_name",
      type           : "STRING",
      path           : "first_name",
      length         : 14
    ],
    [ name           : "last_name",
      type           : "STRING",
      path           : "last_name",
      length         : 16
    ],
    [ name           : "gender",
      type           : "STRING",
      path           : "gender",
      length         : 1
    ],
    [ name           : "emp_no",
      type           : "NUMBER",
    ],
    [ name           : "birth_date",
      type           : "DATE",
      path           : "birth_date"
    ]
]

// salary
attributes = [
    [ name           : "emp_no",
      type           : "NUMBER",
      path           : "emp_no",
      mapsTo         : "employee",
      relationType   : "AGGREGATION",
      cardinality    : "ONE_TO_MANY"
    ],
    [ name           : "salary",
      type           : "NUMBER",
      path           : "salary",
    ],
    [ name           : "from_date",
      type           : "DATE",
      path           : "from_date"
    ],
    [ name           : "to_date",
      type           : "DATE",
      path           : "to_date"
    ]
]

// title
attributes = [
    [ name           : "emp_no",
      type           : "NUMBER",
      path           : "emp_no",
      mapsTo         : "employee",
      relationType   : "AGGREGATION",
      cardinality    : "ONE_TO_MANY"
    ],
    [ name           : "tile",
      type           : "STRING",
      path           : "title",
      length         : 50
    ],
    [ name           : "from_date",
      type           : "DATE",
      path           : "from_date"
    ],
    [ name           : "to_date",
      type           : "DATE",
      path           : "to_date"
    ]
]
  • Groovy

void generateScenario() {
    def employeeGenerator = processorFactory.createEntityGenerator(processor, "employee", [empNo: parameters.empNo]);
    def employee = employeeGenerator.generateEntity();
    flush(employee);

    def salaryGenerator = processorFactory.createEntityGenerator(processor, "salary", [emp: employee]);
    def titleGenerator = processorFactory.createEntityGenerator(processor, "title", [emp: employee]);
    def salary = salaryGenerator.generateEntity();
    def title = titleGenerator.generateEntity();
    flush(salary);
    flush(title);
}

Environment

activationScript

This script dynamically controls which installed applications in the environment are active or inactive for a task to run. The script is called for each defined installed application and must return the active status for it.

Return value: The script must return either true or false, depending on whether the provided installed application should be active or inactive for a given task execution.

Return type: Boolean

Parameter

Type

Optional

Mutable

Description

api

API

NO

NO

Provides access to lists of all XDM objects.

environment

Environment

NO

NO

The environment for which the active installed applications are determined.

installedApplication

InstalledApplication

NO

NO

The installed application that the script should check and return it’s active status.

user

User

NO

NO

Provides information about the current user that executes this script.

Example

This example assumes a custom parameter called section that is declared in the environment and its installed application. The value of the custom parameter in the environment can be overwritten in the task template.

The script activates only those installed applications where the value of section matches with the value of the environment.

The table below shows which installed applications would be used for a copy process when the value of section is 'hr'. All other installed applications are not used during the copy process.

Diagram
  • Groovy

  • Java Script

return environment.section == installedApplication.section
return environment.section == installedApplication.section

startModelScript

This script dynamically returns one of the environment’s installed applications to be used as the start model for a task execution.

Return value: Must return the installed application that is used as start model.

Return type: InternalMapWrapper

Parameter

Type

Optional

Mutable

Description

api

API

NO

NO

Provides access to lists of all XDM objects.

environment

Environment

NO

NO

The environment for which the start model must be returned.

user

User

NO

NO

Provides information about the current user that executes this script.

Example

This example assumes the presence of a custom parameter called section in the environment and each installed application. Additionally, there is a custom parameter called primary on each installed application that indicates if it is the primary entry point for the respective section.

The script identifies the installed application as start model, that matches the section of the environment and that is declared as primary.

The table displays the installed applications that serve as the starting model for a copy process when the value of section in the environment is 'hr'.

Diagram
  • Groovy

  • Java Script

return environment.installedApplications.find {
    ia -> ia.section == environment.section && ia.primary.equals('true')
}
for (var i=0; i<environment.installedApplications.length; i++) {
     var ia = environment.installedApplications[i];
     if (ia.section == environment.section && ia.primary === 'true') {
         return ia;
     }
}
return null;

Event

script

A script is executed when the corresponding event is triggered. The event is triggered when the configured operation is performed, such as creating a new object. The script can execute custom code to implement user specific workflows.

Parameter

Type

Optional

Mutable

Description

api

API

NO

NO

Provides access to lists of all XDM objects.

object

Object

YES

YES

The associated object that caused the event trigger.

The object is as follows: Data Shop Requested, Data Shop Approved, Data Shop Rejected:: The requested execution object. Object Delete, Object Deleted, Object Create, Object Created, Object Updated:: The object on which the operation has been performed. Execution Started, Execution Finished:: The execution object. Casc Finished, Scheduled Execution Triggered:: The scheduled execution.

This parameter is not present for the events On Export or On Import.

objects

List

YES

YES

The associated list of objects that caused the event trigger.

The object is as follows: On Import, On Export:: The list contains the objects and their attributes and children.

This parameter is only available for the events On Export and On Import.

user

User

YES

NO

The user that caused the event to be triggered.

On the Data Shop Approved, or Data Shop Rejected events this indicates the user that performed the approve or reject.

This parameter is not present for the events Execution Finished, Casc Finished, or Scheduled Execution Triggered.

Examples

Example: Preset properties in create file dialog

This example provides a template for files created by a specific user group. Each user in the PURCHASER role must create CSV files using ';' as the delimiter.

  • Groovy

  • Java Script

if(user.authorities.contains('PURCHASER')) {
    object.csvDelimiter = ';';
    object.fileType = 'CSV';
}
if(user.authorities.indexOf('PURCHASER') != -1) {
    object.csvDelimiter = ';';
    object.fileType = 'CSV';
}
Example: Send an e-mail after a data shop was requested

The following example is more complex, but covers a typical scenario. Each time a data shop is requested and the request is awaiting approval, a dedicated user should receive an email asking for review.

The following code sends an email to a specific user, with the SMTP server configured in the script. Authentication is required for the SMTP server, and the user for this authentication is read from the defined XDM credentials. The user with the display name SMTP User is used. This credential contains the password for the SMTP server. To ensure proper functionality, grant the SCRIPT permission to the technical user of the event.

The script requires the external Java libraries Angus Mail Default Provider and Jakarta Activation.

Both can be downloaded below: * Angus Mail Default Provider * Jakarta Activation

Add the downloaded '.jar' files as JAVA XDM files and use them as libraries in the event.

  • Groovy

import jakarta.mail.*
import jakarta.mail.internet.*;

def from = user.email
def to = "approver@example.com"
def subject = "Approve data shop ${object.dataShop.displayName}"
def content = """
    The user '${user.name}' requested an execution for data shop '${object.dataShop.displayName}'.
    Please review this request here: https://xdm.example.com/gen/res${object.contextPath}
    """

def smtpHost = "example.com"
def smtpPort = 587
def smtpCredential = api.credentials.find{c -> c.displayName == 'SMTP User'}

// Setup internal properties
def props = [
    "mail.smtp.user": smtpCredential.user,
    "mail.smtp.host": smtpHost,
    "mail.smtp.port": smtpPort,
    "mail.smtp.starttls.enable": true,
    "mail.smtp.socketFactory.class": "javax.net.ssl.SSLSocketFactory",
    "mail.smtp.ssl.trust": smtpHost
]

// Setup the message.
def message = new MimeMessage(Session.getDefaultInstance(new Properties(props)))
message.setFrom(new InternetAddress(from))
message.addRecipients(Message.RecipientType.TO, new InternetAddress(to))
message.setSubject(subject)
message.setContent(content.toString(), "text/plain")

// Send the mail.
Transport.send(message, smtpCredential.user, smtpCredential.password)
Example: Sort export objects by type and attribute display name

This example demonstrates how to sort the export objects by the type and the attribute display name. Additionally, the event log prints the original order and the order after sorting.

  • Groovy

println("Original export")
println('-' * 80)
objects.each{it ->
    printExportData(it, 0)
    println()
}

// Sort export objects by type and attribute display names
def sortedObjects = objects.sort{a,b -> a.type <=> b.type ?: a.attributes.displayName <=> b.attributes.displayName}
sortedObjects.each{obj ->
    if (obj.children) {
        sortChildren(obj)
    }
}

println("Sorted export")
println('-' * 80)
sortedObjects.each{it ->
    printExportData(it, 0)
    println()
}

def printExportData(entity, indentLevel) {
    def displayName = entity.attributes.displayName ?: "n/a"
    def indent = '  ' * indentLevel
    ++indentLevel;
    def message = "${displayName}, ${entity.type}"
    println("${indent}${message}")
    if (entity.children) {
        entity.children.each{child -> printExportData(child, indentLevel)}
    }
}

def sortChildren(entity) {
    entity.children = entity.children.sort{ a,b ->
        def aName = a.attributes.displayName ?: ""
        def bName = b.attributes.displayName ?: ""
        a.type <=> b.type ?: aName <=> bName
    }
    entity.children.each{child ->
        if (child.children) {
            sortChildren(child)
        }
    }
}
Example: Change the retention period for the execution before it is imported into the xdm3 environment via import

The following example demonstrates how to change an attribute before it is imported into the xdm3 environment via import. Additionally, the event log prints information about the changed attribute and the new value.

  • Groovy

// Filter objects by type, variation, and attribute value
def filteredObjects  = objects.findAll{ obj->
    obj.type == "TaskTemplate" &&
    obj.variation == "subset-task-template" &&
    obj.attributes.executionRetentionPeriod == "P5D"
}

// Change the attribute value
filteredObjects.each{obj ->
    obj.attributes.executionRetentionPeriod = "-1"
    println("Attribute execution retention period of type '${obj.type}' with the name '${obj.attributes.displayName}' has been changed, new value: ${obj.attributes.executionRetentionPeriod}")
}

FileOperator

template

A script to generate a single value for an attribute. These scripts are used in EntityGenerator scripts to generate values for attributes.

Parameter

Type

Optional

Mutable

Description

api

API

NO

NO

Provides access to lists of all XDM objects.

fileOperator

FileOperator

NO

NO

The file operator definition that provides all properties of the file operator inside the script.

Functions

The following table shows the functions that must be declared in the script and their purpose. The functions have access to all variables described in the above parameter list.

Name

Description

Return value

Return value description

init()

This function is executed once prior to the start of the processing.

n/a

n/a

process()

This function is called multiple times to generate a new value.

String

The generated string value should be written to the file, or null if the function did not generate a value.

Example

Example: Generate a CSV file

This example demonstrates how to generate a CSV file with a file operator. The script initially writes the header line to the file, followed by a line for each entity instance.

  • Groovy

String init(EntityDefinition definition) {
    // Write the header line to the file
    def header = definition.fields.collect { field -> field.name }.join(';')
    return header + '\n';
}

String process(List<EntityInstance> instances, EntityDefinition definition) {
    // Generate one or more lines for the entity instance
    return instances.collect { instance ->
        return definition.fields
            .collect { field -> instance.attributes[field.name] }
            .join(';')
    }.join('\n') + '\n'
}

ModificationMethod

code

The script of the modification method. It defines how values or entire rows of a table are modified.

Parameter

Type

Optional

Mutable

Description

api

API

NO

NO

Provides access to lists of all XDM objects.

data

Row Data List

NO

NO

This array contains the values of the currently processed row. The row belongs to the table for which the modification rule is declared.

ctx

ModificationContext

NO

NO

A class with common utility methods for modification.

columnIndex

Number

NO

NO

The position of the column that matched the pattern of the modification rule within the data array

replacementData

List

NO

NO

A list of rows that is empty by default. It can be filled with data rows and will then be used instead of the current data of the modification method. It can also be utilized to multiply rows using helper methods stashRow(..) and unstashRows() of the ctx object.

modificationRule

Map

NO

NO

Internal parameter

modificationMethod

ModificationMethod

NO

NO

The modification method itself. It contains all the defined parameters.

task

Task

NO

NO

The task of the current execution.

startedBy

User

NO

NO

Reference to the user that is executing the task.

requestedBy

User

YES

NO

Reference to the user that has requested this execution.

sourceConnection

Connection

YES

NO

Source table connection.

targetConnection

Connection

YES

NO

Target table connection.

Functions

The following table shows the functions that must be declared in the script and their purpose. The functions have access to all variables described in the above parameter list.

Name

Description

Return value

Return value description

apply()

Contains all code of the modification method that will be executed to modify every row of the data. The apply function will be executed for every single row.

Boolean

The method must return a boolean, that controls whether the row should be filtered by the modification method. A value of true indicates, that the row is written to the target table. A value of false indicates that the row will be skipped and not written to the target.

init()

This function is executed once prior to the start of modification, when the modification context is created.

n/a

n/a

close()

This function is executed once after modification is complete, immediately before the modification context is closed.

n/a

n/a

Example

This modification method uses an employee’s gender to mask their first name. Female employees will be referred to as 'Jane', while male employees will be referred to as 'John'. The script demonstrates how to make decisions based on other columns in a row.

The overview below shows how the modification method script would mask data based on the employees table.

Diagram
  • Groovy

  • Java Script

def apply() {
     def genderColumnIndex = ctx.indexOf('gender');
     if (data[genderColumnIndex].equals('M')) {
         data[columnIndex] = "John";
     } else {
         data[columnIndex] = "Jane";
     }
     return true;
}
function apply() {
     var genderColumnIndex = ctx.indexOf('gender');
     if (data[genderColumnIndex] === 'M') {
         data[columnIndex] = "John";
     } else {
         data[columnIndex] = "Jane";
     }
     return true;
}

ModificationRule

adaptionScript

This script controls how a data values must be changed before and after the invocation of the respective modification method.

Parameter

Type

Optional

Mutable

Description

api

API

NO

NO

Provides access to lists of all XDM objects.

data

Row Data List

NO

NO

This array contains the values of the currently processed row. The row belongs to the table for which the modification rule is declared.

ctx

ModificationContext

NO

NO

A class with common utility methods for modification.

columnIndex

Number

NO

NO

The position of the column that matched the pattern of the modification rule within the data array

modificationRule

Map

NO

NO

Internal parameter

Functions

The following table shows the functions that must be declared in the script and their purpose. The functions have access to all variables described in the above parameter list.

Name

Description

Return value

Return value description

convertTo()

Changes the data value before it is provided to the modification method.

n/a

n/a

convertFrom()

Converts the value returned form the modification method to the desired format of the target table.

n/a

n/a

Example

Assuming that the employees table stores both first and last names in one single column, the first name in that column should be masked. To achieve this, we use a modification method that can handle first names. Therefore, the first names passed to the modification method must be extracted from the name column. Once the modification is complete, the original last name should be appended to the masked first name before it is written to the target table.

The example utilizes a global variable to retain the original name and append the last name in the convertFrom() method. Currently, there is no alternative to obtain the original value within the convertFrom() method.
Diagram
  • Groovy

  • Java Script

oldName = '';

void convertTo() {
    oldName = data[columnIndex]
    def array = oldName.split(' ')
    data[columnIndex] = array[0]
}

void convertFrom() {
    def array = oldName.split(' ')
    def newFirstName = data[columnIndex]
    data[columnIndex] = "${newFirstName} ${array[-1]}"
}
var oldName = '';

function convertTo() {
    oldName = data[columnIndex]
    var array = oldName.split(' ')
    data[columnIndex] = array[0]
}

function convertFrom() {
    var array = oldName.split(' ')
    var newFirstName = data[columnIndex]
    data[columnIndex] = newFirstName + ' ' + array[array.length - 1]
}

conditionScript

This script decides whether the modification method should be applied for a row or not. The script is called for all selected rows of the table on which the modification rules is defined.

Parameter

Type

Optional

Mutable

Description

api

API

NO

NO

Provides access to lists of all XDM objects.

data

Row Data List

NO

NO

This array contains the values of the currently processed row. The row belongs to the table for which the modification rule is declared.

ctx

ModificationContext

NO

NO

A class with common utility methods for modification.

columnIndex

Number

NO

NO

The position of the column that matched the pattern of the modification rule within the data array

modificationRule

Map

NO

NO

Internal usage

Functions

The following table shows the functions that must be declared in the script and their purpose. The functions have access to all variables described in the above parameter list.

Name

Description

Return value

Return value description

condition()

n/a

Boolean

Returns true if the row should be modified by the associated modification method, or false if the row should be skipped.

Example

The script assumes that only female first names should be masked. It determines the gender of the current employee and returns true for female employees and false for male employees. The script uses the indexOf() method of the global variable ctx to determine the index of the gender column within the current row.

Diagram
  • Groovy

  • Java Script

boolean condition() {
    def columnIndex = ctx.indexOf('gender')
    return data[columnIndex].equals("F");
}
function condition() {
    var columnIndex = ctx.indexOf('gender')
    return data[columnIndex] === "F";
}

ModificationSet

activationScript

This script dynamically controls if the modification set is active or inactive for a task to run.

Return value: The script must return either true or false, depending on whether or not the modification set should be used during a task execution.

Return type: Boolean

Parameter

Type

Optional

Mutable

Description

api

API

NO

NO

Provides access to lists of all XDM objects.

modificationSet

ModificationSet

NO

NO

The modification set that the script should check and return it’s active status.

user

User

NO

NO

Provides information about the current user that executes this script.

Example

Assuming that the custom parameter employee contains the employee number to be copied from source to target, the employee with the ID '10001' must be masked due to the sensitive data it contains.

The activation script dynamically decides whether the modification rules should be applied during the copy process or not. The modification set is only used if the employee with the number '10001' is being copied. All other employees will not be masked by the current modification set.

  • Groovy

  • Java Script

return modificationSet.employee == 10001
return modificationSet.employee === 10001

TaskStageHook

code

The script that runs when the task stage hook is executed.

Parameter

Type

Optional

Mutable

Description

api

API

NO

NO

Provides access to lists of all XDM objects.

properties

Map

NO

YES

A map of properties that are returned to the caller of the stage hook. If the stage hook is called from a task, this map also contains internal runtime properties of the task execution.

execution

Execution

NO

NO

Reference to the execution object in which the task is called.

taskStageHook

TaskStageHook

NO

NO

Reference to the task stage hook object.

taskStageHookUsage

TaskStageHookUsage

YES

NO

Reference to the task stage hook usage, that setups the hook invocation. This parameter is only available if the hook is called from a task.

task

Task

YES

NO

Reference to the task object. This parameter is only available if the hook is called from a task.

workflow

Workflow

YES

NO

Reference to the workflow object and only available, if the hook is called from a workflow.

generation

IceBox Generation

YES

NO

This parameter contains information about the icebox generation and is only available if the hook is called by from icebox task.

restTemplate

RestTemplate

YES

NO

An accessor to access REST endpoints of the core service.

dataConnection

JDBC Connection

YES

NO

A JDBC connection to access the extracted data in the data preview of a task execution. Data is only available if the Data Preview is activated in the task.

startedBy

User

NO

NO

Reference to the user that is executing the task stage hook.

requestedBy

User

YES

NO

Reference to the user that has requested this execution.

Example

This hook example randomly selects an employee from the employees table using a SQL query. The script executes the SQL and reads the result using native functions. The employee number is then stored in the properties map of the task, which can be accessed from outside the hook, such as from the calling workflow.

  • Groovy

  • Java Script

import groovy.sql.Sql;
def sql = new Sql(taskStageHook.connection.jdbcConnection);

def first = sql.firstRow("SELECT emp_no FROM production.employees ORDER BY RANDOM() LIMIT 1");
properties.put("employee", first.values()[0]);
var query = "SELECT emp_no FROM production.employees ORDER BY RANDOM() LIMIT 1"
var stmt = taskStageHook.connection.jdbcConnection.createStatement();
var rs = stmt.executeQuery(query);

if(rs.next()) {
    properties.put("employee", rs.getInt(1));
}

rs.close();
stmt.close();

Webservice

configurationScript

The script is executed whenever there is a need to start communication with the webservice. It’s executed once to create a personal access token which is later used in every request send to the webservice.

Parameter

Type

Optional

Mutable

Description

api

API

NO

NO

Provides access to lists of all XDM objects.

Functions

The following table shows the functions that must be declared in the script and their purpose. The functions have access to all variables described in the above parameter list.

Name

Description

Return value

Return value description

login()

This function is a setup method to provide a access token for all subsequent requests.

String

A token as a string

Example

Example: Example of a webservice configuration script

This example provides a login method that creates a personal login token. The login token is requested from the external webservice login endpoint and is return in the header of the response.

  • Groovy

def login() {
  payload = [
    username: webservice.credential.user,
    password: webservice.credential.password
  ]
  response = post(webservice.baseUrl + '/foobar/login', payload)
  return response.getHeader('Token')
}

WorkflowTemplate

workflow

The script that runs when the workflow is executed.

Parameter

Type

Optional

Mutable

Description

api

API

NO

NO

Provides access to lists of all XDM objects.

workflow

Workflow

NO

NO

Reference to the workflow object that is being executed.

startedBy

User

NO

NO

Reference to the user that is executing the workflow.

requestedBy

User

YES

NO

Reference to the user that has requested this execution.

execution

Execution

NO

NO

Reference to the execution object in which the task is called.

dataShop

DataShop

YES

NO

A reference to the data shop object, that triggered the workflow. This property is only available, if the workflow was triggered by a data shop execution.

executionContext

Map

NO

NO

A map used to store information permanently for an execution of the workflow template. The data of this map will be displayed in the summary of a workflow execution. This map is also returned to the caller of the workflow

runnerFactory

Internal

NO

NO

An internal property used by the invocations of the xxxRunner() methods.

jobExecutor

Internal

NO

NO

An internal executor object that runs objects started by the xxxRunner() methods.

workflowLogWriter

Internal

NO

NO

Internal log writer to capture outputs of some internal methods and write them to the execution log of the workflow.

Example

This workflow triggers the copying of a specific employee. The employee to be copied is stored in the custom parameter employee, which is defined in the workflow template. The employee number is then stored in the executionContext variable, which contains important information that is also displayed in the execution overview.

The TaskTemplateRunner() method selects a task template and creates a new task with the appropriate parameters. The new task is then executed.

  • Groovy

  • Java Script

executionContext.put("employee", workflow.employee)

TaskTemplateRunner()
  .taskTemplate("Copy employee")
  .taskNamePattern("Employee: ${workflow.employee} (\${autoIncrement})")
  .parameter("employee", workflow.employee)
  .parameter("sourceEnvironment", workflow.productionEnvironment)
  .parameter("targetEnvironment", workflow.testEnvironment)
  .run();
executionContext.put("employee", workflow.employee)

TaskTemplateRunner()
  .taskTemplate("Copy employee")
  .taskNamePattern("Employee: " + workflow.employee + " (${autoIncrement})")
  .parameter("employee", workflow.employee)
  .parameter("sourceEnvironment", workflow.productionEnvironment)
  .parameter("targetEnvironment", workflow.testEnvironment)
  .run();