Row Level Processor Task Template

Specifies that the grantee can grant and revoke permissions to and from other users.

A user that creates an object automatically receives the ADMINISTRATION permission on that object.

Specifies that the grantee can delete objects of the selected types.

This permission controls access to diagnostic data of a task execution. The diagnostic data consists of the stages, their outputs and the batch reports.

Specifies that the grantee may execute the object, or schedule the object for later execution. This permission only applies to objects that are executable, i.e. tasks, task templates, workflows, workflow templates, and data shops.

Specifies that the grantee has read permission on the object. The grantee is able to see the object in lists and can see all of the object’s details, such as rules or access permissions.

In addition, the grantee can reference this object. For example, a user who has READ permission on a credential object can refer to this credential object when creating a new database connection.

Specifies that the grantee has the permission to change the settings and attributes of an object. This also includes modifying any rule lists that might be associated with the object (for example, the selection rules of a task template).

Boolean

true

This option specifies whether XDM checks uniqueness of modified unique key fields. When a modification method of modification type COLUMN modifies a column which is part of a unique key, XDM will check whether all output values of the modification are unique. This only checks the values processed by the task and does not involve other values that might exist in the target table.

String

-1

This option specifies when object selection results for a task should be overwritten.

If there is an object selection result for a task that is younger than the specified retention period, then XDM will use this result for the task execution. Otherwise, XDM will re-fetch the object structures from the database catalog and create a new object selection result and delete the old one.

For example, if the retention period is set to P5D, and the last execution of the task was no more than four days ago, then XDM will use the existing object selection result for the task execution and skip object selection for the new run. However, if the last execution of the task was five or more days ago, then XDM will re-fetch the object structures from the database catalog and delete the existing object selection result.

This option is useful if the object structures within the source and target systems do not change between XDM task runs. This means XDM does not need to re-fetch the structures of the required objects from the database catalog. This can save time and resources, especially for large databases with many objects. However, this will lead to problems if the object structures change between task runs.

Per default the container retention period is set to -1, which means that XDM should always fetch structure definitions from the database catalog and delete any existing object selection results. You should use a retention period of -1 if object structures change frequently. Furthermore, running a task once with a retention period of -1 can be used to reset the object selection results for a task if the object structures have changed once.

A detailed description of the period syntax can be found here.

Boolean

false

This flag controls if logging should be disabled before execution the Subset Apply. This flag is supported for PostgreSQL and Db2. Deactivating logging for tables can improve the performance. For example for Db2 z/OS before inserting data starts XDM will make a ALTER TABLESPACE …​ NOT LOGGED on all tablespace that are affected. After the Subset Apply is done, XDM will activate logging by ALTER TABLESPACE …​ LOGGED again. For Db2 LUW XDM will execute a ALTER TABLE …​ ACTIVATE NOT LOGGED INITIALLY to disable logging. For PostgreSQL XDM will execute a ALTER TABLE …​ SET UNLOGGED to disable logging and ALTER TABLE …​ SET LOGGED again to enable logging. WARNING: When this property is used, then a recovery of the table is not possible anymore in all cases. Any failure can result in a a useless table, because the DBMS will mark the table as inaccessible.

Boolean

false

Specifies if working files of a task execution should be deleted in case of a successful task execution. This reduces the required space for a task and should therefore prevent space problems on the server when many tasks exist.

String

n/a

An optional description for this object. The description can contain multiple lines to give more context on the configured object. The description is not used in a technical context.

String

n/a

Specifies the name of the object. The name is used to display and identify the object in lists. The name can contain any valid UTF-8 characters.

String

default

This property defines whether and on which platform the Dataflow server is being built for a task or workflow execution. The property on the template is overwritten by the property on the task or workflow.

String

-1

Specifies how long executions are kept for the specific task or workflow. If an execution is older than the specified retention period, the XDM service will remove it. If the executions should not be deleted automatically, the retention period must be set to -1. Otherwise, the input must start with P for a date period or with PT with a time period.

A detailed description of the period syntax can be found here.

Number

50

The number of packages that can be cached by the subset extractor, before the process is slowed down to empty the cache. A bigger cache leads to a better performance, but also uses more memory. It also influences the number of rows that are inserted with a single query. This can lead to queries which are too long for the target database system.

Number

1

Defines the number of threads that are used during subset extract. Using more threads in subset extract can enhance the processing speed, but uses more memory.

Boolean

false

This option prepares an additional temporary database, that allows previewing of the extracted data in the user interface. When enabled, the data can be investigated in the task execution detail view using the Schema Browser button.

If this option is enabled, additional space and time will be used during the execution of the task. Use this option during task development or debugging, but disable it for productive task usages.

For this option to work, the XDM core service needs access to the task directories. Please ensure, that the /xdm/tasks directories of your xdm-dataflow service, and your xdm-core service are mounted to the same storage.

Boolean

false

This option specifies whether XDM uses a temporary table to fetch keys. If this property is set to true, XDM will use a temporary table instead of OR and AND clauses to fetch key values during data extraction to improve performance.

The performance enhancement is typically only taking effect if the extract package size is increased (for example 5.000 as package size). Generally, this feature should only be used when extracting more than 25.000 rows from a table.

This currently only works with Db2 for z/OS.

However, this property will be deactivated if at least one of the key values consists of one or more null values. Then the conventional behavior is used as a fallback (using OR and AND clauses) to fetch the key values.

Enum

INSERT

The fill mode defines how source rows are inserted into the corresponding tables of the target. It is used in all row level processing tasks, as well as in table copying tasks in which SQL data transport is activated.

Enum

ABORT

Defines XDM behavior, when source and target table structures are deemed to be incompatible.

Boolean

false

Activate the performance supervision in the associated task template (all task types are supported). If this option is activated certain warnings in the task logs will be activated and profiling statistics for JDBC queries are logged. They are available through a task execution export in the task folder files/performanceSupervision.

Loglevel

INFO

Controls the granularity of the log messages for the XDM task issued directly by XDM. The number of log messages decreases in the following order: Trace → Info → Warning → Error

It is also possible to overwrite the log level in specific program parts. This can be done by adding the prefix logger.level. + package name via a task stage hook to the global property map. For example, to set a specific log level to all components of the package de.ubs.xdm.batch.subsetapply you have to set the property logger.level.de.ubs.xdm.batch.subsetapply with the wanted required log level. inside a task stage hook. Supported values are info, warning, error, verbose and debug. A example how to set a property in a task stage hook can be found here. Typically, the support team will instruct to set a package log level for detailed analysis if necessary.

Possible values are:

ModificationSet

n/a

The property contains a list of modification sets that can be used to modify data during the task execution. For every modification set it is also stored, if it is selected or not. Only selected modification sets are shown in the view mode of the object.

Boolean

false

By enabling this property, values affected by modification methods will be checked if they can be inserted in the target. The task will abort if this is not possible.

Boolean

true

Allows a task to be resumed in the final stage, where the data is written to the target. With this option enabled, a task which fails or is canceled can be resumed at the last written row. The option ensures that rows will not be applied twice. When this option is disabled, then a canceled or aborted task will be resumed from the beginning. Whether rows are applied twice in the target then depends on the Fill mode setting.

Enum

QUERY

The row selection mode defines how the keys, which will be used to select rows in the start table, are to be determined. The condition defining the start value selection will be stored in the Start Condition.

Environment

n/a

Specifies the environment that is used as source environment in the task. A source environment must be set in either the task template or the task. If it is set in both, the environment set in the task is used.

String

n/a

The start condition defines, depending on the row selection mode, which or how many keys will be used to select rows in the start table. Depending on the row selection mode, the start condition will be set as follows:

File

n/a

The start condition CSV file contains selection values that should be used to select data in a row level processing task. The property is only used, if row selection mode File is selected.

You can choose an existing XDM file object containing the selection values, or you can upload a new one. If a new file is used, it will be stored as a CSV file in XDM.

When using a CSV file for a start condition, the following requirements must be met:

The list Columns in file defines which columns are used in the file.

String

n/a

Contains a comma-separated list of columns from the start table. This list specifies, how the columns defined in the start condition file are mapped to the columns of the start table.

Boolean

true

Defines, whether discards should be aggregated in the report or not. If set to true, all discards during subset apply will be aggregated. Otherwise, the discards will be listed per row.

Number

1

Defines the maximum number of threads to be used in subset apply to insert data into tables in parallel. Higher numbers can reduce run time, but increase CPU and memory load.

Tag

n/a

Contains the tags that apply to this object. These tags can be used in the search to find objects quickly and effortlessly.

Environment

n/a

Specifies the environment that is used as the target in the task. A target environment must either be specified in the task template or in the task. If it is specified in both, then the environment specified in the task is used.

TaskExecutionReport

n/a

This list defines which reports are created at the end of an execution and are displayed in the task execution. The list includes all defined reports to which the current user has READ access. The selected reports are generated at the end of a task or workflow execution.

Enum

1

If a source table differs in structure from a destination table you have to specify how differences are treated while copying it from source to target. The values are still written to the target database as they are. It depends on the handling of the database if there is a truncation or an error message. It is necessary to use a modification rule on the respective column to implement a proper data truncation.

For more details, see toleration mode in Comparing and Generating Objects.

String

n/a

This property specifies a list with one or more values which should be traced during XDM task processing. If multiple values are specified, these should be separated by semicolons. The specified values are reported in a trace data report at the end of a task stage in which the value was detected. Reported are the rows of a table in which one of the specified values occurs in any of the row’s columns, whenever that row is part of an INSERT, UPDATE, DELETE, or SELECT action.

If a specified value is detected in a stage as part of one of the above named actions, then the row in which it occurs is reported in the trace data report at the end of that stage. In the event that a specified value is affected by a modification method, the row in which it occurs is reported in the trace data modification report.

Boolean

false

Defines whether XDM should use the unmodified keys for generating update statements instead of the modified keys, which is the default behavior for the modification of key columns. This property only has effect if a modification rule is defined on a key column and scope of the modification is set to scope target.

It is not possible to update a key to a value that already exists in the target table. Furthermore, this mode is currently only supported for the fill mode Only update existing rows. This mode is incompatible with filtering of values in modification methods.

Boolean

false

This property enables a task to use an alias on a table as a replacement for this table. This is useful in the following scenario: You have defined a uniform application model that names tables in the rules. In a certain environment the tables are named differently and have an alias for the original name. Then XDM creates a virtual table object in the task process under the name of the alias, and the column list of the underlying table.

Boolean

false

Using viewed tables controls when views should be handled like tables. In the event that a view references another view or an alias, XDM will follow the chain of references to a depth of 5 steps by default. You can control the selection depth by setting a custom parameter objSel_viewSelectionDepth with your preferred value.

Views can be used as a source in most cases, even if they are based on multiple tables or using functions to modify column values. For target usage, only views can be used that are based on a single table and reflect the columns without modifying any value. Please refer to the definition on deletable views, insertable views and updatable views of the database system documentation. When inserting on a view, the view will receive the DELETE, UPDATE and INSERT statements and the database system evaluates if the operation is permitted.