Row Level Delete Task Template
An XDM row level delete task template is a type of task template that is not designed to copy data, but to remove specific rows from a set of tables. It uses relations between the rows to identify the rows to delete.
An XDM delete task template is not defined on a connection but rather on an environment.
XDM honors referential integrity constraints when deleting rows, if these have been described in the form of a data relation rule. This ensures that for every row that is removed from a table, all dependent child rows are also removed. Information about the relationships between tables is extracted from the source RDBMS catalog, or manually defined by data relation rules in the version entities of the environment. Data relation rules reflect relationships defined by the applications that use the tables. Delete tasks can be used to remove all rows that constitute a test case from a set of tables in a development or quality assurance environment without affecting unrelated rows in those tables.
Permissions
Task Templates have specific permissions to manage user access. The table below displays the available permissions and their purposes.
For more details about the concept of XDMs permission management refer to Permission Management.
Permission |
Description |
|---|---|
ADMINISTRATION |
Specifies that the grantee can grant and revoke permissions to and from other users. A user that creates an object automatically receives the |
DELETE |
Specifies that the grantee can delete objects of the selected types. |
DIAGNOSE |
This permission controls access to diagnostic data of a task execution. The diagnostic data consists of the stages, their outputs and the batch reports. |
EXECUTE |
Specifies that the grantee may execute the object, or schedule the object for later execution. This permission only applies to objects that are executable, i.e. tasks, task templates, workflows, workflow templates, and data shops. |
READ |
Specifies that the grantee has read permission on the object. The grantee is able to see the object in lists and can see all of the object’s details, such as rules or access permissions. In addition, the grantee can reference this object.
For example, a user who has |
WRITE |
Specifies that the grantee has the permission to change the settings and attributes of an object. This also includes modifying any rule lists that might be associated with the object (for example, the selection rules of a task template). |
Properties
The table below documents the available properties for task templates. The 'name' column displays the property name as it can be used in Groovy and Java Scripts.
Name |
Type |
Default |
Description |
||||
|---|---|---|---|---|---|---|---|
Object container retention period containerRetentionPeriod |
String |
-1 |
This option specifies when object selection results for a task should be overwritten. If there is an object selection result for a task that is younger than the specified retention period, then XDM will use this result for the task execution. Otherwise, XDM will re-fetch the object structures from the database catalog and create a new object selection result and delete the old one. For example, if the retention period is set to This option is useful if the object structures within the source and target systems do not change between XDM task runs. This means XDM does not need to re-fetch the structures of the required objects from the database catalog. This can save time and resources, especially for large databases with many objects. However, this will lead to problems if the object structures change between task runs. Per default the container retention period is set to -1, which means that XDM should always fetch structure definitions from the database catalog and delete any existing object selection results. You should use a retention period of -1 if object structures change frequently. Furthermore, running a task once with a retention period of -1 can be used to reset the object selection results for a task if the object structures have changed once. A detailed description of the period syntax can be found here. |
||||
Deactivate logging before apply deactivateLoggingBeforeApply |
Boolean |
false |
This flag controls if logging should be disabled before execution the Subset Apply. This flag is supported for PostgreSQL and Db2. Deactivating logging for tables can improve the performance. For example for Db2 z/OS before inserting data starts XDM will make a ALTER TABLESPACE … NOT LOGGED on all tablespace that are affected. After the Subset Apply is done, XDM will activate logging by ALTER TABLESPACE … LOGGED again. For Db2 LUW XDM will execute a ALTER TABLE … ACTIVATE NOT LOGGED INITIALLY to disable logging. For PostgreSQL XDM will execute a ALTER TABLE … SET UNLOGGED to disable logging and ALTER TABLE … SET LOGGED again to enable logging. WARNING: When this property is used, then a recovery of the table is not possible anymore in all cases. Any failure can result in a a useless table, because the DBMS will mark the table as inaccessible. |
||||
Delete execution files on success deleteExecutionFilesOnSuccess |
Boolean |
false |
Specifies if working files of a task execution should be deleted in case of a successful task execution. This reduces the required space for a task and should therefore prevent space problems on the server when many tasks exist. |
||||
|
description |
String |
n/a |
An optional description for this object. The description can contain multiple lines to give more context on the configured object. The description is not used in a technical context. |
||||
|
displayName |
String |
n/a |
Specifies the name of the object. The name is used to display and identify the object in lists. The name can contain any valid UTF-8 characters. |
||||
|
environment |
Environment |
n/a |
Specifies the environment that used in the row level delete task. An environment must be set to execute a task, either in the task template, or in the task. If it is set in both, the environment set in the task is used. The environment must be usable as target environment. |
||||
|
executionPlatform |
String |
default |
This property defines whether and on which platform the Dataflow server is being built for a task or workflow execution. The property on the template is overwritten by the property on the task or workflow. |
||||
|
executionRetentionPeriod |
String |
-1 |
Specifies how long executions are kept for the specific task or workflow.
If an execution is older than the specified retention period, the XDM service will remove it.
If the executions should not be deleted automatically, the retention period must be set to -1. Otherwise, the input must start with
A detailed description of the period syntax can be found here. |
||||
|
extractTraceActive |
Boolean |
false |
This option prepares an additional temporary database, that allows previewing of the extracted data in the user interface. When enabled, the data can be investigated in the task execution detail view using the Schema Browser button. If this option is enabled, additional space and time will be used during the execution of the task. Use this option during task development or debugging, but disable it for productive task usages. For this option to work, the XDM core service needs access to the task directories.
Please ensure, that the |
||||
Fetch keys with temporary table fetchKeysWithTempTable |
Boolean |
false |
This option specifies whether XDM uses a temporary table to fetch keys. If this property is set to true, XDM will use a temporary table instead of OR and AND clauses to fetch key values during data extraction to improve performance. The performance enhancement is typically only taking effect if the extract package size is increased (for example 5.000 as package size). Generally, this feature should only be used when extracting more than 25.000 rows from a table. This currently only works with Db2 for z/OS. However, this property will be deactivated if at least one of the key values consists of one or more null values. Then the conventional behavior is used as a fallback (using OR and AND clauses) to fetch the key values. |
||||
|
jdbcProfiling |
Boolean |
false |
Activate the performance supervision in the associated task template (all task types are supported).
If this option is activated certain warnings in the task logs will be activated and profiling statistics for JDBC queries are logged.
They are available through a task execution export in the task folder |
||||
|
logLevel |
Loglevel |
INFO |
Controls the granularity of the log messages for the XDM task issued directly by XDM. The number of log messages decreases in the following order: Trace → Info → Warning → Error
It is also possible to overwrite the log level in specific program parts. This can be done by adding the prefix Possible values are:
|
||||
Enable resume cache for data apply resumeSubsetApply |
Boolean |
true |
Allows a task to be resumed in the final stage, where the data is written to the target. With this option enabled, a task which fails or is canceled can be resumed at the last written row. The option ensures that rows will not be applied twice. When this option is disabled, then a canceled or aborted task will be resumed from the beginning. Whether rows are applied twice in the target then depends on the Fill mode setting.
|
||||
|
rowSelectionMode |
Enum |
QUERY |
The row selection mode defines how the keys, which will be used to select rows in the start table, are to be determined. The condition defining the start value selection will be stored in the Start Condition.
|
||||
|
startCondition |
String |
n/a |
The start condition defines, depending on the row selection mode, which or how many keys will be used to select rows in the start table. Depending on the row selection mode, the start condition will be set as follows:
|
||||
|
startConditionFile |
File |
n/a |
The start condition CSV file contains selection values that should be used to select data in a row level processing task. The property is only used, if row selection mode File is selected. You can choose an existing XDM file object containing the selection values, or you can upload a new one. If a new file is used, it will be stored as a CSV file in XDM. When using a CSV file for a start condition, the following requirements must be met:
The list Columns in file defines which columns are used in the file. |
||||
|
startConditionFileMapping |
String |
n/a |
Contains a comma-separated list of columns from the start table. This list specifies, how the columns defined in the start condition file are mapped to the columns of the start table. |
||||
|
subsetApplyThreadCount |
Number |
1 |
Defines the maximum number of threads to be used in subset apply to insert data into tables in parallel. Higher numbers can reduce run time, but increase CPU and memory load. |
||||
|
tags |
Tag |
n/a |
Contains the tags that apply to this object. These tags can be used in the search to find objects quickly and effortlessly. |
||||
|
taskExecutionReport |
TaskExecutionReport |
n/a |
This list defines which reports are created at the end of an execution and are displayed in the task execution. The list includes all defined reports to which the current user has READ access. The selected reports are generated at the end of a task or workflow execution.
|
||||
|
traceDataConfiguration |
String |
n/a |
This property specifies a list with one or more values which should be traced during XDM task processing. If multiple values are specified, these should be separated by semicolons. The specified values are reported in a trace data report at the end of a task stage in which the value was detected. Reported are the rows of a table in which one of the specified values occurs in any of the row’s columns, whenever that row is part of an INSERT, UPDATE, DELETE, or SELECT action. If a specified value is detected in a stage as part of one of the above named actions, then the row in which it occurs is reported in the trace data report at the end of that stage. In the event that a specified value is affected by a modification method, the row in which it occurs is reported in the trace data modification report. |
||||
|
usingAliasedTables |
Boolean |
false |
This property enables a task to use an alias on a table as a replacement for this table. This is useful in the following scenario: You have defined a uniform application model that names tables in the rules. In a certain environment the tables are named differently and have an alias for the original name. Then XDM creates a virtual table object in the task process under the name of the alias, and the column list of the underlying table.
|
||||
|
usingViewedTables |
Boolean |
false |
Using viewed tables controls when views should be handled like tables.
In the event that a view references another view or an alias, XDM will follow the chain of references to a depth of 5 steps by default.
You can control the selection depth by setting a custom parameter Views can be used as a source in most cases, even if they are based on multiple tables or using functions to modify column values. For target usage, only views can be used that are based on a single table and reflect the columns without modifying any value. Please refer to the definition on deletable views, insertable views and updatable views of the database system documentation. When inserting on a view, the view will receive the DELETE, UPDATE and INSERT statements and the database system evaluates if the operation is permitted.
|
Actions
The available actions are described below. Some actions apply to the list, while others are specific to selected task templates.
List Actions
The following actions are available on the task templates list. If the action is disabled a tooltip will provide the exact reason for the deactivation. The required permissions are described in detail for each action.
-
Bulk Create Permission
-
Bulk Delete
-
Bulk Export
-
Create
-
List History
Create a new permission on the selected objects. Shows in the result list whether the permission could be granted on the respective object. Only these permissions can be granted that are existing on the underlying object.
A permission in the result list can have three different states, these are:
- CREATED
-
The permission successfully granted on the object.
- MERGED
-
The granted permission already exists on the object and merged with the new permission.
- SKIPPED
-
The permission could not be granted, because of missing administration permission on the object.
The following permissions are required on the list:
-
ADMINISTRATION
-
READ
Delete the selected objects.
The following options are available:
- Cascade
-
Recursively delete depending objects.
|
When using cascade, dependent objects are deleted first also with cascade enabled. Thus, a cascade deletion is a recursive function that deeply searches for dependent objects and deletes them first. There is only a confirmation for the first object. The dependent objects are deleted without confirmation but only when the user has the DELETE permission. This feature is only available in development mode. More information about development mode can be found in the chapter User Settings. It should be used with caution. |
An object in the result list can have two different states, these are:
- DELETED
-
The object could be deleted.
- NOT_DELETED
-
The object could be not deleted. This may be because the executing person does not have a delete permission on the object or the object is still referenced by others. A detailed reason can be determined with the help of the error message. If the object is still in use, these objects are also displayed.
The following permissions are required on the list:
-
DELETE
-
READ
Exports the selected objects.
- YAML
-
Generates a YAML file containing all the object’s settings. The user has the option to download the export file, or to paste the content in the import dialog. The YAML export is particularly suitable for importing the exported objects again via the XDM UI.
- ZIP
-
This export writes several individual YAML-files. Each YAML-file is stored in a directory according to its type. For example, when exporting a native table backup task template named 'A backup template', a YAML-file 'A backup template.yaml' is created inside the directory /TaskTemplate/native-table-backup-task-template/ of the ZIP-file. This kind of export is suitable for usage in git-repositories together with XDM’s configuration as code feature.
Related and dependent objects can optionally be included in the export. The export dialog has the following options:
- Include dependent objects
-
Dependent objects only belong to the exported object like rules and tasks.
- Include permissions
-
Permissions of each exported object, only when the object supports permissions. Some objects like rules don’t have permissions.
- Include referenced objects
-
Referenced objects exist by their own and are used in the exported object like connections and environments.
- Include objects that depend on referenced objects
-
Also include the dependent objects of the referenced objects. E.g. the rules of a modification set or the rules in an application model version.
| Objects on which the user does not have READ permission are not exported. This includes dependent and referenced objects. However, the reference to an object will be exported. For example a connection object would refer to the credential, even if the user does not have READ permission on the credential. The definition of the credential object itself will not be part of the export file. This can lead to issues during the import, because the connection cannot be created without an existing credential. |
The following permissions are required on the list:
-
READ
Creates a new object in the current list. Depending on the object type either a popup dialog is shown for the most important settings, or the complete object is shown in edit mode. The dialog provides the option to create the object and remain in the current list or to switch to the newly created object in edit mode to perform further changes.
The following permissions are required on the list:
-
CREATE
The history list tracks all modifications made to objects within it. A new record is added each time an object is created, edited, or deleted. A record indicates who made the change, which object was affected, and when the change was made.
For more information about the concept of the history refer to the history concepts.
The following permissions are required on the list:
-
READ
Object Actions
The following actions are available on specific task templates. In order to execute the action, the user must possess the necessary permissions for the object. The permissions required for each action are described individually. If the user does not have these permissions, the action will be disabled and the tooltip will provide the exact reason for the deactivation.
-
Check
-
Delete
-
Duplicate
-
Edit
-
Event List
-
Export
-
Object History
-
Permission Check
-
Usage
-
Uses
This action validates the object and its dependencies, reporting configuration errors that could cause issues during task or workflow execution. The validation will cascade through the child objects of the checked objects and objects referenced by them.
For instance, if an installed application of an environment is checked, the check will process the application model, the specified version, the connection, modification sets, and involved modification methods. If an object has rules, all active rules will be checked. The modeling connection and version, including their modification sets and methods, will also be checked. Deactivated objects will not be included in recursive checks, but can be checked individually if the check is executed on the object itself.
Checks often require additional information from the context of the objects being checked, such as necessary connections or custom parameter values. The check will gather information from the objects being checked and use it to perform checks on child objects. Any required additional information must be provided before the check begins. The check queries the user to provide these missing information.
- Database object checks
-
For all rules which reference database objects such as tables, columns, etc, the check verifies that the those objects exist in the database system. If a connection can be inferred from the context, then this connection is used. If no connection is available in the context, it must be specified before the check is executed.
- Connection checks
-
For objects which configure access to external systems, such as connections or storage locations, the configuration check verifies that access can be established using the given credentials. Furthermore, additional operations on database connections are performed to check whether the credential user has the necessary authorization to access relevant database objects. In particular, the credential user’s permission to read source tables and write to target tables is verified. Similarly, for storage locations the check verifies that the credential user has permission to write to the working directory.
- Code checks
-
For all entities containing code segments, such as modification methods or condition scripts, the syntax for the code is checked. This does not check, however, whether at run time all necessary variables are likely to be available.
The following permissions are required:
-
READ
Delete the object. If the object is still used by another entity, an error message is displayed, and the object is not deleted. The delete operation must be confirmed in a separate popup.
The following options are available:
- Cascade
-
Recursively delete depending objects.
|
When using cascade, dependent objects are deleted first also with cascade enabled. Thus, a cascade deletion is a recursive function that deeply searches for dependent objects and deletes them first. There is only a confirmation for the first object. The dependent objects are deleted without confirmation but only when the user has the DELETE permission. This feature is only available in development mode. More information about development mode can be found in the chapter User Settings. It should be used with caution. |
The following permissions are required:
-
DELETE
-
READ
Will create an exact copy of the current object with a different display name in the same list. Users can decide whether they want to copy child objects like rules, permissions or tasks. It is only possible to select complete classes of objects and not to select individual child objects. Copied child-objects will preserve their display name. The default is to copy all child objects.
The following permissions are required:
-
CREATE
-
READ
Opens the current entity in edit mode.
The following permissions are required:
-
READ
-
WRITE
This list shows all registered events for the object. It includes events that are specific to the object, or for that type.
The following permissions are required:
-
READ
This action allows to export XDM objects in different formats in order to import them via export or CasC in another environment.
Refer to configuration of export for more information.
Related and dependent objects can optionally be included in the export. The export dialog has the following options:
- Include dependent objects
-
Dependent objects only belong to the exported object like rules and tasks.
- Include permissions
-
Permissions of each exported object, only when the object supports permissions. Some objects like rules don’t have permissions.
- Include referenced objects
-
Referenced objects exist by their own and are used in the exported object like connections and environments.
- Include objects that depend on referenced objects
-
Also include the dependent objects of the referenced objects. E.g. the rules of a modification set or the rules in an application model version.
- Include implicit created objects
-
Implicit created objects are tasks or workflows which were automatically created for execution. These objects won’t be exported by default, but can be included by setting this flag. When exporting implicit objects, make sure that the
Include dependent objectsflag is also enabled.
|
Objects on which the user does not have For example a connection object would refer to the credential, even if the user does not have |
The following permissions are required:
-
READ
The history displays all changes made to the respective XDM object, including any changes made to its rules.
Each change record includes information about the operation performed (e.g. CREATE, UPDATE, DELETE), the timestamp, and the user responsible for the change.
For more information about the concept of the history refer to the history concepts.
The following permissions are required:
-
READ
The check verifies that the current user has the authorization to access the object. The check can also be performed for a specific user or role, if needed. By default, the check is performed using the current user’s credentials. It is then applied to child and referenced objects.
Additional permission checks are applied when these can be inferred from the context in which the check was started. For example, if the check is performed on a table copy task, the referenced source and target connections are checked to determine whether the given identity has source or target usage permission respectively.
The following permissions are required:
-
READ
The Usage List shows all objects that refer to the current object. It provides an overview of the relationships and makes it easy to track these relationships.
The following permissions are required:
-
READ
The Uses List shows all objects that the current object uses. It provides an overview of the relationships and makes it easy to track these relationships.
The following permissions are required:
-
READ
License Options
This object is available if the following license option is enabled:
-
TASK_TYPE:SUBSET_DELETE_TASK
The object is also available if the license package is at least: ESSENTIAL.
Prerequisites
Before a row level delete task template can be completely configured you have to set up an environment, which involves a connection with credential, an application model with a version and installed applications on the environment, where one installed application is marked as start model.
Required permissions
Users
During a task execution, a row level delete task runs under the authorization of the source database user which is defined in the credential of the connection that is used in the installed application.
Authorization for source database user
-
SELECTauthority for the database catalog, -
SELECTauthority for the tables to be processed, -
DELETEauthority to remove data from tables to be processed. -
DB2 LUW only: System monitor authority (
SYSMON) to obtain information from the database catalog (optional, but recommended) -
MS SQL Server only: The user must be a database user, a domain user is not sufficient
-
Oracle only: If the batch mode is used,
CREATE TABLEis required to create the error table to store potential discards.
Possible side effects
Tables can be referenced by foreign keys that are not represented by data relation rules in the application model version that is being used. In addition, it is possible to define triggers on tables that are activated when rows are deleted. Be aware of the following possible restrictions and side effects:
-
When deleting rows from tables that are parent tables in a foreign key relation, and the foreign key was defined with the
ON DELETE CASCADEclause, the database system will automatically delete the corresponding of from the child table. -
When deleting rows from tables that are parent tables in a foreign key relation, and the foreign key was defined with the
ON DELETE SET NULLclause, the database system will automatically set the foreign key fields toNULLin all corresponding rows of the child table. -
When deleting rows from tables that are parent tables in a foreign key relation, and the foreign key was defined with the
ON DELETE RESTRICTclause, the database system will prevent the deletion of those rows and SQL errors may occur. -
When deleting rows from tables with a trigger that is activated before or after rows are deleted, the trigger code is executed by the database system and may cause side effects.
-
When deleting rows from tables with a trigger that is activated instead of the row deletion, rows may not be deleted as intended. In addition, the trigger code is executed by the database system and may cause side effects.
XDM Permissions to execute a task
The user needs the following permissions to successfully start the task execution:
-
READpermission on the task and task template, -
EXECUTEpermission on the task, -
TARGET_USAGEon the environment, -
READpermission on any credential used in the connections linked in the installed applications, -
READpermission on any application model linked from the environments, -
READpermission on any custom parameter used in the task template.
Task procedure
A XDM row level delete task contains three stages. Before executing Stage 1, XDM creates all necessary files and folders in the task directory during the Tailoring process.
- Tailoring
-
All necessary files and folders are created in the task directory.
The following reports are generated by this stage: - Stage 1
-
The structure of the environment is cached and stored in task files. XDM will fetch the start table and all tables related by data relation rules. Data relation rules can be generated from database foreign keys. Tables and columns that match any exclude rule will not be in the task’s selection set.
The following reports are generated by this stage: - Stage 2
-
XDM selects the data to delete from the environment. The data selection is made by using
SELECTstatements, starting at the start table with the start condition. Depending on the definition of data relation rules, data from the parent and child tables is also selected.
The following reports are generated by this stage: - Stage 3
-
The selected data will be deleted from the environment.