Delete expired Objects

Overview

This guide describes, how a task stage hook can be used to delete expired executions, workflows, tasks and also icebox generations. It can be used to have a single point of control to clean up old objects.

Preliminary thoughts

Before using this task stage hook you should think about how long you will have to keep objects in XDM. Executions, workflows and tasks may be kept for auditing purposes to see what data was copied and when or to rerun a specific execution. For icebox generations the time to keep them mostly depends on how long the data will be needed. This is the reason why this script uses the iceboxRetentionPeriod instead of just using the creation date of the generations to decide when the generation should be deleted. The iceboxRetentionPeriod can be set for all icebox generations of a task template, or it can be set for each icebox generation individually.

Process Steps

We will create a task stage hook to identify objects in XDM that have reached a specific age or expiration date and should be deleted.

Permissions required for this task stage hook

To execute this task stage hook successfully the user executing it requires read permissions for the executed task list, the workflow template list and all task template lists. Also, this user needs read and delete permissions for all executions, all workflow templates and task templates.

Set the Backup Retention Period for icebox generations

For icebox generations we will use the Backup retention period (variable name iceboxRetentionPeriod) to specify how long an icebox generation should be stored. For more details refer to the retention description in the reference.

The Backup retention period is used by XDM itself also to delete icebox generations. To use an individual date, it is possible to add a custom parameter to the icebox generation and customize the delete workflow.

Create a task stage hook for the cleanup process

The task stage hook iterates over the list of executions, workflow templates and task templates to find all executions, workflows, tasks and icebox generations. For executions, workflows and tasks the last execution timestamp is used to decide if the execution, the workflow or the task should be deleted. For icebox generations the Backup Retention Period.

Settings

  • url: URL to the API of the XDM instance. This URL depends on the XDM instance

  • retentionPeriod: The period how long a workflow, task or execution should be kept

  • retentionCount: How many workflows and tasks should be kept at minimum per template

  • implicitOnly: Specifies if only implicit created tasks and workflows should be deleted

  • dryRun: Specifies if only a dry run will be executed to generate a list of objects that should be deleted

  • credentialName The name of a credential that will be used to connect to the public API of XDM to read and delete the objects

Setup Environment ID

To run this task stage hook on different environments, it should be customized to use specific parameters depending on the current environment. You can set up an environment-id for each instance of XDM which then in the script can be used to find out on which instance the workflow currently runs. Please find the description to set up in xref:GuidesAndRepositories/Guides/SetupEnvironmentIdAsEnvironmentVariable.adoc

Set up the task stage hook

The complete hook can be found here.

The following part of the task stage hook is used to customize the hook. The variables control which objects should be deleted and which objects should be kept. The example shows how to set the control variables once and overwrite them for specific environments. This is useful to use for example a retention period of 730 days per default, but overwrite it to 30 days in the test systems.

def retentionPeriod = "P730D"                       // The period how long a workflow, task or execution should be kept
def retentionCount = 250                            // How many workflows and tasks should be kept at minimum per template
def implizitOnly = true                             // Specifies if only implizit created tasks and workflows should be deleted
def dryRun = true                                   // Specifies if only a dry run will be done generating a list of objects that should be deleted

/*
 * Switch settings based on the current XDM instance
 */
def environment = System.getenv("xdm.core.environment-id")
switch (environment) {
    case 'int':
        retentionPeriod = "P30D"                        // The period how long a workflow, task or execution should be kept
        retentionCount = 30                             // How many workflows and tasks should be kept at minimum per template
        //implizitOnly = true                             // Specifies if only implizit created tasks and workflows should be deleted
        //dryRun = true                                   // Specifies if only a dry run will be done generating a list of objects that should be deleted
        break
        retentionPeriod = "P30D"                        // The period how long a workflow, task or execution should be kept
        retentionCount = 30                             // How many workflows and tasks should be kept at minimum per template
        //implizitOnly = true                             // Specifies if only implizit created tasks and workflows should be deleted
        //dryRun = true                                   // Specifies if only a dry run will be done generating a list of objects that should be deleted
        break
    default:
        retentionPeriod = "P250D"                       // The period how long a workflow, task or execution should be kept
        retentionCount = 730                            // How many workflows and tasks should be kept at minimum per template
        //implizitOnly = true                             // Specifies if only implizit created tasks and workflows should be deleted
        //dryRun = true                                   // Specifies if only a dry run will be done generating a list of objects that should be deleted
        break
}

Schedule the task stage hook execution

To run this task stage hook create a workflow template and workflow calling this task stage hook. Then schedule this workflow to run for example once a day.

Conclusion

This approach provides an easy way to clean up old objects and keep your XDM clean. With this task stage hook the old stage hooks deleting expired objects which had to be added to each single workflow template and task template can be replaced. This makes it much easier to control the cleanup process, because it is not spread over several objects.