Stage Hooks

Stage hooks are pieces of code that can be executed before or after the individual stages of an XDM task. You can use stage hooks to integrate other processes with the XDM task or to trigger external actions that go beyond the native capabilities of an XDM task.

Types of stage hooks

There are three types of stage hooks:

  • Ant stage hooks

  • Groovy stage hooks

  • JavaScript stage hooks

Ant stage hooks are Apache Ant scripts and can access the standard Ant tasks as well as XDM specific Ant tasks. JavaScript stage hooks are pieces of arbitrary JavaScript code. Groovy stage hooks are pieces of arbitrary Groovy code.

Creating a stage hook

We are going to create a simple task stage hook that executes the df command in order to show the available disk space on the system where the hook is executing.

Ant comes with a pre-defined Ant task called exec, which can be used to execute a system command. On a Linux or Unix server, the df command shows a file system disk space usage report. The hook will execute this command using the invocation parameter -h, which causes the program to print the available disk space in a more human-readable format.

To create a new stage hook:

  1. In the left sidebar, click the menu category Tasks to expand the tasks menu. Then click the menu item Task Stage Hooks.

  2. Click the button + Create. A panel titled Create Task Stage Hook opens. Enter the following information:

    Name

    Show free disk space

  3. Click the button Create and edit.

  4. Optionally enter a description. Under Script language, make sure Ant is selected.

  5. Enter the following text into the Code field:

    <?xml version="1.0"?>
<project name="Hook" default="run">
  <import file="${inst.dir}/config/taskdef.xml"/>
  <target name="run">
    <exec executable="df">
      <arg value="-h"/>
    </exec>
  </target>
</project>

  6. Click the button Save Changes.

Using a stage hook

A hook can be used by multiple tasks, and tasks can have multiple hooks. A task can also invoke the same hook more than once. You can either assign a hook to a task template or to a task. When assigned to a task template, all tasks that are derived from that template will invoke the hook when they are running. When assigned to a task, only that task will invoke the hook when it is running.

XDM tasks have different stages, and you can specify that a hook may run before or after a stage of the task.

In this tutorial, we are going to configure the native table copy task template Copy production schema so that it invokes the hook that displays the available disk space at the very beginning of the process.

Assigning a hook to a template

To assign a hook to a task template:

  1. In the left sidebar, click the menu category Tasks to expand the tasks menu. Then click the menu item Task Templates.

  2. In the tab Table Copy, click the button Button with an arrow pointing to the right-hand direction on the left side of Native Table Copy Task Templates to expand the list. The list should contain the task template Copy production schema. Click the button Button with an arrow pointing to the right-hand direction next to the template.

  3. Open the panel Others at the bottom of the page.

  4. Then select the tab Hook usages and click the button + Create.

  5. A panel titled New Hook Usage opens. Enter the following information:

    Description

    (Optional, may be left empty)

    Active

    (check the box)

    Stage

    Select Stage1

    Hook phase

    Select Pre

    Task stage hook

    Select Show free disk space

  6. Click the button Save Changes.

Verifying the hook usage

To verify that the hook is invoked correctly, you can re-run a task that is based on the template to which the hook was assigned.

To re-run the task Copy production schema to qa1:

If you are still in the panel displaying the task template Copy production schema to qa1, then skip to step 3.

  1. In the left sidebar, click the menu category Tasks to expand the tasks menu. Then click the menu item Task Templates.

  2. In the tab Table Copy click the button Button with an arrow pointing to the right-hand direction on the left side of Native Table Copy Task Templates to expand the list. The list should contain the task template Copy production schema. Click the button Button with an arrow pointing to the right-hand direction next to the template.

  3. In the bottom panel Tasks, select the tab Tasks and click the button Button with an arrow pointing to the right-hand direction on the left of the entry Copy production schema to qa1.

  4. In the right sidebar, click the button Execute. A dialog window opens. Leave Interrupt execution unchecked and click the button Execute and view. This will schedule the task for immediate execution and switch the main view of Executed Tasks.

  5. The task runs asynchronously in the background. Every five seconds, the status is refreshed automatically.

  6. After the tailoring step has been executed, but before Stage 1 starts to run, the stage hook will be executed. In the panel Chronological View, you will see an entry for the stage hook:

    Screenshot of an task execution including the stage hook step

  7. To the right of the entry for the stage hook, click the three dots (Button with three dots). A context menu will open. Click the button Show free disk space. You should see the output of the hook. The end of the output should look similar to this:

    run:
     [exec] Filesystem                Size      Used Available Use% Mounted on
     [exec] overlay                 114.8G     41.8G     67.2G  38% /
     [exec] tmpfs                    64.0M         0     64.0M   0% /dev
     [exec] tmpfs                     3.8G         0      3.8G   0% /sys/fs/cgroup
     [exec] shm                      64.0M         0     64.0M   0% /dev/shm
     [exec] /dev/sda4               114.8G     41.8G     67.2G  38% /xdm/backups
     [exec] /dev/sda4               114.8G     41.8G     67.2G  38% /xdm/config
     [exec] /dev/sda4               114.8G     41.8G     67.2G  38% /xdm/data
     [exec] /dev/sda4               114.8G     41.8G     67.2G  38% /xdm/tasks
     [exec] /dev/sda4               114.8G     41.8G     67.2G  38% /etc/resolv.conf
     [exec] /dev/sda4               114.8G     41.8G     67.2G  38% /etc/hostname
     [exec] /dev/sda4               114.8G     41.8G     67.2G  38% /etc/hosts
     [exec] tmpfs                     3.8G         0      3.8G   0% /proc/asound
     [exec] tmpfs                     3.8G         0      3.8G   0% /proc/acpi
     [exec] tmpfs                    64.0M         0     64.0M   0% /proc/kcore
     [exec] tmpfs                    64.0M         0     64.0M   0% /proc/keys
     [exec] tmpfs                    64.0M         0     64.0M   0% /proc/latency_stats
     [exec] tmpfs                    64.0M         0     64.0M   0% /proc/timer_list
     [exec] tmpfs                    64.0M         0     64.0M   0% /proc/sched_debug
     [exec] tmpfs                     3.8G         0      3.8G   0% /proc/scsi
     [exec] tmpfs                     3.8G         0      3.8G   0% /sys/firmware

  8. Eventually, the task will finish. In the top left corner, the task status should show COMPLETE.

Sample stage hook

Execute custom SQL statements

You can use stage hooks to run custom SQL statements. This can be useful to make updates to target tables after XDM has copied the requested data. For example, an application may use a table to store the highest number that is currently assigned to an ID column of a number of other tables.

In this example, there is a table MAX_ASSIGNED_IDS with the following table definition:

CREATE TABLE QA1.MAX_ASSIGNED_IDS
(TABLE_NAME VARCHAR(100)
, MAX_ID     VARCHAR(20)
, PRIMARY KEY (TABLE_NAME)
);

Assume that the table keeps track of the maximum value of the column emp_no of the table employees, and of the maximum value of the column dept_no of the table departments.

After using an XDM Row Level Processor Task (Copy employees by department name) to copy a department and all of its employees into a target environment, you may want to update the values in this table. You can create the following stage hook and configure the task to execute it after Stage 6 of the Row Level Processor Task:

To create a new stage hook:

  1. In the left sidebar, click the menu category Tasks to expand the tasks menu. Then click the menu item Task Stage Hooks.

  2. Click the button + Create. A panel titled Create Task Stage Hook opens. Enter the following information:

    Name

    Execute custom SQL statements

  3. Click the button Create and edit.

  4. Optionally enter a description. Under Script language, make sure Groovy is selected.

  5. Enter the following text into the Code field:

import groovy.sql.Sql
def startModel = task.targetEnvironment.startModel
def connection = startModel.connection.jdbcConnection //<-- Connection Target
def t_schema = startModel.tableSchema //<-- Schema


def sqlStatements = [
      """
      CREATE TABLE IF NOT EXISTS ${t_schema}.max_assigned_ids(
      table_name VARCHAR(100),
      max_id     VARCHAR(20),
      PRIMARY KEY (TABLE_NAME)
      );
     """ ,
      """
      INSERT INTO ${t_schema}.max_assigned_ids (table_name, max_id)
      VALUES
        ('departments', NULL),
        ('employees', NULL)
      ON CONFLICT (table_name) DO NOTHING;
     """ ,
      """
      UPDATE ${t_schema}.max_assigned_ids
      SET max_id = (SELECT MAX(dept_no)::VARCHAR FROM ${t_schema}.departments)
      WHERE table_name = 'departments';
     """ ,
      """
      UPDATE ${t_schema}.max_assigned_ids
      SET max_id = (SELECT MAX(emp_no)::VARCHAR FROM ${t_schema}.employees)
      WHERE table_name = 'employees';
     """
]


def sql = Sql.newInstance(connection)

try {
    sqlStatements.eachWithIndex { statement, idx ->
        // Ensure we work with a plain String and trim whitespace
        def text = statement.toString().trim()
        if (!text) {
            println("Statement ${idx + 1} skipped (empty)")
            return
        }

        try {
            println("Executing statement ${idx + 1}:")
            println(text)
            def result = sql.execute(text)
            println("-> Success \n")
        } catch (Exception e) {
            println("Error in statement ${idx + 1}:")
            println(text)
            println("${e.class.name}: ${e.message}")
        }
    }
} finally {
    // Always close the Sql object to free resources
    sql.close()
}
If the table does not exist in your target system, this stage hook will create it.

Previous section: Delete tasks | Next section: Configuration of a Complex Data Shop