What is XDM ?

Motivation

To ensure software quality, there are a number of well-defined hierarchical test phases, each with their own particular types of tests. In most companies there will typically be a number of different database environments. Foremost among these is the production environment. Clearly software testing cannot take place in this environment. Thus, there will typically also be a number of testing environments. To supply these environments with usable data, the production data will usually be used as a basis. To conform with various legislative requirements, test data should be masked so that there is no possibility of inferring the original production data to identify individuals and companies.

It is often the case that there are also changes to the database to be applied at some time in the future. Obviously the applications which use the new database structure would require test data based on the new structure. This adds an extra level of complexity to the management of test environments.

How does XDM Help?

XDM supports all aspects of test data provision. It is a tool which enables users to define and execute tasks to accomplish the copying of data from a source to a target environment, whilst simultaneously performing the required masking. To this end there are three broad categories of XDM tasks for transporting data:

  1. The Database Clone Task creates a 1:1 clone of a database under a different name.

  2. A Table Copy Task copies complete tables from a source to a target environment. The tables to be copied are defined with the aid of a variety of rules.

  3. A Row Level Processor Task allows the selection of specific rows from a set of related tables in the source environment to be copied to a target environment, whilst maintaining relational integrity.

Table copy tasks and row level processor tasks also have associated Icebox components for creating and extracting generations of test data. More Information about the icebox concepts can be found in the section Choosing a generation in the chapter Using the Icebox.

XDM also provides a rich palette of features which enable the precise definition of test data requirements and easy test data retrieval. Relational integrity is maintained when data is masked. Tasks can be configured to run against current and future database structures without the need to redefine the tasks each time. With authorization management and the Data Shop, XDM provides a complete test data management system.

The Fundamentals

Task Types

The three categories of task mentioned above build upon one another in a hierarchy. The Database Cloning (DC) component at the database level is the basis. The Table Copying (TC) component (table level) builds on this. The Row Level Processor (RLP) component is the most refined level, which, as the name suggests, deals with subsets of rows within tables.

A complete overview of all XDM task types is to be found in the chapter Overview of XDM Tasks

Table Level

As mentioned above, the table copying (TC) component deals with whole tables. It allows for differing naming conventions in the source and target environments. It compares the source and target environments to verify compatibility and can generate DDL to create missing database objects in the target environment.

Table copying is managed by defining a number of rules. The tables to copy are determined by defining selection rules. If certain tables that fit the selection rules are not wanted, then these can be removed from the selection set by defining exclude rules. Different naming conventions are accommodated by defining mapping rules to rename database objects. The same rules can also be applied to related database objects such as indexes, views, and sequences.

The TC component can transfer data from one database type to another. The transport mechanism is configurable. XDM will always attempt to use the most efficient mechanism available. Where possible, XDM favors the use of database specific tools to unload and load tables. If this is not possible, XDM will fall back to a generic format. As a last option XDM can use SQL data transport, for instance if the users are not authorized to use other tools. With the TC component XDM is able to mask data during the copy.

More Information about these tasks can be found in the section Table Copy Tasks.

Row Level

The row level processor (RLP) component represents the finest level of data extraction. Unlike TC, RLP does not use selection rules to specify the data to be copied. Instead, it employs the concept of an application model which is applied to an environment. This includes the definition of a start table, data relation rules, data apply rules, and other criteria to establish the boundary conditions for table and row selection. As with TC, the RLP component allows the modification (masking) of data during the copy.

Rows are selected from the start table by specifying a number of rows, a percentage of the table, an SQL query, or a file containing the keys to be retrieved. Data relation rules tell XDM which rows to retrieve from related tables. Data apply rules define the order in which related rows are inserted. The specification of the conditions which apply to the selection process are highly configurable. This allows relational integrity to be maintained whilst allowing the user to retrieve precisely the data they require for testing. Related database objects, such as indexes, views, and sequences, are included in the selection set if required.

These capabilities also apply to the associated RLP Icebox component. The user can back up and restore specific rows in a set of tables. The data can be modified in the process.

The delete task also operates at the row level. Data relation rules can be defined so that when a row is removed from a table, referencing rows in related tables will also be removed. This ensures that referential integrity between tables will be maintained.

More Information about these tasks can be found in the section Row Level Processing Tasks.

Analyse

Analyzing structures and data of a database is also part of XDM.

The structure compare task is used to examine specific attributes of source and target databases and reports structural differences. It can also generate DDL to create objects in the target environment.

The PII (personally identifiable information) finder task examines the contents of tables to determine whether or not they contain data which must be masked.

More Information about these tasks can be found in the section Analyse Tasks.

Application Models and Environments

Application models enable the management of relationships between tables. This is of particular importance for task processing at the row level. These relationships can change with time, so an application model can have multiple versions. This allows testing of existing and future versions of the data without the need to modify task definitions.

Usually a number of test environments are required to reflect the requirements of different projects. It is undesirable to have to redevelop the test data extraction for each case. For this reason XDM allows the definition of different environments which can then be applied to different tasks. Each environment specifies a particular application model version. Thereby the same task can be used to extract data in a variety of environments.

Further information can be found in the chapters Application model, Version, Environment, and Installed application.

Data Masking

Data masking is done on the fly during the copy process. The modification is performed with JavaScript. XDM ships with a number of predefined scripts which users can copy and adapt to their requirements. XDM also includes lookup tables of values which can be used for substitution. These include tables with first names, last names, addresses, and banks.

For further information on data masking, see the chapter Masking in XDM.

Data Shop - Test Data on Request

In order to create tasks as described above the user would need fairly detailed knowledge of the existing databases, schemas, and tables. In particular, for RLP tasks the user also needs to understand how the tables are related to each other. This knowledge is necessary to properly define XDM task templates, but an application developer or tester may not necessarily be very familiar with the internal database structures. They simply want to order new test data.

To maximize the benefit for potential users XDM has a so-called Data Shop that allows users to request the execution of a pre-defined task with the click of a button. The task developer can define parameters for the task template. The parameter values are then entered by the requesting user when ordering fresh test data.

This mechanism allows a large number of end users to benefit from the expert knowledge of database architects.

For further information on the data shop, see the chapter XDM Data Shop XDM Data Shop.