Accessing the transfer tool

The transfer tool exports, imports and compares objects between multiple XDM environments. It is a command line tool based on a docker container.

docker run docker.ubs-hainer.com/xdm3-transfer <command>

To pull the docker image from the UBS Hainer repository, the user need to configure the login for the repository in their local docker client using docker login docker.ubs-hainer.com. They can retrieve the necessary credentials from the XDM support.

Interactive help

The tool has an internal help system:

docker run docker.ubs-hainer.com/xdm3-transfer --help

xdm3-transfer <command>

Commands:
  xdm3-transfer export   export XDM objects to files
  xdm3-transfer import   import yaml file into XDM
  xdm3-transfer report   list objects in yaml file
  xdm3-transfer compare  compare objects in two yaml file

Options:
      --help         Show help                                         [boolean]
      --version      Show version number                               [boolean]
  -e, --environment  The environment to export from. Format username:password@ht
                     tp(s)://url:port without /api
  -l, --object-list  Containing URLs (one line each) to the objects to be export
                     ed, may be objects or lists. Lists need to prefixed with 'a
                     ll ' and may contain filters as well
  -o, --output       The file the resulting YAML is written to
  -i, --input        The file containing the YAML to read
  -j, --input2       The file containing the YAML to read for the compare
  -f, --filter       A jq compliant filter to be applied on the exported objects
  -m, --filter-file  A file containing multiple jq compliant filters (each in on
                     e line) to be applied on the exported objects

Export objects

The user can export one or multiple objects from an XDM environment using the export command. The export uses an input file containing the URLs of the objects to be defined. All objects need to be prefixed with /api/. The user can build the URLs by visiting the object in the user interface and copying its URL from the browsers address bar. Replace everything before and including the /res/ with /api. For example the URL in the user interface https://xdm-testing.ubs-hainer.com/gen/res/connection/mssql-connection/731104 becomes the object /api/connection/mssql-connection/731104.

The export result will be written into a local file. The file will contain the YAML export for all objects listed in the input file an all other objects referenced by the exported objects.

Example

When executing the export of xdm3-transfer with

docker run -v $(pwd):/wrk docker.ubs-hainer.com/xdm3-transfer export -l test-objects.txt -e test:default@https://xdm-testing.ubs-hainer.com -o export.yaml

and a local file ´test/test-objects.txt´

/api/connection/mssql-connection/731104

the user will get the following output:

Export objects in test/test-objects.txt from XDM environment ui-test:default@https://xdm-testing.ubs-hainer.com, writing to export.yaml
Exporting object /api/connection/mssql-connection/731104 from https://xdm-testing.ubs-hainer.com
Report on export:
***********************************

Connection - 'Import/Export test object'
-- Permission
Credential - 'testuser'
-- Permission

***********************************

Write export objects to export.yaml

and the export.yaml file containing:

version: 10
data:
  - type: Connection
    variation: mssql-connection
    replacePermissions: true
    replaceObject: true
    attributes:
      bcpTrancheSize: "500000"
      credential: testuser
      displayName: Import/Export Testobjekt
      extractPackageSize: "500"
      jdbcBatchActivated: "true"
      jdbcBatchSize: "500"
      modificationSets: []
      storageLocation: null
      threadCount: "4"
      url: jdbc:sqlserver://test.ubs-hainer.com:1433;databaseName=TMF
      useLocalBcp: "false"
    children:
      - type: Permission
        variation: null
        replacePermissions: true
        replaceObject: true
        attributes:
          administration: true
          applySql: false
          browse: true
          create: true
          delete: true
          execute: true
          grantee: USER
          granteeType: ROLE
          read: true
          sourceUsage: true
          targetUsage: true
          write: true
  - type: Credential
    variation: null
    replacePermissions: true
    replaceObject: true
    attributes:
      certificateBasedAuthentication: false
      description: null
      displayName: testuser
      keyFormat: null
      password: "{aes}cdd005d71eac4fef7cde319ccdb56f1c762e31258068dc491c7a8f171b85e719"
      privateKey: null
      user: testuser
    children:
      - type: Permission
        variation: null
        replacePermissions: true
        replaceObject: true
        attributes:
          administration: true
          applySql: false
          browse: true
          create: true
          delete: true
          execute: true
          grantee: USER
          granteeType: ROLE
          read: true
          sourceUsage: true
          targetUsage: true
          write: true

Exporting lists

It is also possible to export all objects of a list. For this, the URL in the object list file need to be prefixed with all. An example for the object list file:

all /api/connection/mssql-connection/
all /api/connection/oracle-connection/

Using this file as input for the export will generate a YAML file containing the export for all Oracle connections, all MS SQL connection and all credentials and storage locations referenced by this connection.

In the input list file entries for lists and single objects can be mixed.

It is also possible to use object filters, e.g.

all /api/connection/mssql-connection/?search=displayName:*Export*

, to filter only MS SQL connections containing Export in the displayName property.

Import

The user can import a YAML file into an XDM environment using the import command.

Example

docker run -v $(pwd):/wrk docker.ubs-hainer.com/xdm3-transfer import -e test:default@https://xdm-testing.ubs-hainer.com -i export.yaml

Output:

Import objects in export.yaml into XDM environment ui-test:default@https://xdm-testing.ubs-hainer.com
Report on export:
***********************************

Connection - 'Import/Export Testobjekt'
-- Permission
Credential - 'testuser'
-- Permission

***********************************

Run import on https://xdm-testing.ubs-hainer.com with 2 base objects ...

Imported 4 object(s) and children successful

Imported Objects:
* testuser - REPLACED - https://xdm-testing.ubs-hainer.com/api/credential/731100
* Import/Export test object - REPLACED - https://xdm-testing.ubs-hainer.com/api/connection/mssql-connection/731104
* n/a - SKIPPED - https://xdm-testing.ubs-hainer.com/api/credential/731100/permission/666
* n/a - SKIPPED - https://xdm-testing.ubs-hainer.com/api/connection/mssql-connection/731104/permission/666

Report on export file

The user can generate a report on the content of a YAML file using the report command.

Example

docker run -v $(pwd):/wrk docker.ubs-hainer.com/xdm3-transfer report -i export.yaml

Output:

Report on objects in export.yaml
***********************************

Connection - 'Import/Export Test object'
-- Permission
Credential - 'testuser'
-- Permission

***********************************

Compare two export files

The user can generate a report on the difference of two YAML files using the compare command.

Example

docker run -v $(pwd):/wrk docker.ubs-hainer.com/xdm3-transfer compare -i export.yaml -j export2.yaml

Output:

Diff on objects in export.yaml and export2.yaml
***      First object tree      ***
Report on export:
***********************************

Connection - 'Import/Export Test object'
-- Permission
Credential - 'testuser'
-- Permission

***********************************


***     Second object tree     ***
Report on export:
***********************************

Connection - 'Import/Export Test object'
-- Permission
Credential - 'testuser'
-- Permission

***********************************


 [
   {
     attributes: {
-      credential: "testuser"
+      credential: "testuser2"
     }
   }
   {
     attributes: {
-      password: "{aes}cdd005d71eac4fef7cde319ccdb56f1c762e31258068dc491c7a8f171b85e719"
+      password: "{aes}490039a3a3fd5a3e9540481a38117e5d70e08e1522a08337a9f7f694abf19a82"
     }
   }
 ]

Filtering and Changing before operation

For every command the contents of the YAML export can be filtered and changed using a filter. A single filter can be specified in the command line, multiple filter can be defined in a file. The filter commands a evaluated by jq, a common tool to modify JSON content.

Example

The first example uses a single filter defined in the command line:

docker run -v $(pwd):/wrk docker.ubs-hainer.com/xdm3-transfer export -l test-objects.txt -e ui-test:default@https://xdm-testing.ubs-hainer.com -f 'map( select (.type=="Connection"))' -o export.yaml

For this export, only the objects, where type equals Connection are written to the output file.

Multiple filters can be defined (and persisted) in a filter file:

map( select (.type=="Credential"))
map( .attributes.displayName="Exported Credential")

This filter file can be used in the execution like this:

docker run -v $(pwd):/wrk docker.ubs-hainer.com/xdm3-transfer export -l test-objects.txt -e ui-test:default@https://xdm-testing.ubs-hainer.com -m filter-file.txt -o export.yaml

The filters will first only select objects of type Credential and then change the attribute displayName to Exported Credential.

The user can find more examples an documentation for the filter in the jq manual.

Trouble shouting

If the user encounters problems connecting to a secured connection because the trust certificate is missing in the container for the transfer tool, they might either build a custom image around the xdm3-transfer containing the certificate, or they can ignore the certificate error by setting the environment variable NODE_TLS_REJECT_UNAUTHORIZED to 0.

A complete export command would look like this:

docker run -v $(pwd):/wrk -e NODE_TLS_REJECT_UNAUTHORIZED=0 docker.ubs-hainer.com/xdm3-transfer export -l test-objects.txt -e test:default@https://xdm-testing.ubs-hainer.com -o export.yaml