Hop Import

Hop Import is a command line tool to import and convert third party metadata to an Apache Hop format.

See also these plugins:

Usage

Usage help can be obtained by running hop-import with the -h option, even when other options are specified:

  1. Usage: <main class> [-efhlp] [-c=<targetConfigFilename>] [-i=<inputFolderName>]
  2.                     [-j=<jdbcPropertiesFilename>]
  3.                     [-k=<kettlePropertiesFilename>] [-o=<outputFolderName>]
  4.                     [-s=<sharedXmlFilename>] [-t=<type>]
  5.   -c, --target-config-file=<targetConfigFilename>
  6.                         The target config file to write variable to
  7.   -e, --skip-existing   Skip existing files in the target folders
  8.   -f, --skip-folders    Skip import of sub-folders
  9.   -h, --help            Displays this help message and quits.
  10.   -i, --input=<inputFolderName>
  11.                         The input folder to read from
  12.   -j, --jdbc-properties=<jdbcPropertiesFilename>
  13.                         The jdbc.properties file to read from
  14.   -k, --kettle-properties=<kettlePropertiesFilename>
  15.                         The kettle.properties file to read from
  16.   -l, --list-plugins    List the available import plugins
  17.   -o, --output=<outputFolderName>
  18.                         The output folder to write to
  19.   -p, --skip-hidden     Skip import of hidden files and folders
  20.   -s, --shared-xml=<sharedXmlFilename>
  21.                         The shared.xml file to read from
  22.   -t, --type=<type>     The type of import plugin to use (e.g. kettle)

Examples

Import a set of Kettle files and folders into a project stored on Amazon AWS S3:

  1. sh hop-import.sh \
  2.   --type kettle \
  3.   --input /projects/kettle/inventory \
  4.   --output s3:///apache-hop/s3project \
  5.   --target-config-file imported-env-conf.json \
  6.   --kettle-properties /home/etl/.kettle/kettle.properties \
  7.   --shared-xml /home/matt/.kettle/shared.xml

The logging will show what is going on and will print a report at the end:

  1. 2021/06/22 16:39:29 - HopImport - Import is skipping existing target files
  2. 2021/06/22 16:39:29 - HopImport - Import is skipping hidden files and folders
  3. 2021/06/22 16:39:29 - HopImport - Import is not skipping sub-folders
  4. 2021/06/22 16:39:29 - HopImport - Finding files to import
  5. 2021/06/22 16:39:29 - HopImport - We found 84 kettle files.
  6. 2021/06/22 16:39:29 - HopImport - Importing files
  7. 2021/06/22 16:39:29 - HopImport -   - Saving file s3:///apache-hop/s3project/load-nodes/sample-model.json
  8. 2021/06/22 16:39:29 - HopImport -   - Saving file s3:///apache-hop/s3project/scaleable-file-processing/input/file02.csv
  9. 2021/06/22 16:39:29 - HopImport -   - Saving file s3:///apache-hop/s3project/graph-output/READ.me
  10. 2021/06/22 16:39:29 - HopImport -   - Saving file s3:///apache-hop/s3project/scaleable-file-processing/PDI/Check slave server.hpl
  11. 2021/06/22 16:39:29 - HopImport -   - Saving file s3:///apache-hop/s3project/remove-everything/remove all in database.hwf
  12. ...
  13. 2021/06/22 16:39:37 - HopImport -   - Saving file s3:///apache-hop/s3project/scaleable-file-processing/input/file01.csv
  14. 2021/06/22 16:39:37 - HopImport - Importing connections
  15. 2021/06/22 16:39:41 - HopImport - Importing variables
  16. Creating new default Hop configuration file: s3:///apache-hop/s3project/imported-env-conf.json
  18. 2021/06/22 16:39:43 - HopImport - Imported:
  19. 2021/06/22 16:39:43 - HopImport - 10 jobs
  20. 2021/06/22 16:39:43 - HopImport - 31 transformations
  21. 2021/06/22 16:39:43 - HopImport - 43 other files
  22. 2021/06/22 16:39:43 - HopImport - 86 variables were imported into environment config file s3:///apache-hop/s3project/imported-env-conf.json
  23. 2021/06/22 16:39:43 - HopImport - You can use this as a configuration file in an environment.
  24. 2021/06/22 16:39:43 - HopImport - 3 database connections where saved in metadata folder s3:///apache-hop/s3project/metadata
  25. 2021/06/22 16:39:43 - HopImport -
  26. 2021/06/22 16:39:43 - HopImport - Connections with the same name and different configurations have only been saved once.
  27. 2021/06/22 16:39:43 - HopImport - Check the following file for a list of connections that might need extra attention: s3:///apache-hop/s3project/connections.csv