Hop Server Web Services Overview

Hop Server has a rich set of web services that can be used to query and manage the server’s status, or to manage pipeline and workflow execution details.

when specified, the id for a workflow or pipeline represents one execution of a workflow or pipeline on the server.

addExport

name

addExport

description

Upload a resources export file. Add a zipped pipeline or workflow to the body payload as a binary file.

endPoint

GET hop/addExport

parameters

  • type: pipeline or workflow

  • load: -

example request

http://localhost:8081/hop/addExport/?type=workflow with zipped workflow as payload

result

A zip file with the export is created on the server’s file system.

  1. <?xml version="1.0" encoding="UTF-8"?>
  2. <webresult>
  3. <result>OK</result>
  4. <message>file:///tmp/export_70eb8ef1-9721-4cf5-afa3-940cd0f771d9.zip</message>
  5. <id/>
  6. </webresult>

addPipeline

name

addPipeline

description

Add a pipeline for execution

endPoint

GET hop/addPipeline

parameters

  • xml Request body should contain xml containing pipeline_configuration (pipeline and pipeline_execution_configuration wrapped in pipeline_configuration tag).

example request

http://localhost:8081/hop/addPipeline/xml=Y with XML payload

result

-

addWorkflow

name

addWorkflow

description

Add a workflow for execution

endPoint

GET hop/addWorkflow

parameters

  • xml Request body should contain xml containing workflow_configuration (pipeline and workflow_execution_configuration wrapped in pipeline_configuration tag).

example request

http://localhost:8081/hop/addWorkflow/xml=Y with XML payload

result

-

getPipelineImage

name

getPipelineImage

description

Generate a SVG image of a pipeline

endPoint

GET hop/pipelineImage

parameters

  • name: name of the pipeline to generate the image for

  • id: id of the pipeline to generate the image for

example request

GET http://localhost:8081/hop/pipelineImage/?name=remote-pipeline&id=c1451bfb-b867-4c76-b123-c29d2b05da17

result

an SVG image of the pipeline graph

getPipelineStatus

name

getPipelineStatus

description

Get the status of a pipeline

endPoint

GET hop/pipelineStatus

parameters

  • xml: use XML (default: Y)

  • json: use JSON (default: Y)

  • name: name of the pipeline to get the status for

  • id: id of the pipeline to get the status for

example request

GET http://localhost:8081/hop/pipelineStatus/?name=<NAME>>&id=<ID>

result

an HTML response with the execution status, transform details and canvas preview for this pipeline

Status

name

status

description

Get the status of the server

parameters

none

example request

GET http://localhost:8081/hop/status

result

an HTML page with an overview of the pipelines and workflows on the server, their execution details and the server’s configuration details.

getWorkflowImage

name

getWorkflowImage

description

Generate an SVG image of a workflow

endPoint

GET hop/workflowImage

parameters

  • name: name of the workflow to generate the image for

  • id: id of the workflow to generate the image for

example request

GET http://localhost:8081/hop/workflowImage/?name=<NAME>>&id=<ID>

result

an SVG image of the workflow graph

getWorkflowStatus

name

getWorkflowStatus

description

Get the status of a workflow

endPoint

GET hop/workflowStatus

parameters

  • xml: use XML (default: Y)

  • json: use JSON (default: Y)

  • name: name of the workflow to get the status for

  • id: id of the workflow to get the status for

example request

GET http://localhost:8081/hop/workflowStatus/?name=<NAME>&id=<ID>

result

an HTML response with the execution status, action details and canvas preview for this workflow

pausePipeline

name

pausePipeline

description

Pause or continue a pipeline

endPoint

GET /hop/pausePipeline

parameters

  • name: name of the pipeline to pause or restart

  • id: id of the pipeline to pause or restart

example request

GET http://localhost:8081/hop/pausePipeline/?name=<NAME>&id=<ID>

result

HTML page with the request status, e.g.

  1. <HTML>
  2. <HEAD>
  3. <TITLE>Pause pipeline</TITLE>
  4. <META http-equiv="Refresh" content="2;url=/hop/pipelineStatus?name=<NAME>&id=<ID>">
  5. <META http-equiv="Content-Type" content="text/html; charset=UTF-8">
  6. </HEAD>
  7. <BODY>
  8. <H1>Pipeline [tmp] : pause requested.</H1>
  9. <a href="/hop/pipelineStatus?name=<NAME>&id=<ID>">Back to the pipeline status page</a>
  10. <p>
  11. <p>
  12. </BODY>
  13. </HTML>

Prepare Execution

name

prepareExec

description

Prepare the execution of a pipeline

endPoint

GET /hop/prepareExec

parameters

  • xml: use xml, default Y

  • name: the name of the pipeline to prepare execution for

  • id: the id of the pipeline to prepare execution for

example request

GET http://localhost:8081/hop/prepareExec/?xml=Y&name=<NAME>&id=<ID>

result

Example result:

  1. <?xml version="1.0" encoding="UTF-8"?>
  2. <webresult>
  3. <result>OK</result>
  4. <message/>
  5. <id/>
  6. </webresult>

Register Pipeline

name

registerPipeline

description

Register a pipeline for execution

endPoint

GET hop/registerPipeline

parameters

  • xml Request body should contain xml containing pipeline_configuration (pipeline and pipeline_execution_configuration wrapped in pipeline_configuration tag).

example request

http://localhost:8081/hop/registerPipeline/xml=Y

with XML payload (example):

  1. <pipeline_configuration>
  2. <pipeline>
  3. <info>
  4. <name>generate_rows</name>
  5. <name_sync_with_filename>Y</name_sync_with_filename>
  6. <description/>
  7. <extended_description/>
  8. <pipeline_version/>
  9. <pipeline_type>Normal</pipeline_type>
  10. <parameters>
  11. </parameters>
  12. <capture_transform_performance>N</capture_transform_performance>
  13. <transform_performance_capturing_delay>1000</transform_performance_capturing_delay>
  14. <transform_performance_capturing_size_limit>100</transform_performance_capturing_size_limit>
  15. <created_user>-</created_user>
  16. <created_date>2022/02/03 13:47:49.645</created_date>
  17. <modified_user>-</modified_user>
  18. <modified_date>2022/02/03 13:47:49.645</modified_date>
  19. <key_for_session_key>H4sIAAAAAAAA/wMAAAAAAAAAAAA=</key_for_session_key>
  20. <is_key_private>N</is_key_private>
  21. </info>
  22. <notepads>
  23. </notepads>
  24. <order>
  25. <hop>
  26. <from>Generate rows</from>
  27. <to>Delay row</to>
  28. <enabled>Y</enabled>
  29. </hop>
  30. <hop>
  31. <from>Delay row</from>
  32. <to>result</to>
  33. <enabled>Y</enabled>
  34. </hop>
  35. </order>
  36. <transform>
  37. <name>Delay row</name>
  38. <type>Delay</type>
  39. <description/>
  40. <distribute>Y</distribute>
  41. <custom_distribution/>
  42. <copies>1</copies>
  43. <partitioning>
  44. <method>none</method>
  45. <schema_name/>
  46. </partitioning>
  47. <scaletime>seconds</scaletime>
  48. <timeout>1</timeout>
  49. <attributes/>
  50. <GUI>
  51. <xloc>416</xloc>
  52. <yloc>96</yloc>
  53. </GUI>
  54. </transform>
  55. <transform>
  56. <name>Generate rows</name>
  57. <type>RowGenerator</type>
  58. <description/>
  59. <distribute>Y</distribute>
  60. <custom_distribution/>
  61. <copies>1</copies>
  62. <partitioning>
  63. <method>none</method>
  64. <schema_name/>
  65. </partitioning>
  66. <fields>
  67. <field>
  68. <length>-1</length>
  69. <name>value</name>
  70. <precision>-1</precision>
  71. <set_empty_string>N</set_empty_string>
  72. <type>String</type>
  73. <nullif>test</nullif>
  74. </field>
  75. </fields>
  76. <interval_in_ms>5000</interval_in_ms>
  77. <last_time_field>FiveSecondsAgo</last_time_field>
  78. <never_ending>N</never_ending>
  79. <limit>10000</limit>
  80. <row_time_field>now</row_time_field>
  81. <attributes/>
  82. <GUI>
  83. <xloc>160</xloc>
  84. <yloc>96</yloc>
  85. </GUI>
  86. </transform>
  87. <transform>
  88. <name>result</name>
  89. <type>Dummy</type>
  90. <description/>
  91. <distribute>Y</distribute>
  92. <custom_distribution/>
  93. <copies>1</copies>
  94. <partitioning>
  95. <method>none</method>
  96. <schema_name/>
  97. </partitioning>
  98. <attributes/>
  99. <GUI>
  100. <xloc>720</xloc>
  101. <yloc>96</yloc>
  102. </GUI>
  103. </transform>
  104. <transform_error_handling>
  105. </transform_error_handling>
  106. <attributes/>
  107. </pipeline>
  108. <pipeline_execution_configuration>
  109. <pass_export>N</pass_export>
  110. <parameters>
  111. </parameters>
  112. <variables>
  113. <variable><name>HOP_AUDIT_FOLDER</name><value>/Users/hans/config/audit</value></variable>
  114. <variable><name>HOP_AUTO_CREATE_CONFIG</name><value>Y</value></variable>
  115. <variable><name>HOP_CONFIG_FOLDER</name><value>/Users/hans/config</value></variable>
  116. <variable><name>HOP_DATASETS_FOLDER</name><value>/Users/hans/test/datasets</value></variable>
  117. <variable><name>HOP_ENVIRONMENT_NAME</name><value>test1</value></variable>
  118. <variable><name>HOP_METADATA_FOLDER</name><value>/Users/hans/test/metadata</value></variable>
  119. <variable><name>HOP_PIPELINE_PAN_JVM_EXIT_CODE</name><value/></variable>
  120. <variable><name>HOP_PROJECTS</name><value>/Users/hans/tmp/</value></variable>
  121. <variable><name>HOP_PROJECT_NAME</name><value>test</value></variable>
  122. <variable><name>HOP_UNIT_TESTS_FOLDER</name><value>/Users/hans/test</value></variable>
  123. <variable><name>NEO4J_CONNECTION</name><value>neo4j</value></variable>
  124. <variable><name>PROJECT_HOME</name><value>/Users/hans/test</value></variable>
  125. <variable><name>TEST</name><value>TEST</value></variable>
  126. <variable><name>ftp.nonProxyHosts</name><value>local|*.local|169.254/16|*.169.254/16</value></variable>
  127. <variable><name>http.nonProxyHosts</name><value>local|*.local|169.254/16|*.169.254/16</value></variable>
  128. <variable><name>jdk.debug</name><value>release</value></variable>
  129. <variable><name>native.encoding</name><value>UTF-8</value></variable>
  130. <variable><name>p1</name><value>a</value></variable>
  131. <variable><name>socksNonProxyHosts</name><value>local|*.local|169.254/16|*.169.254/16</value></variable>
  132. </variables>
  133. <log_level>Basic</log_level>
  134. <log_file>N</log_file>
  135. <log_filename/>
  136. <log_file_append>N</log_file_append>
  137. <create_parent_folder>N</create_parent_folder>
  138. <clear_log>Y</clear_log>
  139. <show_subcomponents>Y</show_subcomponents>
  140. <run_configuration>local</run_configuration>
  141. </pipeline_execution_configuration>
  142. <metastore_json>H4sIAAAAAAAA/+1XbVPbOBD+Kxl/upshzQsUAt/SxBzchSRNTDudpuNRZNlRkSVXkgk5hv9+K0t+
  143. CSl3cDP37coH8Gq1evTs7rPqo6eIvCfSu/j66G3JephlU5QS78Lzjjyl2EjwmCbeBc8ZKww3IoLV
  144. GDFFjrxMiofdXEht/TdCaW539/pn77rw0wNzhpTaChmB2edY7jJNolZ/Tc4HKMaDU4TO4j45wese
  145. Qv3T9zgyJjyAjVzwuTnhCuIqe4QArJJGxH+gSlOewHpGpKZENUE50w1Smkh7H3uBzGId9AYGmMPK
  146. ydZzd7mqbwCmHLhxX5jlJpb39A08aUYY5aQNW9aw+BVsOae6rYnS9lNG61QVnLq/Hr3ldPb5cjL8
  147. wzcfEdJojRSpuc5YnlB+bUiqPY88yiPyEKA1IypDuLrHFkmyEYDPbjbhDp0QxkSpYJeBpbuXnZeS
  148. 4lUMVYgcwiUX25ihO2LKoqgY3qAVaS3pOtfE3vR2Pp8tgmUYXN/4y2B4Mw/Hw2AYBl/mcHnvC4T4
  149. eDsL/HA4mYSX1/5kvATzFMzVzg+z2cQfTg/2Xc4WIz+8HvvT4Bp2LuCMWTiZffYXo+HSd1HmC3/p
  150. Lz75ofs9Dj/PFuPQeZgwy4+TcDSbTv1RYO/607gA5nncS3+xgHjL0ZV/Mwynwxuz5j0deSniOWK3
  151. kh1Ujvf0VFVaQjiRFPdhR6My5rNl8Btg/Tj5SWlkkLVEArF7JdLY8mKNvLUomMCIme83tOzZOSa9
  152. bnR6fox6+Lwun/cnx/3nJTS3N7GY/6+ht9TQBnHVrKMsKYQIMnQXM7FtM5FY3cGQN8QjidpYcE6w
  153. poLbFU7Eyfd2IlG2aaeg4cyaQfLbJhfUVAjIlZZwVCxk6pI2ztN01/olEi0u9AYE91dAFlNGHJJV
  154. 5xa2q1XHQFx1jAKuOqbyljbou03Gih2ERS7kPWK5kRFmFXypkc5r9XZxFUozRkKAV6KDywmuCde2
  155. dj2UZYxiZG646nxXcM8jD1BBvUPVapmTPamWOTeUwCzLJXKsPII/lCdZ5Hy0v/TofSAoHVMJDM5d
  156. CL/wNWuapNlEYOdbsHGx6pifeyRXnViwqKDkvgu0fE9P77Ik6nbPTjmO7rJ4IJPjaNuFfwkAD1Yd
  157. 0w1awnlAxpXIgjID6kMex0Qu6Z9OsnmerokMRRyaxMMR1hwj/TuSdc0MQWVMD0KsF0NfwjjbXAOd
  158. ErJht1aZnxdNO2JQSkQ1R4EKBCQrcWgesgNHU6F7JH9CkhYK5GrQ5rYYk6BPRGFJM8diIYkv52NB
  159. UqEL9smD0ZgQJFHkEpvQJtkwcktDqJFMiA5tHizYetH+3lu0WhRmgrGQ7nECVRNi9wKy+li8crKw
  160. fDS5t0MzArQW2v0jG49VK/cOqThyPQJFXvS5c5WWgzcyNylgwx8xIdEa4btQ2Yp6b2rQgLet5qy9
  161. pk1DqwElYZJTWJqgIm9SbBXQW/u7HTEJ0+bjUG3ENiwPraxaZAKSYKaEsyRIb4C8lID+41oIinpE
  162. hYIZ/IX1lQVWZmqfJyBHtXaQ/lapCqpVeLJda0v1pgUwWkqDfCIZ2ZUWNE3l3bIc24wUarqnsY/V
  163. S3pvkq4LuqR7JJ+dnJ2491pjxhfRvMYAcNdBuRYp3BOX1yxnZa6god1gLk53nEmRG0mtn8KvHuPr
  164. qH8cRYNujAupfXZAfb5XwHwI6uQsIG+7gJbvypqTCb2H545Sow3Bd8YBwD33ecE8xD9yquiBB3S6
  165. 0kPGRuZ9Hxv1J+oA26iBICa6QubI2fevKH7O4VzAdNlZp7Vgukzg6eDsBSBVBCgCBaeflMrEn2e5
  166. AXEuBKsVvjHyS+8yWBO2qcBU8ERE64M5nyFAVH8iteO4vT/lm3OxejlUb4m3TspKXw4V4KUWbrZp
  167. /2/6tMT0uj4tvas+fZ0s/jvY/x3qb/Z91Y5ITDltJAAzoXJZvgWa/3U9GB/Fc2tcBXByoggUB9JC
  168. ljWkYK7zO7toJAlk3Xw8/QVlC+SCFRAAAA==
  169. </metastore_json>
  170. </pipeline_configuration>

The metastore_json is base64 encoded GZip content

result

-

Register Workflow

name

registerWorkflow

description

Register a workflow on the server

endPoint

GET /hop/registerWorkflow

parameters

  • xml:

example request

-

result

-

Remove Pipeline

name

removePipeline

description

Remove a pipeline from the server

endPoint

GET /hop/removePipeline

parameters

  • name: the name of the pipeline to remove

  • id: the id of the pipeline to remove

example request

GET http://localhost:8081/hop/removePipeline/?name=<NAME>&id=<ID>

result

Example result:

  1. <HTML>
  2. <HEAD>
  3. <TITLE>The pipeline was removed</TITLE>
  4. <META http-equiv="Content-Type" content="text/html; charset=UTF-8">
  5. </HEAD>
  6. <BODY>
  7. <H3>The pipeline with name [<NAME>] and hopServer object id <ID> was removed from Hop
  8. Server.</H3>
  9. <a href="/hop/status">Back to the status page</a><br>
  10. <p>
  11. </BODY>

Remove Workflow

name

removeWorkflow

description

Remove a workflow from the server

endPoint

GET /hop/removeWorkflow

parameters

  • name: the name of the workflow to remove

  • id: the id of the workflow to remove

example request

GET http://localhost:8081/hop/removeWorkflow/?name=<NAME>&id=<ID>

result

Example result:

  1. <HTML>
  2. <HEAD>
  3. <TITLE>The workflow was removed</TITLE>
  4. <META http-equiv="Content-Type" content="text/html; charset=UTF-8">
  5. </HEAD>
  6. <BODY>
  7. <H3>The workflow with name [<NAME>] and hopServer object id <ID> was removed from Hop
  8. Server.</H3>
  9. <a href="/hop/status">Back to the status page</a><br>
  10. <p>
  11. </BODY>

Sniff Transform

name

sniffTransform

description

Sniff test a pipeline transform

endPoint

GET /hop/sniffTransform

parameters

  • xml: use XML (default: Y)

  • pipeline: name of the pipeline to sniff

  • id: id of the pipeline to sniff

  • transform: name of the transform to sniff

  • type: (input/output) sniff input or output

  • lines: number of lines to wait for

  • copynr: when using multiple copies you can specify the copynr

following parameters are required

  • xml

  • id or pipeline

  • transform

example request

GET http://localhost:8081/hop/sniffTransform/?xml=Y&pipeline=<NAME>&id=<ID>&transform=<TRANSFORMNAME>&type=output&lines=100

result

Example result (empty):

  1. <row-buffer>
  2. <row-meta/>
  3. </row-buffer>

Start Pipeline Execution

name

startExec

description

Start the execution of a pipeline

endPoint

GET /hop/startExec

parameters

  • name: the name of the pipeline to start

example request

GET http://localhost:8081/hop/startExec?name=<NAME>

result

Example result:

  1. <HTML>
  2. <HEAD>
  3. <TITLE>Start of pipeline</TITLE>
  4. <META http-equiv="Refresh" content="2;url=/hop/pipelineStatus?name=<NAME>">
  5. <META http-equiv="Content-Type" content="text/html; charset=UTF-8">
  6. </HEAD>
  7. <BODY>
  8. <H1>Pipeline [<NAME>] was started.</H1>
  9. <a href="/hop/pipelineStatus?name=<NAME>&id=8bea27db-de97-4bd0-a210-d9bba3aacac2">Back to the status page</a>
  10. <p>
  11. <p>
  12. </BODY>
  13. </HTML>

Start Pipeline

name

startPipeline

description

Prepare and start the execution of a pipeline

endPoint

GET /hop/startPipeline

parameters

  • name: the name of the pipeline to start

example request

GET http://localhost:8081/hop/startPipeline?name=<NAME>

result

Example result:

  1. <HTML>
  2. <HEAD>
  3. <TITLE>Start of pipeline</TITLE>
  4. <META http-equiv="Refresh" content="2;url=/hop/pipelineStatus?name=<NAME>">
  5. <META http-equiv="Content-Type" content="text/html; charset=UTF-8">
  6. </HEAD>
  7. <BODY>
  8. <H1>Pipeline [<NAME>] was started.</H1>
  9. <a href="/hop/pipelineStatus?name=<NAME>&id=8bea27db-de97-4bd0-a210-d9bba3aacac2">Back to the status page</a>
  10. <p>
  11. <p>
  12. </BODY>
  13. </HTML>

Start Workflow

name

startWorkflow

description

Prepare and start the execution of a workflow

endPoint

GET /hop/startPipeline

parameters

  • name: the name of the workflow to start

example request

GET http://localhost:8081/hop/startWorkflow?name=<NAME>

result

Example result:

  1. <HTML>
  2. <HEAD>
  3. <TITLE>Start of workflow</TITLE>
  4. <META http-equiv="Refresh" content="2;url=/hop/startWorkflow?name=<NAME>">
  5. <META http-equiv="Content-Type" content="text/html; charset=UTF-8">
  6. </HEAD>
  7. <BODY>
  8. <H1>Workflow [<NAME>] was started.</H1>
  9. <a href="/hop/workflowStatus?name=<NAME>&id=8bea27db-de97-4bd0-a210-d9bba3aacac2">Back to the status page</a>
  10. <p>
  11. <p>
  12. </BODY>
  13. </HTML>

Stop Pipeline

name

stopPipeline

description

Stop a pipeline

endPoint

GET /hop/stopPipeline

parameters

  • name: the name of the pipeline to stop

  • id: the id of the pipeline to stop

example request

GET http://localhost:8081/hop/stopPipeline?name=<NAME>&id=<ID>

result

Example Result:

  1. <HTML>
  2. <HEAD>
  3. <TITLE>Stop pipeline</TITLE>
  4. <META http-equiv="Refresh" content="2;url=/hop/pipelineStatus?name=<NAME>>">
  5. <META http-equiv="Content-Type" content="text/html; charset=UTF-8">
  6. </HEAD>
  7. <BODY>
  8. <H1>Pipeline [<NAME>] stop requested.</H1>
  9. <a href="/hop/pipelineStatus?name=<NAME>&id=<ID>">Back to the pipeline status page</a>
  10. <p>
  11. <p>
  12. </BODY>
  13. </HTML>

Stop Workflow

name

stopWorkflow

description

Stop a workflow

endPoint

GET /hop/stopWorkflow

parameters

  • name: the name of the workflow to stop

  • id: the id of the workflow to stop

example request

GET http://localhost:8081/hop/stopWorkflow?name=<NAME>&id=<ID>

result

Example Result:

  1. <HTML>
  2. <HEAD>
  3. <TITLE>Stop workflow</TITLE>
  4. <META http-equiv="Refresh" content="2;url=/hop/workflowStatus?name=<NAME>>">
  5. <META http-equiv="Content-Type" content="text/html; charset=UTF-8">
  6. </HEAD>
  7. <BODY>
  8. <H1>Workflow [<NAME>] stop requested.</H1>
  9. <a href="/hop/workflowStatus?name=<NAME>&id=<ID>">Back to the pipeline status page</a>
  10. <p>
  11. <p>
  12. </BODY>
  13. </HTML>