Reflection

The Hop Reflection plugin adds easy ways for you to get more information about what’s going on inside your pipelines or workflows. It offers 3 main ways of reflecting on what’s going on during execution:

  • Log pipeline and transform execution information

  • Log workflow and action execution information

  • Listen to what the output is of a particular transform

Pipeline Log

You can have a pipeline and its transforms log what it’s doing by creating a new Pipeline Log metadata object. For each metadata object of this type you create Hop will execute a pipeline of your choice. It will pass runtime information of all your pipelines to it. These are the options:

Options

OptionDescription

Name

The name of the metadata object

Enabled

Use this to enable or disable pipeline logging for this object

Logging parent pipelines only?

If you enable this only the parent pipelines, executed by Hop Run, GUI, Server or API will be logged. When disabled every pipeline will be logged with the exclusion of logging and data probing pipelines obviously.

Pipeline executed to capture logging

The filename of the pipeline which will get executed every time a pipeline is executed. You can hit the new button and it will create a sample pipeline for you containing the Pipeline Logging transform which will give you all the possible logging information about your pipelines.

Execute at the start of a pipeline?

Execute right before a pipeline starts. The loggingPhase field of the Pipeline Logging transform will contain start at this time. Please note that there will not be any transform execution information yet during this phase.

Execute at the end of a pipeline?

Execute after the pipeline finished execution. The loggingPhase field of the Pipeline Logging transform will contain end at this time.

Execute periodically during execution?

Enable this flag if you want to be kept apprised of long-running pipelines. The logging pipeline specified above will be called at regular intervals. The loggingPhase field of the Pipeline Logging transform will contain interval at this time.

Interval in seconds

The interval of the period execution of the logging pipeline

Transform

The Pipeline Logging transform only has one option at this time:

  • Also log transform details : besides information about a running pipeline it will give you a series of fields detailed below per running or finished transform copy.
FieldTypeLevelDescription

loggingDate

Date

Pipeline

Logging date

loggingPhase

String

Pipeline

Logging phase (see above)

pipelineName

String

Pipeline

Name of the pipeline

pipelineFilename

String

Pipeline

Filename of the pipeline

pipelineStart

Date

Pipeline

Start date of the pipeline

pipelineEnd

Date

Pipeline

End date of the pipeline

pipelineLogChannelId

String

Pipeline

Pipeline log channel ID

parentLogChannelId

String

Pipeline

Parent log channel ID

pipelineLogging

String

Pipeline

Logging text of the pipeline

pipelineErrorCount

Integer

Pipeline

Number of errors

pipelineStatusDescription

String

Pipeline

Pipeline status description

transformName

String

Transform

Name of the transform

transformCopyNr

Integer

Transform

Copy number of the transform

transformStatusDescription

String

Transform

Transform status description

transformLogChannelId

String

Transform

Transform log channel ID

transformLoggingText

String

Transform

Transform logging text

transformLinesRead

Integer

Transform

Number of lines read

transformLinesWritten

Integer

Transform

Number of lines written

transformLinesInput

Integer

Transform

Number of lines input

transformLinesOutput

Integer

Transform

Number of lines output

transformLinesUpdated

Integer

Transform

Number of lines updated

transformLinesRejected

Integer

Transform

Number of lines rejected by error handling

transformErrors

Integer

Transform

Number of errors

transformStart

Date

Transform

Execution start

transformEnd

Date

Transform

Execution end

transformDuration

Integer

Transform

Execution duration in ms

Workflow Log

You can have a workflow and its actions log what it’s doing by creating a new Workflow Log metadata object. For each metadata object of this type you create Hop will execute a pipeline of your choice. It will pass runtime information of all your workflows to it. These are the options:

Options

OptionDescription

Name

The name of the metadata object

Enabled

Use this to enable or disable workflow logging for this object

Logging parent workflows only?

If you enable this only the parent pipelines, executed by Hop Run, GUI, Server or API will be logged. When disabled every workflow execution will be logged.

Pipeline executed to capture logging

The filename of the pipeline which will get executed every time a workflow is executed. You can hit the new button and it will create a sample pipeline for you containing the Workflow Logging transform which will give you all the possible logging information about your workflows.

Execute at the start of a workflow?

Execute right before a workflow starts. The loggingPhase field of the Workflow Logging transform will contain start at this time.

Execute at the end of a workflow?

Execute after the workflow finished execution. The loggingPhase field of the Workflow Logging transform will contain end at this time.

Execute periodically during execution?

Enable this flag if you want to be kept apprised of long-running workflows. The logging pipeline specified above will be called at regular intervals. The loggingPhase field of the Workflow Logging transform will contain interval at this time.

Interval in seconds

The interval of the period execution of the logging pipeline

Transform

The Workflow Logging transform only has one option at this time:

  • Also log action details : besides information about a running workflow it will give you a series of fields detailed below per running or finished action.
FieldTypeLevelDescription

loggingDate

Date

Workflow

Logging date

loggingPhase

String

Workflow

Logging phase (see above)

workflowName

String

Workflow

Workflow name

workflowFilename

String

Workflow

Workflow filename

workflowStart

Date

Workflow

Execution start

workflowEnd

Date

Workflow

Execution end

workflowLogChannelId

String

Workflow

Workflow log channel ID

workflowParentLogChannelId

String

Workflow

Parent log channel ID

workflowLogging

String

Workflow

Logging text of the workflow

workflowErrorCount

Integer

Workflow

Number of errors

workflowStatusDescription

String

Workflow

Workflow status description

actionName

String

Action

Action name

actionNr

Integer

Action

Action number

actionResult

Boolean

Action

Result (true/false)

actionLogChannelId

String

Action

Log channel ID of the action

actionLoggingText

String

Action

Logging text of the action

actionErrors

Integer

Action

Number of errors

actionLogDate

Date

Action

Action logging date

actionDuration

Integer

Action

Action duration

actionExitStatus

Integer

Action

Action exit status (of shell scripts)

actionNrFilesRetrieved

Integer

Action

Number of files retrieved (from remote systems)

actionFilename

String

Action

The filename of the action (if there is any referenced)

actionComment

String

Action

Action comment

actionReason

String

Action

Action reason

Pipeline Probe

You can have the output of one or more pipeline transforms streamed to another pipeline in a normalized fashion. You can do this by creating a Pipeline Probe metadata object. These are its options:

Options

OptionDescription

Name

The name of the pipeline probe

Enabled

When enabled data from all the specified pipeline transforms will be streamed to the specified pipeline

Pipeline executed to capture data

The filename of the pipeline which will get executed every time a specified source pipeline is execute containing the specified source transform.

Capture output of the following transforms

You can specify a set of source pipeline and source transform combinations.

Please note that to add capture sources you can use a transform action called Add data probe in the Preview section of the context dialog.

Transform

Field nameTypeDescription

sourcePipelineName

String

THe name of the pipeline where this data originates

sourceTransformLogChannelId

String

The log channel ID of the source transform

sourceTransformName

String

The name of the source transform

sourceTransformCopy

Integer

The copy number of the source transform

rowNr

Integer

The source row number. You have one or more fields per source row.

fieldName

String

The field name

fieldType

String

The field type

fieldFormat

String

The field format

fieldLength

Integer

The field length

fieldPrecision

Integer

The field precision

value

String

The value in String format