Transforms APIs

Aside from using OpenSearch Dashboards, you can also use the REST API to create, start, stop, and complete other operations relative to transform jobs.

Create a transform job

Introduced 1.0

Creates a transform job.

Sample Request

  1. PUT _plugins/_transform/<transform_id>
  3. {
  4.   "transform": {
  5.   "enabled": true,
  6.   "continuous": true,
  7.   "schedule": {
  8.     "interval": {
  9.     "period": 1,
  10.     "unit": "Minutes",
  11.     "start_time": 1602100553
  12.     }
  13.   },
  14.   "description": "Sample transform job",
  15.   "source_index": "sample_index",
  16.   "target_index": "sample_target",
  17.   "data_selection_query": {
  18.     "match_all": {}
  19.   },
  20.   "page_size": 1,
  21.   "groups": [
  22.     {
  23.     "terms": {
  24.       "source_field": "customer_gender",
  25.       "target_field": "gender"
  26.     }
  27.     },
  28.     {
  29.     "terms": {
  30.       "source_field": "day_of_week",
  31.       "target_field": "day"
  32.     }
  33.     }
  34.   ],
  35.   "aggregations": {
  36.     "quantity": {
  37.     "sum": {
  38.       "field": "total_quantity"
  39.     }
  40.     }
  41.   }
  42.   }
  43. }

Sample Response

  1. {
  2.   "_id": "sample",
  3.   "_version": 7,
  4.   "_seq_no": 13,
  5.   "_primary_term": 1,
  6.   "transform": {
  7.     "transform_id": "sample",
  8.     "schema_version": 7,
  9.     "continuous": true,
  10.     "schedule": {
  11.       "interval": {
  12.         "start_time": 1621467964243,
  13.         "period": 1,
  14.         "unit": "Minutes"
  15.       }
  16.     },
  17.     "metadata_id": null,
  18.     "updated_at": 1621467964243,
  19.     "enabled": true,
  20.     "enabled_at": 1621467964243,
  21.     "description": "Sample transform job",
  22.     "source_index": "sample_index",
  23.     "data_selection_query": {
  24.       "match_all": {
  25.         "boost": 1.0
  26.       }
  27.     },
  28.     "target_index": "sample_target",
  29.     "roles": [],
  30.     "page_size": 1,
  31.     "groups": [
  32.       {
  33.         "terms": {
  34.           "source_field": "customer_gender",
  35.           "target_field": "gender"
  36.         }
  37.       },
  38.       {
  39.         "terms": {
  40.           "source_field": "day_of_week",
  41.           "target_field": "day"
  42.         }
  43.       }
  44.     ],
  45.     "aggregations": {
  46.       "quantity": {
  47.         "sum": {
  48.           "field": "total_quantity"
  49.         }
  50.       }
  51.     }
  52.   }
  53. }

You can specify the following options in the HTTP request body:

OptionData TypeDescriptionRequired
enabledBooleanIf true, the transform job is enabled at creation.No
continuousBooleanSpecifies whether the transform job should be continuous. Continuous jobs execute every time they are scheduled according to the schedule field and run based off of newly transformed buckets, as well as any new data added to source indexes. Non-continuous jobs execute only once. Default is false.No
scheduleObjectThe schedule the transform job runs on.Yes
start_timeIntegerThe Unix epoch time of the transform job’s start time.Yes
descriptionStringDescribes the transform job.No
metadata_idStringAny metadata to be associated with the transform job.No
source_indexStringThe source index whose data to transform.Yes
target_indexStringThe target index the newly transformed data is added into. You can create a new index or update an existing one.Yes
data_selection_queryObjectThe query DSL to use to filter a subset of the source index for the transform job. See query DSL for more information.Yes
page_sizeIntegerThe number of buckets IM processes and indexes concurrently. Higher number means better performance but requires more memory. If your machine runs out of memory, IM automatically adjusts this field and retries until the operation succeeds.Yes
groupsArraySpecifies the grouping(s) to use in the transform job. Supported groups are terms, histogram, and date_histogram. For more information, see Bucket Aggregations.Yes if not using aggregations
source_fieldStringThe field(s) to transformYes
aggregationsObjectThe aggregations to use in the transform job. Supported aggregations are: sum, max, min, value_count, avg, scripted_metric, and percentiles. For more information, see Metric Aggregations.Yes if not using groups

Update a transform job

Introduced 1.0

Updates a transform job if transform_id already exists.

Sample Request

  1. PUT _plugins/_transform/<transform_id>
  3. {
  4.   "transform": {
  5.   "enabled": true,
  6.   "schedule": {
  7.     "interval": {
  8.     "period": 1,
  9.     "unit": "Minutes",
  10.     "start_time": 1602100553
  11.     }
  12.   },
  13.   "description": "Sample transform job",
  14.   "source_index": "sample_index",
  15.   "target_index": "sample_target",
  16.   "data_selection_query": {
  17.     "match_all": {}
  18.   },
  19.   "page_size": 1,
  20.   "groups": [
  21.     {
  22.     "terms": {
  23.       "source_field": "customer_gender",
  24.       "target_field": "gender"
  25.     }
  26.     },
  27.     {
  28.     "terms": {
  29.       "source_field": "day_of_week",
  30.       "target_field": "day"
  31.     }
  32.     }
  33.   ],
  34.   "aggregations": {
  35.     "quantity": {
  36.     "sum": {
  37.       "field": "total_quantity"
  38.     }
  39.     }
  40.   }
  41.   }
  42. }

Sample Response

  1. {
  2.   "_id": "sample",
  3.   "_version": 2,
  4.   "_seq_no": 14,
  5.   "_primary_term": 1,
  6.   "transform": {
  7.     "transform_id": "sample",
  8.     "schema_version": 7,
  9.     "schedule": {
  10.       "interval": {
  11.         "start_time": 1602100553,
  12.         "period": 1,
  13.         "unit": "Minutes"
  14.       }
  15.     },
  16.     "metadata_id": null,
  17.     "updated_at": 1621889843874,
  18.     "enabled": true,
  19.     "enabled_at": 1621889843874,
  20.     "description": "Sample transform job",
  21.     "source_index": "sample_index",
  22.     "data_selection_query": {
  23.       "match_all": {
  24.         "boost": 1.0
  25.       }
  26.     },
  27.     "target_index": "sample_target",
  28.     "roles": [],
  29.     "page_size": 1,
  30.     "groups": [
  31.       {
  32.         "terms": {
  33.           "source_field": "customer_gender",
  34.           "target_field": "gender"
  35.         }
  36.       },
  37.       {
  38.         "terms": {
  39.           "source_field": "day_of_week",
  40.           "target_field": "day"
  41.         }
  42.       }
  43.     ],
  44.     "aggregations": {
  45.       "quantity": {
  46.         "sum": {
  47.           "field": "total_quantity"
  48.         }
  49.       }
  50.     }
  51.   }
  52. }

The Update operation supports the following URL parameters:

ParameterDescriptionRequired
if_seq_noOnly perform the transform operation if the last operation that changed the transform job has the specified sequence number.Yes
if_primary_termOnly perform the transform operation if the last operation that changed the transform job has the specified sequence term.Yes

You can update the following fields:

OptionData TypeDescription
scheduleObjectThe schedule the transform job runs on. Contains the fields interval.start_time, interval.period, and interval.unit.
start_timeIntegerThe Unix epoch start time of the transform job.
periodIntegerHow often to execute the transform job.
unitStringThe unit of time associated with the execution period. Available options are Minutes, Hours, and Days.
descriptionIntegerDescribes the transform job.
page_sizeIntegerThe number of buckets IM processes and indexes concurrently. Higher number means better performance but requires more memory. If your machine runs out of memory, IM automatically adjusts this field and retries until the operation succeeds.

Get a transform job’s details

Introduced 1.0

Returns a transform job’s details.

Sample Request

  1. GET _plugins/_transform/<transform_id>

Sample Response

  1. {
  2.   "_id": "sample",
  3.   "_version": 7,
  4.   "_seq_no": 13,
  5.   "_primary_term": 1,
  6.   "transform": {
  7.     "transform_id": "sample",
  8.     "schema_version": 7,
  9.     "continuous": true,
  10.     "schedule": {
  11.       "interval": {
  12.         "start_time": 1621467964243,
  13.         "period": 1,
  14.         "unit": "Minutes"
  15.       }
  16.     },
  17.     "metadata_id": null,
  18.     "updated_at": 1621467964243,
  19.     "enabled": true,
  20.     "enabled_at": 1621467964243,
  21.     "description": "Sample transform job",
  22.     "source_index": "sample_index",
  23.     "data_selection_query": {
  24.       "match_all": {
  25.         "boost": 1.0
  26.       }
  27.     },
  28.     "target_index": "sample_target",
  29.     "roles": [],
  30.     "page_size": 1,
  31.     "groups": [
  32.       {
  33.         "terms": {
  34.           "source_field": "customer_gender",
  35.           "target_field": "gender"
  36.         }
  37.       },
  38.       {
  39.         "terms": {
  40.           "source_field": "day_of_week",
  41.           "target_field": "day"
  42.         }
  43.       }
  44.     ],
  45.     "aggregations": {
  46.       "quantity": {
  47.         "sum": {
  48.           "field": "total_quantity"
  49.         }
  50.       }
  51.     }
  52.   }
  53. }

You can also get details of all transform jobs by omitting transform_id.

Sample Request

  1. GET _plugins/_transform/

Sample Response

  1. {
  2.   "total_transforms": 1,
  3.   "transforms": [
  4.     {
  5.       "_id": "sample",
  6.       "_seq_no": 13,
  7.       "_primary_term": 1,
  8.       "transform": {
  9.         "transform_id": "sample",
  10.         "schema_version": 7,
  11.         "continuous": true,
  12.         "schedule": {
  13.           "interval": {
  14.             "start_time": 1621467964243,
  15.             "period": 1,
  16.             "unit": "Minutes"
  17.           }
  18.         },
  19.         "metadata_id": null,
  20.         "updated_at": 1621467964243,
  21.         "enabled": true,
  22.         "enabled_at": 1621467964243,
  23.         "description": "Sample transform job",
  24.         "source_index": "sample_index",
  25.         "data_selection_query": {
  26.           "match_all": {
  27.             "boost": 1.0
  28.           }
  29.         },
  30.         "target_index": "sample_target",
  31.         "roles": [],
  32.         "page_size": 1,
  33.         "groups": [
  34.           {
  35.             "terms": {
  36.               "source_field": "customer_gender",
  37.               "target_field": "gender"
  38.             }
  39.           },
  40.           {
  41.             "terms": {
  42.               "source_field": "day_of_week",
  43.               "target_field": "day"
  44.             }
  45.           }
  46.         ],
  47.         "aggregations": {
  48.           "quantity": {
  49.             "sum": {
  50.               "field": "total_quantity"
  51.             }
  52.           }
  53.         }
  54.       }
  55.     }
  56.   ]
  57. }

You can specify these options as the GET API operation’s URL parameters to filter results:

ParameterDescriptionRequired
fromThe starting index to search from. Default is 0.No
sizeSpecifies the amount of results to return. Default is 10.No
searchThe search term to use to filter results.No
sortFieldThe field to sort results with.No
sortDirectionSpecifies the direction to sort results in. Can be ASC or DESC. Default is ASC.No

For example, this request returns two results starting from the eighth index.

Sample Request

  1. GET _plugins/_transform?size=2&from=8

Sample Response

  1. {
  2.   "total_transforms": 18,
  3.   "transforms": [
  4.     {
  5.       "_id": "sample8",
  6.       "_seq_no": 93,
  7.       "_primary_term": 1,
  8.       "transform": {
  9.         "transform_id": "sample8",
  10.         "schema_version": 7,
  11.         "schedule": {
  12.           "interval": {
  13.             "start_time": 1622063596812,
  14.             "period": 1,
  15.             "unit": "Minutes"
  16.           }
  17.         },
  18.         "metadata_id": "y4hFAB2ZURQ2dzY7BAMxWA",
  19.         "updated_at": 1622063657233,
  20.         "enabled": false,
  21.         "enabled_at": null,
  22.         "description": "Sample transform job",
  23.         "source_index": "sample_index3",
  24.         "data_selection_query": {
  25.           "match_all": {
  26.             "boost": 1.0
  27.           }
  28.         },
  29.         "target_index": "sample_target3",
  30.         "roles": [],
  31.         "page_size": 1,
  32.         "groups": [
  33.           {
  34.             "terms": {
  35.               "source_field": "customer_gender",
  36.               "target_field": "gender"
  37.             }
  38.           },
  39.           {
  40.             "terms": {
  41.               "source_field": "day_of_week",
  42.               "target_field": "day"
  43.             }
  44.           }
  45.         ],
  46.         "aggregations": {
  47.           "quantity": {
  48.             "sum": {
  49.               "field": "total_quantity"
  50.             }
  51.           }
  52.         }
  53.       }
  54.     },
  55.     {
  56.       "_id": "sample9",
  57.       "_seq_no": 98,
  58.       "_primary_term": 1,
  59.       "transform": {
  60.         "transform_id": "sample9",
  61.         "schema_version": 7,
  62.         "schedule": {
  63.           "interval": {
  64.             "start_time": 1622063598065,
  65.             "period": 1,
  66.             "unit": "Minutes"
  67.           }
  68.         },
  69.         "metadata_id": "x8tCIiYMTE3veSbIJkit5A",
  70.         "updated_at": 1622063658388,
  71.         "enabled": false,
  72.         "enabled_at": null,
  73.         "description": "Sample transform job",
  74.         "source_index": "sample_index4",
  75.         "data_selection_query": {
  76.           "match_all": {
  77.             "boost": 1.0
  78.           }
  79.         },
  80.         "target_index": "sample_target4",
  81.         "roles": [],
  82.         "page_size": 1,
  83.         "groups": [
  84.           {
  85.             "terms": {
  86.               "source_field": "customer_gender",
  87.               "target_field": "gender"
  88.             }
  89.           },
  90.           {
  91.             "terms": {
  92.               "source_field": "day_of_week",
  93.               "target_field": "day"
  94.             }
  95.           }
  96.         ],
  97.         "aggregations": {
  98.           "quantity": {
  99.             "sum": {
  100.               "field": "total_quantity"
  101.             }
  102.           }
  103.         }
  104.       }
  105.     }
  106.   ]
  107. }

Start a transform job

Introduced 1.0

Transform jobs created using the API are automatically enabled, but if you ever need to enable a job, you can use the start API operation.

Sample Request

  1. POST _plugins/_transform/<transform_id>/_start

Sample Response

  1. {
  2.   "acknowledged": true
  3. }

Stop a transform job

Introduced 1.0

Stops/disables a transform job.

Sample Request

  1. POST _plugins/_transform/<transform_id>/_stop

Sample Response

  1. {
  2.   "acknowledged": true
  3. }

Get the status of a transform job

Introduced 1.0

Returns the status and metadata of a transform job.

Sample Request

  1. GET _plugins/_transform/<transform_id>/_explain

Sample Response

  1. {
  2.   "sample": {
  3.     "metadata_id": "PzmjweME5xbgkenl9UpsYw",
  4.     "transform_metadata": {
  5.       "continuous_stats": {
  6.         "last_timestamp": 1621883525672,
  7.         "documents_behind": {
  8.           "sample_index": 72
  9.           }
  10.       },
  11.       "transform_id": "sample",
  12.       "last_updated_at": 1621883525873,
  13.       "status": "finished",
  14.       "failure_reason": "null",
  15.       "stats": {
  16.         "pages_processed": 0,
  17.         "documents_processed": 0,
  18.         "documents_indexed": 0,
  19.         "index_time_in_millis": 0,
  20.         "search_time_in_millis": 0
  21.       }
  22.     }
  23.   }
  24. }

Preview a transform job’s results

Introduced 1.0

Returns a preview of what a transformed index would look like.

Sample Request

  1. POST _plugins/_transform/_preview
  3. {
  4.   "transform": {
  5.   "enabled": false,
  6.   "schedule": {
  7.     "interval": {
  8.     "period": 1,
  9.     "unit": "Minutes",
  10.     "start_time": 1602100553
  11.     }
  12.   },
  13.   "description": "test transform",
  14.   "source_index": "sample_index",
  15.   "target_index": "sample_target",
  16.   "data_selection_query": {
  17.     "match_all": {}
  18.   },
  19.   "page_size": 10,
  20.   "groups": [
  21.     {
  22.     "terms": {
  23.       "source_field": "customer_gender",
  24.       "target_field": "gender"
  25.     }
  26.     },
  27.     {
  28.     "terms": {
  29.       "source_field": "day_of_week",
  30.       "target_field": "day"
  31.     }
  32.     }
  33.   ],
  34.   "aggregations": {
  35.     "quantity": {
  36.     "sum": {
  37.       "field": "total_quantity"
  38.     }
  39.     }
  40.   }
  41.   }
  42. }

Sample Response

  1. {
  2.   "documents" : [
  3.   {
  4.     "quantity" : 862.0,
  5.     "gender" : "FEMALE",
  6.     "day" : "Friday"
  7.   },
  8.   {
  9.     "quantity" : 682.0,
  10.     "gender" : "FEMALE",
  11.     "day" : "Monday"
  12.   },
  13.   {
  14.     "quantity" : 772.0,
  15.     "gender" : "FEMALE",
  16.     "day" : "Saturday"
  17.   },
  18.   {
  19.     "quantity" : 669.0,
  20.     "gender" : "FEMALE",
  21.     "day" : "Sunday"
  22.   },
  23.   {
  24.     "quantity" : 887.0,
  25.     "gender" : "FEMALE",
  26.     "day" : "Thursday"
  27.   }
  28.   ]
  29. }

Delete a transform job

Introduced 1.0

Deletes a transform job. This operation does not delete the source or target indexes.

Sample Request

  1. DELETE _plugins/_transform/<transform_id>

Sample Response

  1. {
  2.   "took": 205,
  3.   "errors": false,
  4.   "items": [
  5.     {
  6.       "delete": {
  7.         "_index": ".opensearch-ism-config",
  8.         "_id": "sample",
  9.         "_version": 4,
  10.         "result": "deleted",
  11.         "forced_refresh": true,
  12.         "_shards": {
  13.           "total": 2,
  14.           "successful": 1,
  15.           "failed": 0
  16.         },
  17.         "_seq_no": 6,
  18.         "_primary_term": 1,
  19.         "status": 200
  20.       }
  21.     }
  22.   ]
  23. }