Transforms APIs

Aside from using OpenSearch Dashboards, you can also use the REST API to create, start, stop, and complete other operations relative to transform jobs.

Table of contents

Create a transform job

Introduced 1.0

Creates a transform job.

Sample Request

  1. PUT _plugins/_transform/<transform_id>
  2. {
  3. "transform": {
  4. "enabled": true,
  5. "schedule": {
  6. "interval": {
  7. "period": 1,
  8. "unit": "Minutes",
  9. "start_time": 1602100553
  10. }
  11. },
  12. "description": "Sample transform job",
  13. "source_index": "sample_index",
  14. "target_index": "sample_target",
  15. "data_selection_query": {
  16. "match_all": {}
  17. },
  18. "page_size": 1,
  19. "groups": [
  20. {
  21. "terms": {
  22. "source_field": "customer_gender",
  23. "target_field": "gender"
  24. }
  25. },
  26. {
  27. "terms": {
  28. "source_field": "day_of_week",
  29. "target_field": "day"
  30. }
  31. }
  32. ],
  33. "aggregations": {
  34. "quantity": {
  35. "sum": {
  36. "field": "total_quantity"
  37. }
  38. }
  39. }
  40. }
  41. }

Sample Response

  1. {
  2. "_id": "sample",
  3. "_version": 7,
  4. "_seq_no": 13,
  5. "_primary_term": 1,
  6. "transform": {
  7. "transform_id": "sample",
  8. "schema_version": 7,
  9. "schedule": {
  10. "interval": {
  11. "start_time": 1621467964243,
  12. "period": 1,
  13. "unit": "Minutes"
  14. }
  15. },
  16. "metadata_id": null,
  17. "updated_at": 1621467964243,
  18. "enabled": true,
  19. "enabled_at": 1621467964243,
  20. "description": "Sample transform job",
  21. "source_index": "sample_index",
  22. "data_selection_query": {
  23. "match_all": {
  24. "boost": 1.0
  25. }
  26. },
  27. "target_index": "sample_target",
  28. "roles": [],
  29. "page_size": 1,
  30. "groups": [
  31. {
  32. "terms": {
  33. "source_field": "customer_gender",
  34. "target_field": "gender"
  35. }
  36. },
  37. {
  38. "terms": {
  39. "source_field": "day_of_week",
  40. "target_field": "day"
  41. }
  42. }
  43. ],
  44. "aggregations": {
  45. "quantity": {
  46. "sum": {
  47. "field": "total_quantity"
  48. }
  49. }
  50. }
  51. }
  52. }

You can specify the following options in the HTTP request body:

OptionData TypeDescriptionRequired
enabledBooleanIf true, the transform job is enabled at creation.No
scheduleJSONThe schedule the transform job runs on.Yes
start_timeIntegerThe Unix epoch time of the transform job’s start time.Yes
descriptionStringDescribes the transform job.No
metadata_idStringAny metadata to be associated with the transform job.No
source_indexStringThe source index whose data to transform.Yes
target_indexStringThe target index the newly transformed data is added into. You can create a new index or update an existing one.Yes
data_selection_queryJSONThe query DSL to use to filter a subset of the source index for the transform job. See query DSL for more information.Yes
page_sizeIntegerThe number of fields to transform at a time. Higher number means higher performance but requires more memory and can cause higher latency. (Default: 1)Yes
groupsArraySpecifies the grouping(s) to use in the transform job. Supported groups are terms, histogram, and date_histogram. For more information, see Bucket Aggregations.Yes if not using aggregations
source_fieldStringThe field(s) to transformYes
aggregationsJSONThe aggregations to use in the transform job. Supported aggregations are: sum, max, min, value_count, avg, scripted_metric, and percentiles. For more information, see Metric Aggregations.Yes if not using groups

Update a transform job

Introduced 1.0

Updates a transform job if transform_id already exists.

Sample Request

  1. PUT _plugins/_transform/<transform_id>
  2. {
  3. "transform": {
  4. "enabled": true,
  5. "schedule": {
  6. "interval": {
  7. "period": 1,
  8. "unit": "Minutes",
  9. "start_time": 1602100553
  10. }
  11. },
  12. "description": "Sample transform job",
  13. "source_index": "sample_index",
  14. "target_index": "sample_target",
  15. "data_selection_query": {
  16. "match_all": {}
  17. },
  18. "page_size": 1,
  19. "groups": [
  20. {
  21. "terms": {
  22. "source_field": "customer_gender",
  23. "target_field": "gender"
  24. }
  25. },
  26. {
  27. "terms": {
  28. "source_field": "day_of_week",
  29. "target_field": "day"
  30. }
  31. }
  32. ],
  33. "aggregations": {
  34. "quantity": {
  35. "sum": {
  36. "field": "total_quantity"
  37. }
  38. }
  39. }
  40. }
  41. }

Sample Response

  1. {
  2. "_id": "sample",
  3. "_version": 2,
  4. "_seq_no": 14,
  5. "_primary_term": 1,
  6. "transform": {
  7. "transform_id": "sample",
  8. "schema_version": 7,
  9. "schedule": {
  10. "interval": {
  11. "start_time": 1602100553,
  12. "period": 1,
  13. "unit": "Minutes"
  14. }
  15. },
  16. "metadata_id": null,
  17. "updated_at": 1621889843874,
  18. "enabled": true,
  19. "enabled_at": 1621889843874,
  20. "description": "Sample transform job",
  21. "source_index": "sample_index",
  22. "data_selection_query": {
  23. "match_all": {
  24. "boost": 1.0
  25. }
  26. },
  27. "target_index": "sample_target",
  28. "roles": [],
  29. "page_size": 1,
  30. "groups": [
  31. {
  32. "terms": {
  33. "source_field": "customer_gender",
  34. "target_field": "gender"
  35. }
  36. },
  37. {
  38. "terms": {
  39. "source_field": "day_of_week",
  40. "target_field": "day"
  41. }
  42. }
  43. ],
  44. "aggregations": {
  45. "quantity": {
  46. "sum": {
  47. "field": "total_quantity"
  48. }
  49. }
  50. }
  51. }
  52. }

The Update operation supports the following URL parameters:

ParameterDescriptionRequired
if_seq_noOnly perform the transform operation if the last operation that changed the transform job has the specified sequence number.No
if_primary_termOnly perform the transform operation if the last operation that changed the transform job has the specified sequence term.No

Get a transform job’s details

Introduced 1.0

Returns a transform job’s details.

Sample Request

  1. GET _plugins/_transform/<transform_id>

Sample Response

  1. {
  2. "_id": "sample",
  3. "_version": 7,
  4. "_seq_no": 13,
  5. "_primary_term": 1,
  6. "transform": {
  7. "transform_id": "sample",
  8. "schema_version": 7,
  9. "schedule": {
  10. "interval": {
  11. "start_time": 1621467964243,
  12. "period": 1,
  13. "unit": "Minutes"
  14. }
  15. },
  16. "metadata_id": null,
  17. "updated_at": 1621467964243,
  18. "enabled": true,
  19. "enabled_at": 1621467964243,
  20. "description": "Sample transform job",
  21. "source_index": "sample_index",
  22. "data_selection_query": {
  23. "match_all": {
  24. "boost": 1.0
  25. }
  26. },
  27. "target_index": "sample_target",
  28. "roles": [],
  29. "page_size": 1,
  30. "groups": [
  31. {
  32. "terms": {
  33. "source_field": "customer_gender",
  34. "target_field": "gender"
  35. }
  36. },
  37. {
  38. "terms": {
  39. "source_field": "day_of_week",
  40. "target_field": "day"
  41. }
  42. }
  43. ],
  44. "aggregations": {
  45. "quantity": {
  46. "sum": {
  47. "field": "total_quantity"
  48. }
  49. }
  50. }
  51. }
  52. }

You can also get details of all transform jobs by omitting transform_id.

Sample Request

  1. GET _plugins/_transform/

Sample Response

  1. {
  2. "total_transforms": 1,
  3. "transforms": [
  4. {
  5. "_id": "sample",
  6. "_seq_no": 13,
  7. "_primary_term": 1,
  8. "transform": {
  9. "transform_id": "sample",
  10. "schema_version": 7,
  11. "schedule": {
  12. "interval": {
  13. "start_time": 1621467964243,
  14. "period": 1,
  15. "unit": "Minutes"
  16. }
  17. },
  18. "metadata_id": null,
  19. "updated_at": 1621467964243,
  20. "enabled": true,
  21. "enabled_at": 1621467964243,
  22. "description": "Sample transform job",
  23. "source_index": "sample_index",
  24. "data_selection_query": {
  25. "match_all": {
  26. "boost": 1.0
  27. }
  28. },
  29. "target_index": "sample_target",
  30. "roles": [],
  31. "page_size": 1,
  32. "groups": [
  33. {
  34. "terms": {
  35. "source_field": "customer_gender",
  36. "target_field": "gender"
  37. }
  38. },
  39. {
  40. "terms": {
  41. "source_field": "day_of_week",
  42. "target_field": "day"
  43. }
  44. }
  45. ],
  46. "aggregations": {
  47. "quantity": {
  48. "sum": {
  49. "field": "total_quantity"
  50. }
  51. }
  52. }
  53. }
  54. }
  55. ]
  56. }

You can specify these options as the GET API operation’s URL parameters to filter results:

ParameterDescriptionRequired
fromThe starting index to search from. (Default: 0)No
sizeSpecifies the amount of results to return (Default: 10)No
searchThe search term to use to filter results.No
sortFieldThe field to sort results with.No
sortDirectionSpecifies the direction to sort results in. Can be ASC or DESC. (Default: ASC)No

For example, this request returns two results starting from the eighth index.

Sample Request

  1. GET _plugins/_transform?size=2&from=8

Sample Response

  1. {
  2. "total_transforms": 18,
  3. "transforms": [
  4. {
  5. "_id": "sample8",
  6. "_seq_no": 93,
  7. "_primary_term": 1,
  8. "transform": {
  9. "transform_id": "sample8",
  10. "schema_version": 7,
  11. "schedule": {
  12. "interval": {
  13. "start_time": 1622063596812,
  14. "period": 1,
  15. "unit": "Minutes"
  16. }
  17. },
  18. "metadata_id": "y4hFAB2ZURQ2dzY7BAMxWA",
  19. "updated_at": 1622063657233,
  20. "enabled": false,
  21. "enabled_at": null,
  22. "description": "Sample transform job",
  23. "source_index": "sample_index3",
  24. "data_selection_query": {
  25. "match_all": {
  26. "boost": 1.0
  27. }
  28. },
  29. "target_index": "sample_target3",
  30. "roles": [],
  31. "page_size": 1,
  32. "groups": [
  33. {
  34. "terms": {
  35. "source_field": "customer_gender",
  36. "target_field": "gender"
  37. }
  38. },
  39. {
  40. "terms": {
  41. "source_field": "day_of_week",
  42. "target_field": "day"
  43. }
  44. }
  45. ],
  46. "aggregations": {
  47. "quantity": {
  48. "sum": {
  49. "field": "total_quantity"
  50. }
  51. }
  52. }
  53. }
  54. },
  55. {
  56. "_id": "sample9",
  57. "_seq_no": 98,
  58. "_primary_term": 1,
  59. "transform": {
  60. "transform_id": "sample9",
  61. "schema_version": 7,
  62. "schedule": {
  63. "interval": {
  64. "start_time": 1622063598065,
  65. "period": 1,
  66. "unit": "Minutes"
  67. }
  68. },
  69. "metadata_id": "x8tCIiYMTE3veSbIJkit5A",
  70. "updated_at": 1622063658388,
  71. "enabled": false,
  72. "enabled_at": null,
  73. "description": "Sample transform job",
  74. "source_index": "sample_index4",
  75. "data_selection_query": {
  76. "match_all": {
  77. "boost": 1.0
  78. }
  79. },
  80. "target_index": "sample_target4",
  81. "roles": [],
  82. "page_size": 1,
  83. "groups": [
  84. {
  85. "terms": {
  86. "source_field": "customer_gender",
  87. "target_field": "gender"
  88. }
  89. },
  90. {
  91. "terms": {
  92. "source_field": "day_of_week",
  93. "target_field": "day"
  94. }
  95. }
  96. ],
  97. "aggregations": {
  98. "quantity": {
  99. "sum": {
  100. "field": "total_quantity"
  101. }
  102. }
  103. }
  104. }
  105. }
  106. ]
  107. }

Start a transform job

Introduced 1.0

Transform jobs created using the API are automatically enabled, but if you ever need to enable a job, you can use the start API operation.

Sample Request

  1. POST _plugins/_transform/<transform_id>/_start

Sample Response

  1. {
  2. "acknowledged": true
  3. }

Stop a transform job

Introduced 1.0

Stops/disables a transform job.

Sample Request

  1. POST _plugins/_transform/<transform_id>/_stop

Sample Response

  1. {
  2. "acknowledged": true
  3. }

Get the status of a transform job

Introduced 1.0

Returns the status and metadata of a transform job.

Sample Request

  1. GET _plugins/_transform/<transform_id>/_explain

Sample Response

  1. {
  2. "sample": {
  3. "metadata_id": "PzmjweME5xbgkenl9UpsYw",
  4. "transform_metadata": {
  5. "transform_id": "sample",
  6. "last_updated_at": 1621883525873,
  7. "status": "finished",
  8. "failure_reason": "null",
  9. "stats": {
  10. "pages_processed": 0,
  11. "documents_processed": 0,
  12. "documents_indexed": 0,
  13. "index_time_in_millis": 0,
  14. "search_time_in_millis": 0
  15. }
  16. }
  17. }
  18. }

Preview a transform job’s results

Introduced 1.0

Returns a preview of what a transformed index would look like.

Sample Request

  1. POST _plugins/_transform/_preview
  2. {
  3. "transform": {
  4. "enabled": false,
  5. "schedule": {
  6. "interval": {
  7. "period": 1,
  8. "unit": "Minutes",
  9. "start_time": 1602100553
  10. }
  11. },
  12. "description": "test transform",
  13. "source_index": "sample_index",
  14. "target_index": "sample_target",
  15. "data_selection_query": {
  16. "match_all": {}
  17. },
  18. "page_size": 10,
  19. "groups": [
  20. {
  21. "terms": {
  22. "source_field": "customer_gender",
  23. "target_field": "gender"
  24. }
  25. },
  26. {
  27. "terms": {
  28. "source_field": "day_of_week",
  29. "target_field": "day"
  30. }
  31. }
  32. ],
  33. "aggregations": {
  34. "quantity": {
  35. "sum": {
  36. "field": "total_quantity"
  37. }
  38. }
  39. }
  40. }
  41. }

Sample Response

  1. {
  2. "documents" : [
  3. {
  4. "quantity" : 862.0,
  5. "gender" : "FEMALE",
  6. "day" : "Friday"
  7. },
  8. {
  9. "quantity" : 682.0,
  10. "gender" : "FEMALE",
  11. "day" : "Monday"
  12. },
  13. {
  14. "quantity" : 772.0,
  15. "gender" : "FEMALE",
  16. "day" : "Saturday"
  17. },
  18. {
  19. "quantity" : 669.0,
  20. "gender" : "FEMALE",
  21. "day" : "Sunday"
  22. },
  23. {
  24. "quantity" : 887.0,
  25. "gender" : "FEMALE",
  26. "day" : "Thursday"
  27. }
  28. ]
  29. }

Delete a transform job

Introduced 1.0

Deletes a transform job. This operation does not delete the source or target indices.

Sample Request

  1. DELETE _plugins/_transform/<transform_id>

Sample Response

  1. {
  2. "took": 205,
  3. "errors": false,
  4. "items": [
  5. {
  6. "delete": {
  7. "_index": ".opensearch-ism-config",
  8. "_type": "_doc",
  9. "_id": "sample",
  10. "_version": 4,
  11. "result": "deleted",
  12. "forced_refresh": true,
  13. "_shards": {
  14. "total": 2,
  15. "successful": 1,
  16. "failed": 0
  17. },
  18. "_seq_no": 6,
  19. "_primary_term": 1,
  20. "status": 200
  21. }
  22. }
  23. ]
  24. }