Scripts and search speed

Scripts can’t make use of Elasticsearch’s index structures or related optimizations. This can sometimes result in slower search speeds.

If you often use scripts to transform indexed data, you can speed up search by making these changes during ingest instead. However, that often means slower index speeds.

Example

An index, my_test_scores, contains two long fields:

  • math_score
  • verbal_score

When running searches, users often use a script to sort results by the sum of these two field’s values.

  1. GET /my_test_scores/_search
  2. {
  3. "query": {
  4. "term": {
  5. "grad_year": "2099"
  6. }
  7. },
  8. "sort": [
  9. {
  10. "_script": {
  11. "type": "number",
  12. "script": {
  13. "source": "doc['math_score'].value + doc['verbal_score'].value"
  14. },
  15. "order": "desc"
  16. }
  17. }
  18. ]
  19. }

To speed up search, you can perform this calculation during ingest and index the sum to a field instead.

First, add a new field, total_score, to the index. The total_score field will contain sum of the math_score and verbal_score field values.

  1. PUT /my_test_scores/_mapping
  2. {
  3. "properties": {
  4. "total_score": {
  5. "type": "long"
  6. }
  7. }
  8. }

Next, use an ingest pipeline containing the script processor to calculate the sum of math_score and verbal_score and index it in the total_score field.

  1. PUT _ingest/pipeline/my_test_scores_pipeline
  2. {
  3. "description": "Calculates the total test score",
  4. "processors": [
  5. {
  6. "script": {
  7. "source": "ctx.total_score = (ctx.math_score + ctx.verbal_score)"
  8. }
  9. }
  10. ]
  11. }

To update existing data, use this pipeline to reindex any documents from my_test_scores to a new index, my_test_scores_2.

  1. POST /_reindex
  2. {
  3. "source": {
  4. "index": "my_test_scores"
  5. },
  6. "dest": {
  7. "index": "my_test_scores_2",
  8. "pipeline": "my_test_scores_pipeline"
  9. }
  10. }

Continue using the pipeline to index any new documents to my_test_scores_2.

  1. POST /my_test_scores_2/_doc/?pipeline=my_test_scores_pipeline
  2. {
  3. "student": "kimchy",
  4. "grad_year": "2099",
  5. "math_score": 800,
  6. "verbal_score": 800
  7. }

These changes may slow indexing but allow for faster searches. Users can now sort searches made on my_test_scores_2 using the total_score field instead of using a script.

  1. GET /my_test_scores_2/_search
  2. {
  3. "query": {
  4. "term": {
  5. "grad_year": "2099"
  6. }
  7. },
  8. "sort": [
  9. {
  10. "total_score": {
  11. "order": "desc"
  12. }
  13. }
  14. ]
  15. }

We recommend testing and benchmarking any indexing changes before deploying them in production.

Script errors

Elasticsearch returns error details when there is a compliation or runtime exception. The contents of this response are useful for tracking down the problem.

This functionality is experimental and may be changed or removed completely in a future release. Elastic will take a best effort approach to fix any issues, but experimental features are not subject to the support SLA of official GA features.

The contents of position are experimental and subject to change.