Elasticsearch version 7.6.0

Also see Breaking changes in 7.6.

Known issues

  • Applying deletes or updates on an index after it has been shrunk may corrupt the index. In order to prevent this issue, it is recommended to stop shrinking read-write indices. For read-only indices, it is recommended to force-merge indices after shrinking, which significantly reduces the likeliness of this corruption in the case that deletes/updates would be applied by mistake. This bug is fixed in Elasticsearch 7.7 and later versions. More details can be found on the corresponding issue.
  • Indices created in 6.x with date and date_nanos fields using formats that are incompatible with java.time patterns will cause parsing errors, incorrect date calculations or wrong search results. https://github.com/elastic/elasticsearch/pull/52555 This is fixed in Elasticsearch 7.7 and later versions.

  • Slow loggers can cause Log4j loggers to leak over time. When a new index is created, a new Log4j logger is associated with it. However, when an index is deleted, Log4j keeps an internal reference to its loggers that results in a memory leak (issue: #56171)

    This issue is fixed in Elasticsearch 6.8.10 and 7.7.1.

  • Week-based date patterns are not working correctly with Y. Using Y with w will result in a failed request and an exception in the logs (issue: #57128). Using y with w results in incorrect date calculations. A workaround is to add the following line to the jvm.options file.

    1. 9-:-Djava.locale.providers=SPI,COMPAT

    This issue is fixed in Elasticsearch 7.7.0 and later versions (issue: #50916).

Breaking changes

Mapping

  • Add a cluster setting to disallow loading fielddata on _id field #49166 (issues: #26472, #43599)

Breaking Java changes

Security

  • Support Client and RoleMapping in custom Realms #50534 (issue: #48369)

Deprecations

Analysis

  • Deprecate and remove camel-case nGram and edgeNGram tokenizers #50862 (issue: #50561)

Authorization

  • Deprecating kibana_user and kibana_dashboard_only_user roles #46456

Distributed

  • Deprecate synced flush #50835 (issue: #50776)
  • Deprecate indices without soft-deletes #50502

Features/Indices APIs

  • Emit warnings when index templates have multiple mappings #50982
  • Ensure we emit a warning when using the deprecated template field. #50831 (issue: #49460)

Infra/Core

  • Deprecate the local parameter of /_cat/nodes #50499 (issue: #50088)

Reindex

Search

  • Update the signature of vector script functions. #48604
  • Deprecate the sparse_vector field type. #48315
  • Add a deprecation warning regarding allocation awareness in search request #48351 (issue: #43453)

New features

Aggregations

  • New Histogram field mapper that supports percentiles aggregations. #48580 (issue: #48578)
  • Implement stats aggregation for string terms #47468

Analysis

  • Implement Lucene EstonianAnalyzer, Stemmer #49149 (issue: #48895)

Authentication

  • Password Protected Keystore (Feature Branch) #49210

Features/ILM+SLM

Features/Ingest

Machine Learning

  • Implement precision and recall metrics for classification evaluation #49671 (issue: #48759)
  • Explain data frame analytics API #49455
  • Machine learning model inference ingest processor #49052
  • Implement accuracy metric for multi-class classification #47772 (issue: #48759)
  • Add feature importance values to classification and regression results (using tree SHapley Additive exPlanation, or SHAP) #857

Mapping

Search

  • Add fuzzy intervals source #49762 (issue: #49595)
  • Add a listener to track the progress of a search request locally #49471 (issue: #49091)

Enhancements

Aggregations

  • Add reusable HistogramValue object #49799 (issue: #49683)
  • Optimize composite aggregation based on index sorting #48399 (issue: #48130)

Allocation

  • Auto-expand indices according to allocation filtering rules #48974
  • Do not cancel ongoing recovery for noop copy on broken node #48265 (issue: #47974)
  • Quieter logging from the DiskThresholdMonitor #48115 (issue: #48038)
  • Faster access to INITIALIZING/RELOCATING shards #47817 (issues: #46941, #48579)

Analysis

  • Check for deprecations when analyzers are built #50908 (issue: #42349)
  • Make Multiplexer inherit filter chains analysis mode #50662 (issue: #50554)
  • Allow custom characters in token_chars of ngram tokenizers #49250 (issue: #25894)

Authentication

  • Add Debug/Trace logging for authentication #49575 (issue: #49473)

Authorization

  • Increase Size and lower TTL on DLS BitSet Cache #50535 (issues: #43669, #49260)
  • Add monitor_snapshot cluster privilege #50489 (issue: #50210)
  • Remove reserved roles for code search #50068 (issue: #49842)
  • [Code] Remove code_admin/code_user roles #48164
  • Resolve the role query and the number of docs lazily #48036

CCR

  • Improve error message when pausing index #48915
  • Use MultiFileTransfer in CCR remote recovery #44514 (issue: #44468)

CRUD

  • print id detail when id is too long. #49433
  • Add preflight check to dynamic mapping updates #48817 (issue: #35564)

Cluster Coordination

Distributed

  • Use retention lease in peer recovery of closed indices #48430 (issue: #45136)

Engine

  • Do not force refresh when write indexing buffer #50769
  • Deleted docs disregarded for if_seq_no check #50526
  • Allow realtime get to read from translog #48843
  • Do not warm up searcher in engine constructor #48605 (issue: #47186)
  • Add a new merge policy that interleaves old and new segments on force merge #48533 (issue: #37043)
  • Refresh should not acquire readLock #48414 (issue: #47186)

Features/ILM+SLM

  • Refresh cached phase policy definition if possible on new poli… #50820 (issue: #48431)
  • Make the UpdateRolloverLifecycleDateStep retryable #50702 (issue: #48183)
  • Make InitializePolicyContextStep retryable #50685 (issue: #48183)
  • ILM retryable async action steps #50522 (issues: #44135, #48183)
  • Make the TransportRolloverAction execute in one cluster state update #50388
  • ILM open/close steps are noop if idx is open/close #48614
  • ILM Make the check-rollover-ready step retryable #48256 (issue: #44135)

Features/Ingest

  • Foreach processor - fork recursive call #50514
  • Sync grok patterns with logstash patterns #50381
  • Replace required pipeline with final pipeline #49470 (issue: #49247)
  • Add templating support to enrich processor #49093
  • Introduce on_failure_pipeline ingest metadata inside on_failure block #49076 (issue: #44920)
  • Add templating support to pipeline processor. #49030 (issue: #39955)
  • Add option to split processor for preserving trailing empty fields #48664 (issue: #48498)
  • Change grok watch dog to be Matcher based instead of thread based. #48346 (issues: #43673, #47374)
  • update ingest-user-agent regexes.yml #47807

Features/Java High Level REST Client

Features/Monitoring

  • Significantly Lower Monitoring HttpExport Memory Footprint #48854
  • Validate proxy base path at parse time #47912 (issue: #47711)
  • Validate index name time format setting at parse time #47911 (issue: #47711)
  • Validate monitoring header overrides at parse time #47848 (issue: #47711)
  • Validate monitoring username at parse time #47821 (issue: #47711)
  • Validate monitoring password at parse time #47740 (issue: #47711)

Features/Stats

Features/Watcher

  • Log attachment generation failures #50080
  • Don’t dump a stacktrace for invalid patterns when executing elasticse… #49744 (issue: #49642)

Geo

  • “CONTAINS” support for BKD-backed geo_shape and shape fields #50141 (issue: #41204)
  • Adds support for geo-bounds filtering in geogrid aggregations #50002
  • Introduce faster approximate sinh/atan math functions #49009 (issue: #41166)
  • Add IndexOrDocValuesQuery to GeoPolygonQueryBuilder #48449

Infra/Core

  • Add “did you mean” to ObjectParser #50938
  • Consistent case in CLI option descriptions #49635
  • Improve resiliency to formatting JSON in server #48553 (issue: #48450)
  • Don’t close stderr under --quiet #47208 (issue: #46900)

Infra/Packaging

  • Respect ES_PATH_CONF on package install #50158
  • Restrict support for CMS to pre-JDK 14 #49123 (issue: #46973)
  • Remove parsed JVM settings from general settings in Windows service daemon manager #49061 (issue: #48796)
  • Package the JDK into jdk.app on macOS #48765
  • Add UBI-based Docker images #48710 (issue: #48429)

Infra/Plugins

  • Report progress of multiple plugin installs #51001 (issue: #50924)
  • Allow installing multiple plugins as a transaction #50924 (issue: #50443)

Infra/Scripting

  • Scripting: ScriptFactory not required by compile #50344 (issue: #49466)
  • Scripting: Cache script results if deterministic #50106 (issue: #49466)
  • Scripting: Groundwork for caching script results #49895 (issue: #49466)
  • Scripting: add available languages & contexts API #49652 (issue: #49463)
  • Scripting: fill in get contexts REST API #48319 (issue: #47411)
  • Scripting: get context names REST API #48026 (issue: #47411)

Infra/Settings

  • Add parameter to make sure that log of updating IndexSetting be more detailed #49969 (issue: #49818)
  • Enable dependent settings values to be validated #49942
  • Do not reference values for filtered settings #48066

License

  • Add max_resource_units to enterprise license #50735
  • Add setting to restrict license types #49418 (issue: #48508)
  • Support “enterprise” license types #49223 (issue: #48510)

Machine Learning

  • Add audit warning for 1000 categories found early in job #51146 (issue: #50749)
  • Add num_top_feature_importance_values param to regression and classification #50914
  • Implement force deleting a data frame analytics job #50553 (issue: #48124)
  • Delete unused data frame analytics state #50243
  • Make each analysis report desired field mappings to be copied #50219 (issue: #50119)
  • Retry bulk indexing of state docs #50149 (issue: #50143)
  • Persist/restore state for data frame analytics classification #50040
  • Introduce randomize_seed setting for regression and classification #49990
  • Pass prediction_field_type to C++ analytics process #49861 (issue: #49796)
  • Add optional source filtering during data frame reindexing #49690 (issue: #49531)
  • Add default categorization analyzer definition to ML info #49545
  • Add graceful retry for anomaly detector result indexing failures #49508 (issue: #45711)
  • Lower minimum model memory limit value for data frame analytics jobs from 1MB to 1kB #49227 (issue: #49168)
  • Improve model_memory_limit user experience for data frame analytics jobs #44699
  • Improve performance of boosted tree training for both classification and regression #775
  • Reduce the peak memory used by boosted tree training and fix an overcounting bug estimating maximum memory usage #781
  • Stratified fractional cross validation for regression #784
  • Added geo_point supported output for lat_long function records #809, #47050
  • Use a random bag of the data to compute the loss function derivatives for each new tree which is trained for both regression and classification #811
  • Emit prediction_probability field alongside prediction field in ml results #818
  • Reduce memory usage of machine learning native processes on Windows #844
  • Reduce runtime of classification and regression #863
  • Stop early training a classification and regression forest when the validation error is no longer decreasing #875
  • Emit prediction_field_name in data frame analytics results using the type provided as prediction_field_type parameter #877
  • Improve performance updating quantile estimates #881
  • Migrate to use Bayesian optimisation for initial hyperparameter value line searches and stop early if the expected improvement is too small #903
  • Stop cross-validation early if the predicted test loss has a small chance of being smaller than for the best parameter values found so far #915
  • Optimize decision threshold for classification to maximize minimum class recall #926
  • Include categorization memory usage in the model_bytes field in model_size_stats, so that it is taken into account in node assignment decisions #927 (issue: #724)

Mapping

  • Add telemetry for flattened fields. #48972

Network

  • Add certutil http command #49827
  • Do not load SSLService in plugin contructor #49667 (issue: #44536)
  • Netty4: switch to composite cumulator #49478
  • Add the simple strategy to cluster settings #49414 (issue: #49067)
  • Deprecate misconfigured SSL server config #49280 (issue: #45892)
  • Improved diagnostics for TLS trust failures #48911

Percolator

  • Refactor percolator’s QueryAnalyzer to use QueryVisitors #49238 (issue: #45639)

Ranking

  • Support search_type in Rank Evaluation API #48542 (issue: #48503)

Recovery

  • Use peer recovery retention leases for indices without soft-deletes #50351 (issues: #45136, #46959)
  • Recovery buffer size 16B smaller #50100

Reindex

  • Reindex sort deprecation warning take 2 #49855 (issue: #49458)

SQL

  • SQL: Handle uberjar scenario where the ES jdbc driver file is bundled in another jar #51856 (issue: #50201)
  • SQL: add trace logging for search responses coming from server #50530
  • SQL: Add TRUNC alias for TRUNCATE #49571 (issue: #41195)
  • SQL: binary communication implementation for drivers and the CLI #48261 (issue: #47785)
  • SQL: Verify Full-Text Search functions not allowed in SELECT #51568 (issue: #47446)

Search

  • Add Validation for maxQueryTerms to be greater than 0 for MoreLikeThisQuery #49966 (issue: #49927)
  • Optimize numeric sort on match_all queries #49717 (issue: #48804)
  • Pre-sort shards based on the max/min value of the primary sort field #49092 (issue: #49091)
  • Optimize sort on long field #48804
  • Search optimisation - add canMatch early aborts for queries on “_index” field #48681 (issue: #48473)

  • 48475 Pure disjunctions should rewrite to a MatchNoneQueryBuilder #48557

  • Disable caching when queries are profiled #48195 (issue: #33298)
  • BlendedTermQuery’s equals method should consider boosts #48193 (issue: #48184)
  • Increase the number of vector dims to 2048 #46895

Security

  • Make .async-search-* a restricted namespace #50294
  • Security should not reload files that haven’t changed #50207 (issue: #50063)

Snapshot/Restore

Store

Transform

  • Improve force stop robustness in case of an error #51072
  • Add actual timeout in message #50140
  • Automatic deletion of old checkpoints #49496
  • Improve error handling of script errors #48887 (issue: #48467)
  • Add wait_for_checkpoint flag to stop #47935 (issue: #45293)

Bug fixes

Aggregations

  • Use #name() instead of #simpleName() when generating doc values #51920 (issues: #50307, #51847)
  • Fix a sneaky bug in rare_terms #51868 (issue: #51020)
  • Support time_zone on composite’s date_histogram #51172 (issues: #45199, #45200)
  • Fix format problem in composite of unmapped #50869 (issue: #50600)
  • SingleBucket aggs need to reduce their bucket’s pipelines first #50103 (issue: #50054)
  • Avoid precision loss in DocValueFormat.RAW#parseLong #49063 (issue: #38692)
  • Fix ignoring missing values in min/max aggregations #48970 (issue: #48905)

Allocation

  • Collect shard sizes for closed indices #50645 (issue: #33888)
  • Auto-expand replicated closed indices #48973
  • Ignore dangling indices created in newer versions #48652 (issue: #34264)
  • Handle negative free disk space in deciders #48392 (issue: #48380)

Analysis

  • Fix caching for PreConfiguredTokenFilter #50912 (issue: #50734)
  • Throw Error on deprecated nGram and edgeNGram custom filters #50376 (issue: #50360)
  • _analyze api does not correctly use normalizers when specified #48866 (issue: #48650)

Audit

Authentication

  • Preserve ApiKey credentials for async verification #51244
  • Don’t fallback to anonymous for tokens/apikeys #51042 (issue: #50171)
  • Populate User metadata with OpenIDConnect collections #50521 (issue: #50250)
  • Always return 401 for not valid tokens #49736 (issue: #38866)
  • Fix iterate-from-1 bug in smart realm order #49473
  • Remove unnecessary details logged for OIDC #48746
  • Add owner flag parameter to the rest spec #48500 (issue: #48499)

Authorization

CCR

  • CCR should auto-retry rejected execution exceptions #49213

CRUD

  • Block too many concurrent mapping updates #51038 (issue: #50670)
  • Ensure meta and document field maps are never null in GetResult #50112 (issue: #48215)
  • Replicate write actions before fsyncing them #49746
  • Do not mutate request on scripted upsert #49578 (issue: #48670)
  • Fix Transport Stopped Exception #48930 (issue: #42612)
  • Return consistent source in updates #48707
  • Close query cache on index service creation failure #48230 (issue: #48186)

Cluster Coordination

  • Import replicated closed dangling indices #50649
  • Ignore metadata of deleted indices at start #48918
  • Make elasticsearch-node tools custom metadata-aware #48390

Discovery-Plugins

  • Make EC2 Discovery Cache Empty Seed Hosts List #50607 (issue: #50550)
  • Make EC2 Discovery Plugin Retry Requests #50550 (issue: #50462)

Distributed

  • Exclude nested documents in LuceneChangesSnapshot #51279
  • Closed shard should never open new engine #47186 (issues: #45263, #47060)
  • Fix meta version of task index mapping #50363 (issue: #48393)

Engine

  • Do not wrap soft-deletes reader for segment stats #51331 (issues: #51192, #51303)
  • Account soft-deletes in FrozenEngine #51192 (issue: #50775)
  • Account trimAboveSeqNo in committed translog generation #50205 (issue: #49970)
  • Greedily advance safe commit on new global checkpoint #48559 (issue: #48532)
  • Do not ignore exception when trim unreferenced readers #48470

Features/Features

  • Fix X-Pack SchedulerEngine Shutdown #48951

Features/ILM+SLM

  • Fix SLM check for restore in progress #50868
  • Handle failure to retrieve ILM policy step better #49193 (issue: #49128)
  • Don’t halt policy execution on policy trigger exception #49128
  • Re-read policy phase JSON when using ILM’s move-to-step API #48827
  • Don’t schedule SLM jobs when services have been stopped #48658 (issue: #47749)
  • Ensure SLM stats does not block an in-place upgrade from 7.4 #48367
  • Ensure SLM stats does not block an in-place upgrade from 7.4 #48361
  • Add SLM support to xpack usage and info APIs #48096 (issue: #43663)
  • Change policy_id to list type in slm.get_lifecycle #47766 (issue: #47765)

Features/Ingest

  • Fix ignore_missing in CsvProcessor #51600
  • Don’t overwrite target field with SetSecurityUserProcessor #51454 (issue: #51428)
  • Fix ingest simulate response document order if processor executes async #50244
  • Allow list of IPs in geoip ingest processor #49573 (issue: #46193)
  • Do not wrap ingest processor exception with IAE #48816 (issue: #48810)
  • Introduce dedicated ingest processor exception #48810 (issue: #48803)

Features/Java High Level REST Client

  • Support es7 node http publish_address format #49279 (issue: #48950)
  • Add slices to delete and update by query in HLRC #48420
  • fix incorrect comparison #48208
  • Fix HLRC parsing of CancelTasks response #47017
  • Prevent deadlock by using separate schedulers #48697 (issues: #41451, #47599)

Features/Java Low Level REST Client

  • Improve warning value extraction performance in Response #50208 (issue: #24114)

Features/Monitoring

Geo

  • Guard against null geoBoundingBox #50506 (issue: #50505)
  • Geo: Switch generated GeoJson type names to camel case (#50285) #50400 (issue: #49568)
  • Geo: Switch generated WKT to upper case #50285 (issue: #49568)
  • Fix typo when assigning null_value in GeoPointFieldMapper #49645
  • Fix handling of circles in legacy geo_shape queries #49410 (issue: #49296)
  • GEO: intersects search for geo_shape return wrong result #49017
  • Geo: improve handling of out of bounds points in linestrings #47939 (issue: #43916)

Highlighting

  • Fix invalid break iterator highlighting on keyword field #49566

Infra/Core

  • Ignore virtual ethernet devices that disappear #51581 (issue: #49914)
  • Guess root cause support unwrap #50525 (issue: #50417)
  • Allow parsing timezone without fully provided time #50178 (issue: #49351)
  • [Java.time] Retain prefixed date pattern in formatter #48703 (issue: #48698)
  • Don’t drop user’s MaxDirectMemorySize flag on jdk8/windows #48657 (issues: #44174, #48365)
  • Warn when MaxDirectMemorySize may be incorrect (Windows/JDK8 only issue) #48365 (issue: #47384)
  • [Java.time] Calculate week of a year with ISO rules #48209 (issues: #41670, #42588, #43275, #43652)

Infra/Logging

  • Slow log must use separate underlying logger for each index #47234 (issue: #42432)

Infra/Packaging

  • Extend systemd timeout during startup #49784 (issue: #49593)

Infra/REST API

  • Return 400 when handling invalid JSON #49552 (issue: #49428)
  • Slash missed in indices.put_mapping url #49468

Machine Learning

  • Fix 2 digit year regex in find_file_structure #51469
  • Validate classification dependent_variable cardinality is at least two #51232
  • Do not copy mapping from dependent variable to prediction field in regression analysis #51227
  • Handle nested and aliased fields correctly when copying mapping #50918 (issue: #50787)
  • Fix off-by-one error in ml_classic tokenizer end offset #50655
  • Improve uniqueness of result document IDs #50644 (issue: #50613)
  • Fix accuracy metric in multi-class confusion matrix #50310 (issue: #48759)
  • Fix race condition when stopping a data frame analytics jobs immediately after starting it #50276 (issues: #49680, #50177)
  • Apply source query on data frame analytics memory estimation #49517 (issue: #49454)
  • Fix r_squared eval when variance is 0 #49439
  • Blacklist a number of prediction field names #49371 (issue: #48808)
  • Make data frame analytics more robust for very short-lived analyses #49282 (issue: #49095)
  • Fixes potential memory corruption when determining seasonality #852
  • Prevent prediction_field_name clashing with other fields in machine learning results #861
  • Include out-of-order as well as in-order terms in categorization reverse searches #950 (issue: #949)

Mapping

  • Ensure that field collapsing works with field aliases. #50722 (issues: #32648, #50121)
  • Improve DateFieldMapper ignore_malformed handling #50090 (issues: #46675, #50081)
  • Annotated text type should extend TextFieldType #49555 (issue: #49289)
  • Ensure parameters are updated when merging flattened mappings. #48971 (issue: #48907)

Network

  • Fix TransportMasterNodeAction not Retrying NodeClosedException #51325

Percolator

  • Correctly handle MSM for nested disjunctions #50669 (issue: #50305)
  • Fix query analyzer logic for mixed conjunctions of terms and ranges #49803 (issue: #49684)

Recovery

  • Check allocation id when failing shard on recovery #50656 (issue: #50508)
  • Migrate peer recovery from translog to retention lease #49448 (issue: #45136)
  • Ignore Lucene index in peer recovery if translog corrupted #49114

Reindex

SQL

Search

  • Fix upgrade of custom similarity #50851 (issue: #50763)
  • Fix NPE bug inner_hits #50709 (issue: #50539)
  • Collect results in a local list when notifying partial results #49828 (issue: #49778)
  • Fixes a bug in interval filter serialization #49793 (issue: #49519)
  • Correctly handle duplicates in unordered interval matching #49775
  • Correct rewritting of script_score query #48425 (issue: #48081)
  • Do not throw errors on unknown types in SearchAfterBuilder #48147 (issue: #48074)

Security

  • Always consume the body in has privileges #50298 (issue: #50288)

Snapshot/Restore

  • Fix Overly Aggressive Request DeDuplication #51270 (issue: #51253)
  • Guard Repository#getRepositoryData for exception throw #50970
  • Fix Index Deletion During Partial Snapshot Create #50234 (issues: #50200, #50202)
  • Fix Index Deletion during Snapshot Finalization #50202 (issues: #45689, #50200)
  • Fix RepoCleanup not Removed on Master-Failover #49217
  • Make FsBlobContainer Listing Resilient to Concurrent Modifications #49142 (issue: #37581)
  • Fix SnapshotShardStatus Reporting for Failed Shard #48556 (issue: #48526)
  • Cleanup Concurrent RepositoryData Loading #48329 (issue: #48122)

Transform

  • Fix mapping deduction for scaled_float #51990 (issue: #51780)
  • Fix stats can return old state information if security is enabled #51732 (issue: #51728)
  • Fail to start/put on missing pipeline #50701 (issue: #50135)
  • Fix possible audit logging disappearance after rolling upgrade #49731 (issue: #49730)
  • Do not fail checkpoint creation due to global checkpoint mismatch #48423 (issue: #48379)

Upgrades

Engine

  • Upgrade to Lucene 8.4.0. #50518

Infra/Packaging

  • Upgrade the bundled JDK to JDK 13.0.2 #51511