7.5. Elasticsearch Connector

Overview

The Elasticsearch Connector allows access to Elasticsearch data from Presto. This document describes how to setup the Elasticsearch Connector to run SQL queries against Elasticsearch.

Note

Elasticsearch 6.0.0 or later is required.

Configuration

To configure the Elasticsearch connector, create a catalog properties file etc/catalog/elasticsearch.properties with the following contents, replacing the properties as appropriate:

  1. connector.name=elasticsearch
  2. elasticsearch.host=localhost
  3. elasticsearch.port=9200
  4. elasticsearch.default-schema=default

Configuration Properties

The following configuration properties are available:

Property NameDescription
elasticsearch.hostHost name of the Elasticsearch server.
elasticsearch.portPort of the Elasticsearch server.
elasticsearch.default-schemaDefault schema name for tables.
elasticsearch.scroll-sizeMaximum number of hits to be returned with each Elasticsearch scroll request.
elasticsearch.scroll-timeoutAmount of time Elasticsearch will keep the search context alive for scroll requests.
elasticsearch.max-hitsMaximum number of hits a single Elasticsearch request can fetch.
elasticsearch.request-timeoutTimeout for Elasticsearch requests.
elasticsearch.connect-timeoutTimeout for connections to Elasticsearch hosts.
elasticsearch.max-retry-timeMaximum duration across all retry attempts for a single request.
elasticsearch.node-refresh-intervalHow often to refresh the list of available Elasticsearch nodes.

elasticsearch.host

Specifies the hostname of the Elasticsearch node to connect to.

This property is required.

elasticsearch.port

Specifies the port of the Elasticsearch node to connect to.

This property is optional; the default is 9200.

elasticsearch.default-schema

Defines the schema that will contain all tables defined without a qualifying schema name.

This property is optional; the default is default.

elasticsearch.scroll-size

This property defines the maximum number of hits that can be returned with each Elasticsearch scroll request.

This property is optional; the default is 1000.

elasticsearch.scroll-timeout

This property defines the amount of time Elasticsearch will keep the search context alive for scroll requests

This property is optional; the default is 1m.

elasticsearch.max-hits

This property defines the maximum number of hits an Elasticsearch request can fetch.

This property is optional; the default is 1000000.

elasticsearch.request-timeout

This property defines the timeout value for all Elasticsearch requests.

This property is optional; the default is 10s.

elasticsearch.connect-timeout

This property defines the timeout value for all Elasticsearch connection attempts.

This property is optional; the default is 1s.

elasticsearch.max-retry-time

This property defines the maximum duration across all retry attempts for a single request to Elasticsearch.

This property is optional; the default is 20s.

elasticsearch.node-refresh-interval

This property controls how often the list of available Elasticsearch nodes is refreshed.

This property is optional; the default is 1m.

TLS Security

The Elasticsearch connector provides additional security options to support Elasticsearch clusters that have been configured to use TLS.

The connector supports key stores and trust stores in PEM or Java Key Store (JKS) format. The allowed configuration values are:

Property NameDescription
elasticsearch.tls.enabledWhether TLS security is enabled.
elasticsearch.tls.verify-hostnamesWhether to verify Elasticsearch server hostnames.
elasticsearch.tls.keystore-pathPath to the PEM or JKS key store.
elasticsearch.tls.truststore-pathPath to the PEM or JKS trust store.
elasticsearch.tls.keystore-passwordPassword for the key store.
elasticsearch.tls.truststore-passwordPassword for the trust store.

elasticsearch.tls.keystore-path

The path to the PEM or JKS key store. This file must be readable by the operating system user running Presto.

This property is optional.

elasticsearch.tls.truststore-path

The path to PEM or JKS trust store. This file must be readable by the operating system user running Presto.

This property is optional.

elasticsearch.tls.keystore-password

The key password for the key store specified by elasticsearch.tls.keystore-path.

This property is optional.

elasticsearch.tls.truststore-password

The key password for the trust store specified by elasticsearch.tls.truststore-path.

This property is optional.

Data Types

The data type mappings are as follows:

ElasticsearchPresto
binaryVARBINARY
booleanBOOLEAN
doubleDOUBLE
floatREAL
byteTINYINT
shortSMALLINT
integerINTEGER
keywordVARCHAR
longBIGINT
textVARCHAR
dateTIMESTAMP
(others)(unsupported)

Array Types

Fields in Elasticsearch can contain zero or more values , but there is no dedicated array type. To indicate a field contains an array, it can be annotated in a Presto-specific structure in the _meta section of the index mapping.

For example, you can have an Elasticsearch index that contains documents with the following structure:

  1. {
  2. "array_string_field": ["presto","is","the","besto"],
  3. "long_field": 314159265359,
  4. "id_field": "564e6982-88ee-4498-aa98-df9e3f6b6109",
  5. "timestamp_field": "1987-09-17T06:22:48.000Z",
  6. "object_field": {
  7. "array_int_field": [86,75,309],
  8. "int_field": 2
  9. }
  10. }

The array fields of this structure can be defined by using the following command to add the field property definition to the _meta.presto property of the target index mapping.

  1. curl --request PUT \
  2. --url localhost:9200/doc/_mapping \
  3. --header 'content-type: application/json' \
  4. --data '
  5. {
  6. "_meta": {
  7. "presto":{
  8. "array_string_field":{
  9. "isArray":true
  10. },
  11. "object_field":{
  12. "array_int_field":{
  13. "isArray":true
  14. }
  15. },
  16. }
  17. }
  18. }'

Special Columns

The following hidden columns are available:

ColumnDescription
_idThe Elasticsearch document ID
_scoreThe document score returned by the Elasticsearch query
_sourceThe source of the original document

Full Text Queries

Presto SQL queries can be combined with Elasticsearch queries by providing the full text query as part of the table name, separated by a colon. For example:

  1. SELECT * FROM "tweets: +presto DB^2"

AWS Authorization

To enable AWS authorization using IAM policies, the elasticsearch.security option needs to be set to AWS. Additionally, the following options need to be configured appropriately:

Property NameDescription
elasticsearch.aws.regionAWS region or the Elasticsearch endpoint. This option is required.
elasticsearch.aws.access-keyAWS access key to use to connect to the Elasticsearch domain.
elasticsearch.aws.secret-keyAWS secret key to use to connect to the Elasticsearch domain.
elasticsearch.aws.use-instance-credentialsUse the EC2 metadata service to retrieve API credentials.