Thrift Connector

The Thrift connector makes it possible to integrate with external storage systems without a custom Presto connector implementation.

In order to use the Thrift connector with an external system, you need to implement the PrestoThriftService interface, found below. Next, you configure the Thrift connector to point to a set of machines, called Thrift servers, that implement the interface. As part of the interface implementation, the Thrift servers will provide metadata, splits and data. The connector will randomly choose a server to talk to from the available instances for metadata calls, or for data calls unless the splits include a list of addresses. All requests are assumed to be idempotent and can be retried freely among any server.

Configuration

To configure the Thrift connector, create a catalog properties file etc/catalog/thrift.properties with the following content, replacing the properties as appropriate:

  1. connector.name=presto-thrift
  2. presto.thrift.client.addresses=host:port,host:port

Multiple Thrift Systems

You can have as many catalogs as you need, so if you have additional Thrift systems to connect to, simply add another properties file to etc/catalog with a different name (making sure it ends in .properties).

Configuration Properties

The following configuration properties are available:

Property Name

Description

presto.thrift.client.addresses

Location of Thrift servers

presto-thrift.max-response-size

Maximum size of data returned from Thrift server

presto-thrift.metadata-refresh-threads

Number of refresh threads for metadata cache

presto.thrift.client.max-retries

Maximum number of retries for failed Thrift requests

presto.thrift.client.max-backoff-delay

Maximum interval between retry attempts

presto.thrift.client.min-backoff-delay

Minimum interval between retry attempts

presto.thrift.client.max-retry-time

Maximum duration across all attempts of a Thrift request

presto.thrift.client.backoff-scale-factor

Scale factor for exponential back off

presto.thrift.client.connect-timeout

Connect timeout

presto.thrift.client.request-timeout

Request timeout

presto.thrift.client.socks-proxy

SOCKS proxy address

presto.thrift.client.max-frame-size

Maximum size of a raw Thrift response

presto.thrift.client.transport

Thrift transport type (UNFRAMED, FRAMED, HEADER)

presto.thrift.client.protocol

Thrift protocol type (BINARY, COMPACT, FB_COMPACT)

presto.thrift.client.addresses

Comma-separated list of thrift servers in the form of host:port. For example:

  1. presto.thrift.client.addresses=192.0.2.3:7777,192.0.2.4:7779

This property is required; there is no default.

presto-thrift.max-response-size

Maximum size of a data response that the connector accepts. This value is sent by the connector to the Thrift server when requesting data, allowing it to size the response appropriately.

This property is optional; the default is 16MB.

presto-thrift.metadata-refresh-threads

Number of refresh threads for metadata cache.

This property is optional; the default is 1.

Thrift IDL File

The following IDL describes the PrestoThriftService that must be implemented:

  1. enum PrestoThriftBound {
  2. BELOW = 1;
  3. EXACTLY = 2;
  4. ABOVE = 3;
  5. }
  6. exception PrestoThriftServiceException {
  7. 1: string message;
  8. 2: bool retryable;
  9. }
  10. struct PrestoThriftNullableSchemaName {
  11. 1: optional string schemaName;
  12. }
  13. struct PrestoThriftSchemaTableName {
  14. 1: string schemaName;
  15. 2: string tableName;
  16. }
  17. struct PrestoThriftTableMetadata {
  18. 1: PrestoThriftSchemaTableName schemaTableName;
  19. 2: list<PrestoThriftColumnMetadata> columns;
  20. 3: optional string comment;
  21. /**
  22. * Returns a list of key sets which can be used for index lookups.
  23. * The list is expected to have only unique key sets.
  24. * {@code set<set<string>>} is not used here because some languages (like php) don't support it.
  25. */
  26. 4: optional list<set<string>> indexableKeys;
  27. }
  28. struct PrestoThriftColumnMetadata {
  29. 1: string name;
  30. 2: string type;
  31. 3: optional string comment;
  32. 4: bool hidden;
  33. }
  34. struct PrestoThriftNullableColumnSet {
  35. 1: optional set<string> columns;
  36. }
  37. struct PrestoThriftTupleDomain {
  38. /**
  39. * Return a map of column names to constraints.
  40. */
  41. 1: optional map<string, PrestoThriftDomain> domains;
  42. }
  43. /**
  44. * Set that either includes all values, or excludes all values.
  45. */
  46. struct PrestoThriftAllOrNoneValueSet {
  47. 1: bool all;
  48. }
  49. /**
  50. * A set containing values that are uniquely identifiable.
  51. * Assumes an infinite number of possible values. The values may be collectively included (aka whitelist)
  52. * or collectively excluded (aka !whitelist).
  53. * This structure is used with comparable, but not orderable types like "json", "map".
  54. */
  55. struct PrestoThriftEquatableValueSet {
  56. 1: bool whiteList;
  57. 2: list<PrestoThriftBlock> values;
  58. }
  59. /**
  60. * Elements of {@code nulls} array determine if a value for a corresponding row is null.
  61. * Elements of {@code ints} array are values for each row. If row is null then value is ignored.
  62. */
  63. struct PrestoThriftInteger {
  64. 1: optional list<bool> nulls;
  65. 2: optional list<i32> ints;
  66. }
  67. /**
  68. * Elements of {@code nulls} array determine if a value for a corresponding row is null.
  69. * Elements of {@code longs} array are values for each row. If row is null then value is ignored.
  70. */
  71. struct PrestoThriftBigint {
  72. 1: optional list<bool> nulls;
  73. 2: optional list<i64> longs;
  74. }
  75. /**
  76. * Elements of {@code nulls} array determine if a value for a corresponding row is null.
  77. * Elements of {@code doubles} array are values for each row. If row is null then value is ignored.
  78. */
  79. struct PrestoThriftDouble {
  80. 1: optional list<bool> nulls;
  81. 2: optional list<double> doubles;
  82. }
  83. /**
  84. * Elements of {@code nulls} array determine if a value for a corresponding row is null.
  85. * Each elements of {@code sizes} array contains the length in bytes for the corresponding element.
  86. * If row is null then the corresponding element in {@code sizes} is ignored.
  87. * {@code bytes} array contains UTF-8 encoded byte values.
  88. * Values for all rows are written to {@code bytes} array one after another.
  89. * The total number of bytes must be equal to the sum of all sizes.
  90. */
  91. struct PrestoThriftVarchar {
  92. 1: optional list<bool> nulls;
  93. 2: optional list<i32> sizes;
  94. 3: optional binary bytes;
  95. }
  96. /**
  97. * Elements of {@code nulls} array determine if a value for a corresponding row is null.
  98. * Elements of {@code booleans} array are values for each row. If row is null then value is ignored.
  99. */
  100. struct PrestoThriftBoolean {
  101. 1: optional list<bool> nulls;
  102. 2: optional list<bool> booleans;
  103. }
  104. /**
  105. * Elements of {@code nulls} array determine if a value for a corresponding row is null.
  106. * Elements of {@code dates} array are date values for each row represented as the number
  107. * of days passed since 1970-01-01.
  108. * If row is null then value is ignored.
  109. */
  110. struct PrestoThriftDate {
  111. 1: optional list<bool> nulls;
  112. 2: optional list<i32> dates;
  113. }
  114. /**
  115. * Elements of {@code nulls} array determine if a value for a corresponding row is null.
  116. * Elements of {@code timestamps} array are values for each row represented as the number
  117. * of milliseconds passed since 1970-01-01T00:00:00 UTC.
  118. * If row is null then value is ignored.
  119. */
  120. struct PrestoThriftTimestamp {
  121. 1: optional list<bool> nulls;
  122. 2: optional list<i64> timestamps;
  123. }
  124. /**
  125. * Elements of {@code nulls} array determine if a value for a corresponding row is null.
  126. * Each elements of {@code sizes} array contains the length in bytes for the corresponding element.
  127. * If row is null then the corresponding element in {@code sizes} is ignored.
  128. * {@code bytes} array contains UTF-8 encoded byte values for string representation of json.
  129. * Values for all rows are written to {@code bytes} array one after another.
  130. * The total number of bytes must be equal to the sum of all sizes.
  131. */
  132. struct PrestoThriftJson {
  133. 1: optional list<bool> nulls;
  134. 2: optional list<i32> sizes;
  135. 3: optional binary bytes;
  136. }
  137. /**
  138. * Elements of {@code nulls} array determine if a value for a corresponding row is null.
  139. * Each elements of {@code sizes} array contains the length in bytes for the corresponding element.
  140. * If row is null then the corresponding element in {@code sizes} is ignored.
  141. * {@code bytes} array contains encoded byte values for HyperLogLog representation as defined in
  142. * Airlift specification: href="https://github.com/airlift/airlift/blob/master/stats/docs/hll.md
  143. * Values for all rows are written to {@code bytes} array one after another.
  144. * The total number of bytes must be equal to the sum of all sizes.
  145. */
  146. struct PrestoThriftHyperLogLog {
  147. 1: optional list<bool> nulls;
  148. 2: optional list<i32> sizes;
  149. 3: optional binary bytes;
  150. }
  151. /**
  152. * Elements of {@code nulls} array determine if a value for a corresponding row is null.
  153. * Each elements of {@code sizes} array contains the number of elements in the corresponding values array.
  154. * If row is null then the corresponding element in {@code sizes} is ignored.
  155. * {@code values} is a bigint block containing array elements one after another for all rows.
  156. * The total number of elements in bigint block must be equal to the sum of all sizes.
  157. */
  158. struct PrestoThriftBigintArray {
  159. 1: optional list<bool> nulls;
  160. 2: optional list<i32> sizes;
  161. 3: optional PrestoThriftBigint values;
  162. }
  163. /**
  164. * A set containing zero or more Ranges of the same type over a continuous space of possible values.
  165. * Ranges are coalesced into the most compact representation of non-overlapping Ranges.
  166. * This structure is used with comparable and orderable types like bigint, integer, double, varchar, etc.
  167. */
  168. struct PrestoThriftRangeValueSet {
  169. 1: list<PrestoThriftRange> ranges;
  170. }
  171. struct PrestoThriftId {
  172. 1: binary id;
  173. }
  174. struct PrestoThriftSplitBatch {
  175. 1: list<PrestoThriftSplit> splits;
  176. 2: optional PrestoThriftId nextToken;
  177. }
  178. struct PrestoThriftSplit {
  179. /**
  180. * Encodes all the information needed to identify a batch of rows to return to Presto.
  181. * For a basic scan, includes schema name, table name, and output constraint.
  182. * For an index scan, includes schema name, table name, set of keys to lookup and output constraint.
  183. */
  184. 1: PrestoThriftId splitId;
  185. /**
  186. * Identifies the set of hosts on which the rows are available. If empty, then the rows
  187. * are expected to be available on any host. The hosts in this list may be independent
  188. * from the hosts used to serve metadata requests.
  189. */
  190. 2: list<PrestoThriftHostAddress> hosts;
  191. }
  192. struct PrestoThriftHostAddress {
  193. 1: string host;
  194. 2: i32 port;
  195. }
  196. struct PrestoThriftPageResult {
  197. /**
  198. * Returns data in a columnar format.
  199. * Columns in this list must be in the order they were requested by the engine.
  200. */
  201. 1: list<PrestoThriftBlock> columnBlocks;
  202. 2: i32 rowCount;
  203. 3: optional PrestoThriftId nextToken;
  204. }
  205. struct PrestoThriftNullableTableMetadata {
  206. 1: optional PrestoThriftTableMetadata tableMetadata;
  207. }
  208. struct PrestoThriftValueSet {
  209. 1: optional PrestoThriftAllOrNoneValueSet allOrNoneValueSet;
  210. 2: optional PrestoThriftEquatableValueSet equatableValueSet;
  211. 3: optional PrestoThriftRangeValueSet rangeValueSet;
  212. }
  213. struct PrestoThriftBlock {
  214. 1: optional PrestoThriftInteger integerData;
  215. 2: optional PrestoThriftBigint bigintData;
  216. 3: optional PrestoThriftDouble doubleData;
  217. 4: optional PrestoThriftVarchar varcharData;
  218. 5: optional PrestoThriftBoolean booleanData;
  219. 6: optional PrestoThriftDate dateData;
  220. 7: optional PrestoThriftTimestamp timestampData;
  221. 8: optional PrestoThriftJson jsonData;
  222. 9: optional PrestoThriftHyperLogLog hyperLogLogData;
  223. 10: optional PrestoThriftBigintArray bigintArrayData;
  224. }
  225. /**
  226. * LOWER UNBOUNDED is specified with an empty value and an ABOVE bound
  227. * UPPER UNBOUNDED is specified with an empty value and a BELOW bound
  228. */
  229. struct PrestoThriftMarker {
  230. 1: optional PrestoThriftBlock value;
  231. 2: PrestoThriftBound bound;
  232. }
  233. struct PrestoThriftNullableToken {
  234. 1: optional PrestoThriftId token;
  235. }
  236. struct PrestoThriftDomain {
  237. 1: PrestoThriftValueSet valueSet;
  238. 2: bool nullAllowed;
  239. }
  240. struct PrestoThriftRange {
  241. 1: PrestoThriftMarker low;
  242. 2: PrestoThriftMarker high;
  243. }
  244. /**
  245. * Presto Thrift service definition.
  246. * This thrift service needs to be implemented in order to be used with Thrift Connector.
  247. */
  248. service PrestoThriftService {
  249. /**
  250. * Returns available schema names.
  251. */
  252. list<string> prestoListSchemaNames()
  253. throws (1: PrestoThriftServiceException ex1);
  254. /**
  255. * Returns tables for the given schema name.
  256. *
  257. * @param schemaNameOrNull a structure containing schema name or {@literal null}
  258. * @return a list of table names with corresponding schemas. If schema name is null then returns
  259. * a list of tables for all schemas. Returns an empty list if a schema does not exist
  260. */
  261. list<PrestoThriftSchemaTableName> prestoListTables(
  262. 1: PrestoThriftNullableSchemaName schemaNameOrNull)
  263. throws (1: PrestoThriftServiceException ex1);
  264. /**
  265. * Returns metadata for a given table.
  266. *
  267. * @param schemaTableName schema and table name
  268. * @return metadata for a given table, or a {@literal null} value inside if it does not exist
  269. */
  270. PrestoThriftNullableTableMetadata prestoGetTableMetadata(
  271. 1: PrestoThriftSchemaTableName schemaTableName)
  272. throws (1: PrestoThriftServiceException ex1);
  273. /**
  274. * Returns a batch of splits.
  275. *
  276. * @param schemaTableName schema and table name
  277. * @param desiredColumns a superset of columns to return; empty set means "no columns", {@literal null} set means "all columns"
  278. * @param outputConstraint constraint on the returned data
  279. * @param maxSplitCount maximum number of splits to return
  280. * @param nextToken token from a previous split batch or {@literal null} if it is the first call
  281. * @return a batch of splits
  282. */
  283. PrestoThriftSplitBatch prestoGetSplits(
  284. 1: PrestoThriftSchemaTableName schemaTableName,
  285. 2: PrestoThriftNullableColumnSet desiredColumns,
  286. 3: PrestoThriftTupleDomain outputConstraint,
  287. 4: i32 maxSplitCount,
  288. 5: PrestoThriftNullableToken nextToken)
  289. throws (1: PrestoThriftServiceException ex1);
  290. /**
  291. * Returns a batch of index splits for the given batch of keys.
  292. * This method is called if index join strategy is chosen for a query.
  293. *
  294. * @param schemaTableName schema and table name
  295. * @param indexColumnNames specifies columns and their order for keys
  296. * @param outputColumnNames a list of column names to return
  297. * @param keys keys for which records need to be returned; includes only unique and non-null values
  298. * @param outputConstraint constraint on the returned data
  299. * @param maxSplitCount maximum number of splits to return
  300. * @param nextToken token from a previous split batch or {@literal null} if it is the first call
  301. * @return a batch of splits
  302. */
  303. PrestoThriftSplitBatch prestoGetIndexSplits(
  304. 1: PrestoThriftSchemaTableName schemaTableName,
  305. 2: list<string> indexColumnNames,
  306. 3: list<string> outputColumnNames,
  307. 4: PrestoThriftPageResult keys,
  308. 5: PrestoThriftTupleDomain outputConstraint,
  309. 6: i32 maxSplitCount,
  310. 7: PrestoThriftNullableToken nextToken)
  311. throws (1: PrestoThriftServiceException ex1);
  312. /**
  313. * Returns a batch of rows for the given split.
  314. *
  315. * @param splitId split id as returned in split batch
  316. * @param columns a list of column names to return
  317. * @param maxBytes maximum size of returned data in bytes
  318. * @param nextToken token from a previous batch or {@literal null} if it is the first call
  319. * @return a batch of table data
  320. */
  321. PrestoThriftPageResult prestoGetRows(
  322. 1: PrestoThriftId splitId,
  323. 2: list<string> columns,
  324. 3: i64 maxBytes,
  325. 4: PrestoThriftNullableToken nextToken)
  326. throws (1: PrestoThriftServiceException ex1);
  327. }