Queries

SELECT statements and VALUES statements are specified with the sqlQuery() method of the TableEnvironment. The method returns the result of the SELECT statement (or the VALUES statements) as a Table. A Table can be used in subsequent SQL and Table API queries, be converted into a DataSet or DataStream, or written to a TableSink. SQL and Table API queries can be seamlessly mixed and are holistically optimized and translated into a single program.

In order to access a table in a SQL query, it must be registered in the TableEnvironment. A table can be registered from a TableSource, Table, CREATE TABLE statement, DataStream, or DataSet. Alternatively, users can also register catalogs in a TableEnvironment to specify the location of the data sources.

For convenience, Table.toString() automatically registers the table under a unique name in its TableEnvironment and returns the name. So, Table objects can be directly inlined into SQL queries as shown in the examples below.

Note: Queries that include unsupported SQL features cause a TableException. The supported features of SQL on batch and streaming tables are listed in the following sections.

SELECT statements and VALUES statements are specified with the sql_query() method of the TableEnvironment. The method returns the result of the SELECT statement (or the VALUES statements) as a Table. A Table can be used in subsequent SQL and Table API queries or written to a TableSink. SQL and Table API queries can be seamlessly mixed and are holistically optimized and translated into a single program.

In order to access a table in a SQL query, it must be registered in the TableEnvironment. A table can be registered from a TableSource, Table, CREATE TABLE statement. Alternatively, users can also register catalogs in a TableEnvironment to specify the location of the data sources.

For convenience, str(Table) automatically registers the table under a unique name in its TableEnvironment and returns the name. So, Table objects can be directly inlined into SQL queries as shown in the examples below.

Note: Queries that include unsupported SQL features cause a TableException. The supported features of SQL on batch and streaming tables are listed in the following sections.

Specifying a Query

The following examples show how to specify a SQL queries on registered and inlined tables.

  1. StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
  2. StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env);
  3. // ingest a DataStream from an external source
  4. DataStream<Tuple3<Long, String, Integer>> ds = env.addSource(...);
  5. // SQL query with an inlined (unregistered) table
  6. Table table = tableEnv.fromDataStream(ds, $("user"), $("product"), $("amount"));
  7. Table result = tableEnv.sqlQuery(
  8. "SELECT SUM(amount) FROM " + table + " WHERE product LIKE '%Rubber%'");
  9. // SQL query with a registered table
  10. // register the DataStream as view "Orders"
  11. tableEnv.createTemporaryView("Orders", ds, $("user"), $("product"), $("amount"));
  12. // run a SQL query on the Table and retrieve the result as a new Table
  13. Table result2 = tableEnv.sqlQuery(
  14. "SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'");
  15. // create and register a TableSink
  16. final Schema schema = new Schema()
  17. .field("product", DataTypes.STRING())
  18. .field("amount", DataTypes.INT());
  19. tableEnv.connect(new FileSystem().path("/path/to/file"))
  20. .withFormat(...)
  21. .withSchema(schema)
  22. .createTemporaryTable("RubberOrders");
  23. // run an INSERT SQL on the Table and emit the result to the TableSink
  24. tableEnv.executeSql(
  25. "INSERT INTO RubberOrders SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'");
  1. val env = StreamExecutionEnvironment.getExecutionEnvironment
  2. val tableEnv = StreamTableEnvironment.create(env)
  3. // read a DataStream from an external source
  4. val ds: DataStream[(Long, String, Integer)] = env.addSource(...)
  5. // SQL query with an inlined (unregistered) table
  6. val table = ds.toTable(tableEnv, $"user", $"product", $"amount")
  7. val result = tableEnv.sqlQuery(
  8. s"SELECT SUM(amount) FROM $table WHERE product LIKE '%Rubber%'")
  9. // SQL query with a registered table
  10. // register the DataStream under the name "Orders"
  11. tableEnv.createTemporaryView("Orders", ds, $"user", $"product", $"amount")
  12. // run a SQL query on the Table and retrieve the result as a new Table
  13. val result2 = tableEnv.sqlQuery(
  14. "SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'")
  15. // create and register a TableSink
  16. val schema = new Schema()
  17. .field("product", DataTypes.STRING())
  18. .field("amount", DataTypes.INT())
  19. tableEnv.connect(new FileSystem().path("/path/to/file"))
  20. .withFormat(...)
  21. .withSchema(schema)
  22. .createTemporaryTable("RubberOrders")
  23. // run an INSERT SQL on the Table and emit the result to the TableSink
  24. tableEnv.executeSql(
  25. "INSERT INTO RubberOrders SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'")
  1. env = StreamExecutionEnvironment.get_execution_environment()
  2. table_env = StreamTableEnvironment.create(env)
  3. # SQL query with an inlined (unregistered) table
  4. # elements data type: BIGINT, STRING, BIGINT
  5. table = table_env.from_elements(..., ['user', 'product', 'amount'])
  6. result = table_env \
  7. .sql_query("SELECT SUM(amount) FROM %s WHERE product LIKE '%%Rubber%%'" % table)
  8. # create and register a TableSink
  9. t_env.connect(FileSystem().path("/path/to/file")))
  10. .with_format(Csv()
  11. .field_delimiter(',')
  12. .deriveSchema())
  13. .with_schema(Schema()
  14. .field("product", DataTypes.STRING())
  15. .field("amount", DataTypes.BIGINT()))
  16. .create_temporary_table("RubberOrders")
  17. # run an INSERT SQL on the Table and emit the result to the TableSink
  18. table_env \
  19. .execute_sql("INSERT INTO RubberOrders SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'")

Execute a Query

A SELECT statement or a VALUES statement can be executed to collect the content to local through the TableEnvironment.executeSql() method. The method returns the result of the SELECT statement (or the VALUES statement) as a TableResult. Similar to a SELECT statement, a Table object can be executed using the Table.execute() method to collect the content of the query to the local client. TableResult.collect() method returns a closeable row iterator. The select job will not be finished unless all result data has been collected. We should actively close the job to avoid resource leak through the CloseableIterator#close() method. We can also print the select result to client console through the TableResult.print() method. The result data in TableResult can be accessed only once. Thus, collect() and print() must not be called after each other.

TableResult.collect() and TableResult.print() have slightly different behaviors under different checkpointing settings (to enable checkpointing for a streaming job, see checkpointing config).

  • For batch jobs or streaming jobs without checkpointing, TableResult.collect() and TableResult.print() have neither exactly-once nor at-least-once guarantee. Query results are immediately accessible by the clients once they’re produced, but exceptions will be thrown when the job fails and restarts.
  • For streaming jobs with exactly-once checkpointing, TableResult.collect() and TableResult.print() guarantee an end-to-end exactly-once record delivery. A result will be accessible by clients only after its corresponding checkpoint completes.
  • For streaming jobs with at-least-once checkpointing, TableResult.collect() and TableResult.print() guarantee an end-to-end at-least-once record delivery. Query results are immediately accessible by the clients once they’re produced, but it is possible for the same result to be delivered multiple times.

A SELECT statement or a VALUES statement can be executed to collect the content to local through the TableEnvironment.execute_sql() method. The method returns the result of the SELECT statement (or the VALUES statement) as a TableResult. Similar to a SELECT statement, a Table object can be executed using the Table.execute() method to collect the content of the query to the local client. TableResult.collect() method returns a closeable row iterator. The select job will not be finished unless all result data has been collected. We should actively close the job to avoid resource leak through the CloseableIterator#close() method. We can also print the select result to client console through the TableResult.print() method. The result data in TableResult can be accessed only once. Thus, collect() and print() must not be called after each other.

TableResult.collect() and TableResult.print() have slightly different behaviors under different checkpointing settings (to enable checkpointing for a streaming job, see checkpointing config).

  • For batch jobs or streaming jobs without checkpointing, TableResult.collect() and TableResult.print() have neither exactly-once nor at-least-once guarantee. Query results are immediately accessible by the clients once they’re produced, but exceptions will be thrown when the job fails and restarts.
  • For streaming jobs with exactly-once checkpointing, TableResult.collect() and TableResult.print() guarantee an end-to-end exactly-once record delivery. A result will be accessible by clients only after its corresponding checkpoint completes.
  • For streaming jobs with at-least-once checkpointing, TableResult.collect() and TableResult.print() guarantee an end-to-end at-least-once record delivery. Query results are immediately accessible by the clients once they’re produced, but it is possible for the same result to be delivered multiple times.
  1. StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
  2. StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env, settings);
  3. tableEnv.executeSql("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)");
  4. // execute SELECT statement
  5. TableResult tableResult1 = tableEnv.executeSql("SELECT * FROM Orders");
  6. // use try-with-resources statement to make sure the iterator will be closed automatically
  7. try (CloseableIterator<Row> it = tableResult1.collect()) {
  8. while(it.hasNext()) {
  9. Row row = it.next();
  10. // handle row
  11. }
  12. }
  13. // execute Table
  14. TableResult tableResult2 = tableEnv.sqlQuery("SELECT * FROM Orders").execute();
  15. tableResult2.print();
  1. val env = StreamExecutionEnvironment.getExecutionEnvironment()
  2. val tableEnv = StreamTableEnvironment.create(env, settings)
  3. // enable checkpointing
  4. tableEnv.getConfig.getConfiguration.set(
  5. ExecutionCheckpointingOptions.CHECKPOINTING_MODE, CheckpointingMode.EXACTLY_ONCE)
  6. tableEnv.getConfig.getConfiguration.set(
  7. ExecutionCheckpointingOptions.CHECKPOINTING_INTERVAL, Duration.ofSeconds(10))
  8. tableEnv.executeSql("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)")
  9. // execute SELECT statement
  10. val tableResult1 = tableEnv.executeSql("SELECT * FROM Orders")
  11. val it = tableResult1.collect()
  12. try while (it.hasNext) {
  13. val row = it.next
  14. // handle row
  15. }
  16. finally it.close() // close the iterator to avoid resource leak
  17. // execute Table
  18. val tableResult2 = tableEnv.sqlQuery("SELECT * FROM Orders").execute()
  19. tableResult2.print()
  1. env = StreamExecutionEnvironment.get_execution_environment()
  2. table_env = StreamTableEnvironment.create(env, settings)
  3. # enable checkpointing
  4. table_env.get_config().get_configuration().set_string("execution.checkpointing.mode", "EXACTLY_ONCE")
  5. table_env.get_config().get_configuration().set_string("execution.checkpointing.interval", "10s")
  6. table_env.execute_sql("CREATE TABLE Orders (`user` BIGINT, product STRING, amount INT) WITH (...)")
  7. # execute SELECT statement
  8. table_result1 = table_env.execute_sql("SELECT * FROM Orders")
  9. table_result1.print()
  10. # execute Table
  11. table_result2 = table_env.sql_query("SELECT * FROM Orders").execute()
  12. table_result2.print()

Syntax

Flink parses SQL using Apache Calcite, which supports standard ANSI SQL.

The following BNF-grammar describes the superset of supported SQL features in batch and streaming queries. The Operations section shows examples for the supported features and indicates which features are only supported for batch or streaming queries.

  1. query:
  2. values
  3. | {
  4. select
  5. | selectWithoutFrom
  6. | query UNION [ ALL ] query
  7. | query EXCEPT query
  8. | query INTERSECT query
  9. }
  10. [ ORDER BY orderItem [, orderItem ]* ]
  11. [ LIMIT { count | ALL } ]
  12. [ OFFSET start { ROW | ROWS } ]
  13. [ FETCH { FIRST | NEXT } [ count ] { ROW | ROWS } ONLY]
  14. orderItem:
  15. expression [ ASC | DESC ]
  16. select:
  17. SELECT [ ALL | DISTINCT ]
  18. { * | projectItem [, projectItem ]* }
  19. FROM tableExpression
  20. [ WHERE booleanExpression ]
  21. [ GROUP BY { groupItem [, groupItem ]* } ]
  22. [ HAVING booleanExpression ]
  23. [ WINDOW windowName AS windowSpec [, windowName AS windowSpec ]* ]
  24. selectWithoutFrom:
  25. SELECT [ ALL | DISTINCT ]
  26. { * | projectItem [, projectItem ]* }
  27. projectItem:
  28. expression [ [ AS ] columnAlias ]
  29. | tableAlias . *
  30. tableExpression:
  31. tableReference [, tableReference ]*
  32. | tableExpression [ NATURAL ] [ LEFT | RIGHT | FULL ] JOIN tableExpression [ joinCondition ]
  33. joinCondition:
  34. ON booleanExpression
  35. | USING '(' column [, column ]* ')'
  36. tableReference:
  37. tablePrimary
  38. [ matchRecognize ]
  39. [ [ AS ] alias [ '(' columnAlias [, columnAlias ]* ')' ] ]
  40. tablePrimary:
  41. [ TABLE ] tablePath [ dynamicTableOptions ] [systemTimePeriod] [[AS] correlationName]
  42. | LATERAL TABLE '(' functionName '(' expression [, expression ]* ')' ')'
  43. | UNNEST '(' expression ')'
  44. tablePath:
  45. [ [ catalogName . ] schemaName . ] tableName
  46. systemTimePeriod:
  47. FOR SYSTEM_TIME AS OF dateTimeExpression
  48. dynamicTableOptions:
  49. /*+ OPTIONS(key=val [, key=val]*) */
  50. key:
  51. stringLiteral
  52. val:
  53. stringLiteral
  54. values:
  55. VALUES expression [, expression ]*
  56. groupItem:
  57. expression
  58. | '(' ')'
  59. | '(' expression [, expression ]* ')'
  60. | CUBE '(' expression [, expression ]* ')'
  61. | ROLLUP '(' expression [, expression ]* ')'
  62. | GROUPING SETS '(' groupItem [, groupItem ]* ')'
  63. windowRef:
  64. windowName
  65. | windowSpec
  66. windowSpec:
  67. [ windowName ]
  68. '('
  69. [ ORDER BY orderItem [, orderItem ]* ]
  70. [ PARTITION BY expression [, expression ]* ]
  71. [
  72. RANGE numericOrIntervalExpression {PRECEDING}
  73. | ROWS numericExpression {PRECEDING}
  74. ]
  75. ')'
  76. matchRecognize:
  77. MATCH_RECOGNIZE '('
  78. [ PARTITION BY expression [, expression ]* ]
  79. [ ORDER BY orderItem [, orderItem ]* ]
  80. [ MEASURES measureColumn [, measureColumn ]* ]
  81. [ ONE ROW PER MATCH ]
  82. [ AFTER MATCH
  83. ( SKIP TO NEXT ROW
  84. | SKIP PAST LAST ROW
  85. | SKIP TO FIRST variable
  86. | SKIP TO LAST variable
  87. | SKIP TO variable )
  88. ]
  89. PATTERN '(' pattern ')'
  90. [ WITHIN intervalLiteral ]
  91. DEFINE variable AS condition [, variable AS condition ]*
  92. ')'
  93. measureColumn:
  94. expression AS alias
  95. pattern:
  96. patternTerm [ '|' patternTerm ]*
  97. patternTerm:
  98. patternFactor [ patternFactor ]*
  99. patternFactor:
  100. variable [ patternQuantifier ]
  101. patternQuantifier:
  102. '*'
  103. | '*?'
  104. | '+'
  105. | '+?'
  106. | '?'
  107. | '??'
  108. | '{' { [ minRepeat ], [ maxRepeat ] } '}' ['?']
  109. | '{' repeat '}'

Flink SQL uses a lexical policy for identifier (table, attribute, function names) similar to Java:

  • The case of identifiers is preserved whether or not they are quoted.
  • After which, identifiers are matched case-sensitively.
  • Unlike Java, back-ticks allow identifiers to contain non-alphanumeric characters (e.g. "SELECT a AS `my field` FROM t").

String literals must be enclosed in single quotes (e.g., SELECT 'Hello World'). Duplicate a single quote for escaping (e.g., SELECT 'It''s me.'). Unicode characters are supported in string literals. If explicit unicode code points are required, use the following syntax:

  • Use the backslash (\) as escaping character (default): SELECT U&'\263A'
  • Use a custom escaping character: SELECT U&'#263A' UESCAPE '#'

Operations

Scan, Projection, and Filter

OperationDescription
Scan / Select / As
Batch Streaming
  1. SELECT FROM Orders
  2. SELECT a, c AS d FROM Orders
Where / Filter
Batch Streaming
  1. SELECT FROM Orders WHERE b = red
  2. SELECT * FROM Orders WHERE a % 2 = 0
User-defined Scalar Functions (Scalar UDF)
Batch Streaming

UDFs must be registered in the TableEnvironment. See the UDF documentation for details on how to specify and register scalar UDFs.

  1. SELECT PRETTY_PRINT(user) FROM Orders

Aggregations

OperationDescription
GroupBy Aggregation
Batch Streaming
Result Updating

Note: GroupBy on a streaming table produces an updating result. See the Dynamic Tables Streaming Concepts page for details.

  1. SELECT a, SUM(b) as d
  2. FROM Orders
  3. GROUP BY a
GroupBy Window Aggregation
Batch Streaming

Use a group window to compute a single result row per group. See Group Windows section for more details.

  1. SELECT user, SUM(amount)
  2. FROM Orders
  3. GROUP BY TUMBLE(rowtime, INTERVAL 1 DAY), user
Over Window aggregation
Streaming

Note: All aggregates must be defined over the same window, i.e., same partitioning, sorting, and range. Currently, only windows with PRECEDING (UNBOUNDED and bounded) to CURRENT ROW range are supported. Ranges with FOLLOWING are not supported yet. ORDER BY must be specified on a single time attribute

  1. SELECT COUNT(amount) OVER (
  2. PARTITION BY user
  3. ORDER BY proctime
  4. ROWS BETWEEN 2 PRECEDING AND CURRENT ROW)
  5. FROM Orders
  6. SELECT COUNT(amount) OVER w, SUM(amount) OVER w
  7. FROM Orders
  8. WINDOW w AS (
  9. PARTITION BY user
  10. ORDER BY proctime
  11. ROWS BETWEEN 2 PRECEDING AND CURRENT ROW)
Distinct
Batch Streaming
Result Updating
  1. SELECT DISTINCT users FROM Orders

Note: For streaming queries the required state to compute the query result might grow infinitely depending on the number of distinct fields. Please provide a query configuration with valid retention interval to prevent excessive state size. See Query Configuration for details.

Grouping sets, Rollup, Cube
Batch Streaming Result Updating
  1. SELECT SUM(amount)
  2. FROM Orders
  3. GROUP BY GROUPING SETS ((user), (product))

Note: Streaming mode Grouping sets, Rollup and Cube are only supported in Blink planner.

Having
Batch Streaming
  1. SELECT SUM(amount)
  2. FROM Orders
  3. GROUP BY users
  4. HAVING SUM(amount) > 50
User-defined Aggregate Functions (UDAGG)
Batch Streaming

UDAGGs must be registered in the TableEnvironment. See the UDF documentation for details on how to specify and register UDAGGs.

  1. SELECT MyAggregate(amount)
  2. FROM Orders
  3. GROUP BY users

Joins

OperationDescription
Inner Equi-join
Batch Streaming

Currently, only equi-joins are supported, i.e., joins that have at least one conjunctive condition with an equality predicate. Arbitrary cross or theta joins are not supported.

Note: The order of joins is not optimized. Tables are joined in the order in which they are specified in the FROM clause. Make sure to specify tables in an order that does not yield a cross join (Cartesian product) which are not supported and would cause a query to fail.

  1. SELECT
  2. FROM Orders INNER JOIN Product ON Orders.productId = Product.id

Note: For streaming queries the required state to compute the query result might grow infinitely depending on the number of distinct input rows. Please provide a query configuration with valid retention interval to prevent excessive state size. See Query Configuration for details.

Outer Equi-join
Batch Streaming Result Updating

Currently, only equi-joins are supported, i.e., joins that have at least one conjunctive condition with an equality predicate. Arbitrary cross or theta joins are not supported.

Note: The order of joins is not optimized. Tables are joined in the order in which they are specified in the FROM clause. Make sure to specify tables in an order that does not yield a cross join (Cartesian product) which are not supported and would cause a query to fail.

  1. SELECT
  2. FROM Orders LEFT JOIN Product ON Orders.productId = Product.id
  3. SELECT
  4. FROM Orders RIGHT JOIN Product ON Orders.productId = Product.id
  5. SELECT
  6. FROM Orders FULL OUTER JOIN Product ON Orders.productId = Product.id

Note: For streaming queries the required state to compute the query result might grow infinitely depending on the number of distinct input rows. Please provide a query configuration with valid retention interval to prevent excessive state size. See Query Configuration for details.

Inner/Outer Interval Join
Batch Streaming

Note: Interval joins are a subset of regular joins that can be processed in a streaming fashion. Both inner and outer joins are supported.

A interval join requires at least one equi-join predicate and a join condition that bounds the time on both sides. Such a condition can be defined by two appropriate range predicates (<, <=, >=, >), a BETWEEN predicate, or a single equality predicate that compares time attributes of the same type (i.e., processing time or event time) of both input tables.

For example, the following predicates are valid interval join conditions:

  • ltime = rtime
  • ltime >= rtime AND ltime < rtime + INTERVAL ‘10’ MINUTE
  • ltime BETWEEN rtime - INTERVAL ‘10’ SECOND AND rtime + INTERVAL ‘5’ SECOND
  1. SELECT
  2. FROM Orders o, Shipments s
  3. WHERE o.id = s.orderId AND
  4. o.ordertime BETWEEN s.shiptime - INTERVAL 4 HOUR AND s.shiptime
The example above will join all orders with their corresponding shipments if the order was shipped four hours after the order was received.
Expanding arrays into a relation
Batch Streaming

Unnesting WITH ORDINALITY is not supported yet.

  1. SELECT users, tag
  2. FROM Orders CROSS JOIN UNNEST(tags) AS t (tag)
Join with Table Function (UDTF)
Batch Streaming

Joins a table with the results of a table function. Each row of the left (outer) table is joined with all rows produced by the corresponding call of the table function.

User-defined table functions (UDTFs) must be registered before. See the UDF documentation for details on how to specify and register UDTFs.

Inner Join

A row of the left (outer) table is dropped, if its table function call returns an empty result.

  1. SELECT users, tag
  2. FROM Orders, LATERAL TABLE(unnest_udtf(tags)) AS t(tag)
  3. from 1.11, we can also do it like below:
  4. SELECT users, tag
  5. FROM Orders, LATERAL TABLE(unnest_udtf(tags))

Left Outer Join

If a table function call returns an empty result, the corresponding outer row is preserved and the result padded with null values.

  1. SELECT users, tag
  2. FROM Orders LEFT JOIN LATERAL TABLE(unnest_udtf(tags)) AS t(tag) ON TRUE
  3. from 1.11, we can also do it like below:
  4. SELECT users, tag
  5. FROM Orders LEFT JOIN LATERAL TABLE(unnest_udtf(tags)) ON TRUE

Note: Currently, only literal TRUE is supported as predicate for a left outer join against a lateral table.

Join with Temporal Table
Batch Streaming

Temporal Tables are tables that track changes over time. A Temporal Table provides access to the versions of a temporal table at a specific point in time.

Processing-time temporal join and event-time temporal join are supported, inner join and left join are supported.

The event-time temporal join is not suppored in Batch

The following example assumes that LatestRates is a Temporal Table which is materialized with the latest rate.

  1. SELECT
  2. o.amount, o.currency, r.rate, o.amount r.rate
  3. FROM
  4. Orders AS o
  5. JOIN LatestRates FOR SYSTEM_TIME AS OF o.proctime AS r
  6. ON r.currency = o.currency

The RHS table can be named with an alias using optional clause [[AS] correlationName], note that the AS keyword is also optional.

For more information please check the more detailed Temporal Tables concept description.

Only supported in Blink planner.

Join with Temporal Table Function
Streaming

Temporal tables are tables that track changes over time.

A Temporal table function provides access to the state of a temporal table at a specific point in time. The syntax to join a table with a temporal table function is the same as in Join with Table Function.

Note: Currently only inner joins with temporal tables are supported.

Assuming Rates is a temporal table function, the join can be expressed in SQL as follows:

  1. SELECT
  2. o_amount, r_rate
  3. FROM
  4. Orders,
  5. LATERAL TABLE (Rates(o_proctime))
  6. WHERE
  7. r_currency = o_currency

For more information please check the more detailed temporal tables concept description.

Set Operations

OperationDescription
Union
Batch
  1. SELECT
  2. FROM (
  3. (SELECT user FROM Orders WHERE a % 2 = 0)
  4. UNION
  5. (SELECT user FROM Orders WHERE b = 0)
  6. )
UnionAll
Batch Streaming
  1. SELECT
  2. FROM (
  3. (SELECT user FROM Orders WHERE a % 2 = 0)
  4. UNION ALL
  5. (SELECT user FROM Orders WHERE b = 0)
  6. )
Intersect / Except
Batch
  1. SELECT
  2. FROM (
  3. (SELECT user FROM Orders WHERE a % 2 = 0)
  4. INTERSECT
  5. (SELECT user FROM Orders WHERE b = 0)
  6. )
  1. SELECT
  2. FROM (
  3. (SELECT user FROM Orders WHERE a % 2 = 0)
  4. EXCEPT
  5. (SELECT user FROM Orders WHERE b = 0)
  6. )
In
Batch Streaming

Returns true if an expression exists in a given table sub-query. The sub-query table must consist of one column. This column must have the same data type as the expression.

  1. SELECT user, amount
  2. FROM Orders
  3. WHERE product IN (
  4. SELECT product FROM NewProducts
  5. )

Note: For streaming queries the operation is rewritten in a join and group operation. The required state to compute the query result might grow infinitely depending on the number of distinct input rows. Please provide a query configuration with valid retention interval to prevent excessive state size. See Query Configuration for details.

Exists
Batch Streaming

Returns true if the sub-query returns at least one row. Only supported if the operation can be rewritten in a join and group operation.

  1. SELECT user, amount
  2. FROM Orders
  3. WHERE product EXISTS (
  4. SELECT product FROM NewProducts
  5. )

Note: For streaming queries the operation is rewritten in a join and group operation. The required state to compute the query result might grow infinitely depending on the number of distinct input rows. Please provide a query configuration with valid retention interval to prevent excessive state size. See Query Configuration for details.

OrderBy & Limit

OperationDescription
Order By
Batch Streaming
Note: The result of streaming queries must be primarily sorted on an ascending time attribute. Additional sorting attributes are supported.
  1. SELECT
  2. FROM Orders
  3. ORDER BY orderTime
Limit
Batch
Note: The LIMIT clause requires an ORDER BY clause.
  1. SELECT
  2. FROM Orders
  3. ORDER BY orderTime
  4. LIMIT 3

Top-N

Attention Top-N is only supported in Blink planner.

Top-N queries ask for the N smallest or largest values ordered by columns. Both smallest and largest values sets are considered Top-N queries. Top-N queries are useful in cases where the need is to display only the N bottom-most or the N top- most records from batch/streaming table on a condition. This result set can be used for further analysis.

Flink uses the combination of a OVER window clause and a filter condition to express a Top-N query. With the power of OVER window PARTITION BY clause, Flink also supports per group Top-N. For example, the top five products per category that have the maximum sales in realtime. Top-N queries are supported for SQL on batch and streaming tables.

The following shows the syntax of the TOP-N statement:

  1. SELECT [column_list]
  2. FROM (
  3. SELECT [column_list],
  4. ROW_NUMBER() OVER ([PARTITION BY col1[, col2...]]
  5. ORDER BY col1 [asc|desc][, col2 [asc|desc]...]) AS rownum
  6. FROM table_name)
  7. WHERE rownum <= N [AND conditions]

Parameter Specification:

  • ROW_NUMBER(): Assigns an unique, sequential number to each row, starting with one, according to the ordering of rows within the partition. Currently, we only support ROW_NUMBER as the over window function. In the future, we will support RANK() and DENSE_RANK().
  • PARTITION BY col1[, col2...]: Specifies the partition columns. Each partition will have a Top-N result.
  • ORDER BY col1 [asc|desc][, col2 [asc|desc]...]: Specifies the ordering columns. The ordering directions can be different on different columns.
  • WHERE rownum <= N: The rownum <= N is required for Flink to recognize this query is a Top-N query. The N represents the N smallest or largest records will be retained.
  • [AND conditions]: It is free to add other conditions in the where clause, but the other conditions can only be combined with rownum <= N using AND conjunction.

Attention in Streaming Mode The TopN query is Result Updating. Flink SQL will sort the input data stream according to the order key, so if the top N records have been changed, the changed ones will be sent as retraction/update records to downstream. It is recommended to use a storage which supports updating as the sink of Top-N query. In addition, if the top N records need to be stored in external storage, the result table should have the same unique key with the Top-N query.

The unique keys of Top-N query is the combination of partition columns and rownum column. Top-N query can also derive the unique key of upstream. Take following job as an example, say product_id is the unique key of the ShopSales, then the unique keys of the Top-N query are [category, rownum] and [product_id].

The following examples show how to specify SQL queries with Top-N on streaming tables. This is an example to get “the top five products per category that have the maximum sales in realtime” we mentioned above.

  1. StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
  2. StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
  3. // ingest a DataStream from an external source
  4. DataStream<Tuple4<String, String, String, Long>> ds = env.addSource(...);
  5. // register the DataStream as table "ShopSales"
  6. tableEnv.createTemporaryView("ShopSales", ds, $("product_id"), $("category"), $("product_name"), $("sales"));
  7. // select top-5 products per category which have the maximum sales.
  8. Table result1 = tableEnv.sqlQuery(
  9. "SELECT * " +
  10. "FROM (" +
  11. " SELECT *," +
  12. " ROW_NUMBER() OVER (PARTITION BY category ORDER BY sales DESC) as row_num" +
  13. " FROM ShopSales)" +
  14. "WHERE row_num <= 5");
  1. val env = StreamExecutionEnvironment.getExecutionEnvironment
  2. val tableEnv = TableEnvironment.getTableEnvironment(env)
  3. // read a DataStream from an external source
  4. val ds: DataStream[(String, String, String, Long)] = env.addSource(...)
  5. // register the DataStream under the name "ShopSales"
  6. tableEnv.createTemporaryView("ShopSales", ds, $"product_id", $"category", $"product_name", $"sales")
  7. // select top-5 products per category which have the maximum sales.
  8. val result1 = tableEnv.sqlQuery(
  9. """
  10. |SELECT *
  11. |FROM (
  12. | SELECT *,
  13. | ROW_NUMBER() OVER (PARTITION BY category ORDER BY sales DESC) as row_num
  14. | FROM ShopSales)
  15. |WHERE row_num <= 5
  16. """.stripMargin)

No Ranking Output Optimization

As described above, the rownum field will be written into the result table as one field of the unique key, which may lead to a lot of records being written to the result table. For example, when the record (say product-1001) of ranking 9 is updated and its rank is upgraded to 1, all the records from ranking 1 ~ 9 will be output to the result table as update messages. If the result table receives too many data, it will become the bottleneck of the SQL job.

The optimization way is omitting rownum field in the outer SELECT clause of the Top-N query. This is reasonable because the number of the top N records is usually not large, thus the consumers can sort the records themselves quickly. Without rownum field, in the example above, only the changed record (product-1001) needs to be sent to downstream, which can reduce much IO to the result table.

The following example shows how to optimize the above Top-N example in this way:

  1. StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
  2. StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
  3. // ingest a DataStream from an external source
  4. DataStream<Tuple4<String, String, String, Long>> ds = env.addSource(...);
  5. // register the DataStream as table "ShopSales"
  6. tableEnv.createTemporaryView("ShopSales", ds, $("product_id"), $("category"), $("product_name"), $("sales"));
  7. // select top-5 products per category which have the maximum sales.
  8. Table result1 = tableEnv.sqlQuery(
  9. "SELECT product_id, category, product_name, sales " + // omit row_num field in the output
  10. "FROM (" +
  11. " SELECT *," +
  12. " ROW_NUMBER() OVER (PARTITION BY category ORDER BY sales DESC) as row_num" +
  13. " FROM ShopSales)" +
  14. "WHERE row_num <= 5");
  1. val env = StreamExecutionEnvironment.getExecutionEnvironment
  2. val tableEnv = TableEnvironment.getTableEnvironment(env)
  3. // read a DataStream from an external source
  4. val ds: DataStream[(String, String, String, Long)] = env.addSource(...)
  5. // register the DataStream under the name "ShopSales"
  6. tableEnv.createTemporaryView("ShopSales", ds, $"product_id", $"category", $"product_name", $"sales")
  7. // select top-5 products per category which have the maximum sales.
  8. val result1 = tableEnv.sqlQuery(
  9. """
  10. |SELECT product_id, category, product_name, sales -- omit row_num field in the output
  11. |FROM (
  12. | SELECT *,
  13. | ROW_NUMBER() OVER (PARTITION BY category ORDER BY sales DESC) as row_num
  14. | FROM ShopSales)
  15. |WHERE row_num <= 5
  16. """.stripMargin)

Attention in Streaming Mode In order to output the above query to an external storage and have a correct result, the external storage must have the same unique key with the Top-N query. In the above example query, if the product_id is the unique key of the query, then the external table should also has product_id as the unique key.

Deduplication

Attention Deduplication is only supported in Blink planner.

Deduplication is removing rows that duplicate over a set of columns, keeping only the first one or the last one. In some cases, the upstream ETL jobs are not end-to-end exactly-once, this may result in there are duplicate records in the sink in case of failover. However, the duplicate records will affect the correctness of downstream analytical jobs (e.g. SUM, COUNT). So a deduplication is needed before further analysis.

Flink uses ROW_NUMBER() to remove duplicates just like the way of Top-N query. In theory, deduplication is a special case of Top-N which the N is one and order by the processing time or event time.

The following shows the syntax of the Deduplication statement:

  1. SELECT [column_list]
  2. FROM (
  3. SELECT [column_list],
  4. ROW_NUMBER() OVER ([PARTITION BY col1[, col2...]]
  5. ORDER BY time_attr [asc|desc]) AS rownum
  6. FROM table_name)
  7. WHERE rownum = 1

Parameter Specification:

  • ROW_NUMBER(): Assigns an unique, sequential number to each row, starting with one.
  • PARTITION BY col1[, col2...]: Specifies the partition columns, i.e. the deduplicate key.
  • ORDER BY time_attr [asc|desc]: Specifies the ordering column, it must be a time attribute. Currently Flink supports processing time attribute and event time atttribute. Ordering by ASC means keeping the first row, ordering by DESC means keeping the last row.
  • WHERE rownum = 1: The rownum = 1 is required for Flink to recognize this query is deduplication.

The following examples show how to specify SQL queries with Deduplication on streaming tables.

  1. StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
  2. StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
  3. // ingest a DataStream from an external source
  4. DataStream<Tuple4<String, String, String, Integer>> ds = env.addSource(...);
  5. // register the DataStream as table "Orders"
  6. tableEnv.createTemporaryView("Orders", ds, $("order_id"), $("user"), $("product"), $("number"), $("proctime").proctime());
  7. // remove duplicate rows on order_id and keep the first occurrence row,
  8. // because there shouldn't be two orders with the same order_id.
  9. Table result1 = tableEnv.sqlQuery(
  10. "SELECT order_id, user, product, number " +
  11. "FROM (" +
  12. " SELECT *," +
  13. " ROW_NUMBER() OVER (PARTITION BY order_id ORDER BY proctime ASC) as row_num" +
  14. " FROM Orders)" +
  15. "WHERE row_num = 1");
  1. val env = StreamExecutionEnvironment.getExecutionEnvironment
  2. val tableEnv = TableEnvironment.getTableEnvironment(env)
  3. // read a DataStream from an external source
  4. val ds: DataStream[(String, String, String, Int)] = env.addSource(...)
  5. // register the DataStream under the name "Orders"
  6. tableEnv.createTemporaryView("Orders", ds, $"order_id", $"user", $"product", $"number", $"proctime".proctime)
  7. // remove duplicate rows on order_id and keep the first occurrence row,
  8. // because there shouldn't be two orders with the same order_id.
  9. val result1 = tableEnv.sqlQuery(
  10. """
  11. |SELECT order_id, user, product, number
  12. |FROM (
  13. | SELECT *,
  14. | ROW_NUMBER() OVER (PARTITION BY order_id ORDER BY proctime DESC) as row_num
  15. | FROM Orders)
  16. |WHERE row_num = 1
  17. """.stripMargin)

Deduplication can keep the time attribute of input stream, this is very helpful when the downstream operation is window aggregation or join operation. Both processing-time deduplication and event-time deduplication support working on mini-batch mode, this is more performance friendly, please see also mini-batch configuration for how to enable mini-batch mode..

Group Windows

Group windows are defined in the GROUP BY clause of a SQL query. Just like queries with regular GROUP BY clauses, queries with a GROUP BY clause that includes a group window function compute a single result row per group. The following group windows functions are supported for SQL on batch and streaming tables.

Group Window FunctionDescription
TUMBLE(time_attr, interval)Defines a tumbling time window. A tumbling time window assigns rows to non-overlapping, continuous windows with a fixed duration (interval). For example, a tumbling window of 5 minutes groups rows in 5 minutes intervals. Tumbling windows can be defined on event-time (stream + batch) or processing-time (stream).
HOP(time_attr, interval, interval)Defines a hopping time window (called sliding window in the Table API). A hopping time window has a fixed duration (second interval parameter) and hops by a specified hop interval (first interval parameter). If the hop interval is smaller than the window size, hopping windows are overlapping. Thus, rows can be assigned to multiple windows. For example, a hopping window of 15 minutes size and 5 minute hop interval assigns each row to 3 different windows of 15 minute size, which are evaluated in an interval of 5 minutes. Hopping windows can be defined on event-time (stream + batch) or processing-time (stream).
SESSION(time_attr, interval)Defines a session time window. Session time windows do not have a fixed duration but their bounds are defined by a time interval of inactivity, i.e., a session window is closed if no event appears for a defined gap period. For example a session window with a 30 minute gap starts when a row is observed after 30 minutes inactivity (otherwise the row would be added to an existing window) and is closed if no row is added within 30 minutes. Session windows can work on event-time (stream + batch) or processing-time (stream).

Time Attributes

For SQL queries on streaming tables, the time_attr argument of the group window function must refer to a valid time attribute that specifies the processing time or event time of rows. See the documentation of time attributes to learn how to define time attributes.

For SQL on batch tables, the time_attr argument of the group window function must be an attribute of type TIMESTAMP.

Selecting Group Window Start and End Timestamps

The start and end timestamps of group windows as well as time attributes can be selected with the following auxiliary functions:

Auxiliary FunctionDescription
TUMBLE_START(time_attr, interval)
HOP_START(time_attr, interval, interval)
SESSION_START(time_attr, interval)

Returns the timestamp of the inclusive lower bound of the corresponding tumbling, hopping, or session window.

TUMBLE_END(time_attr, interval)
HOP_END(time_attr, interval, interval)
SESSION_END(time_attr, interval)

Returns the timestamp of the exclusive upper bound of the corresponding tumbling, hopping, or session window.

Note: The exclusive upper bound timestamp cannot be used as a rowtime attribute in subsequent time-based operations, such as interval joins and group window or over window aggregations.

TUMBLE_ROWTIME(time_attr, interval)
HOP_ROWTIME(time_attr, interval, interval)
SESSION_ROWTIME(time_attr, interval)

Returns the timestamp of the inclusive upper bound of the corresponding tumbling, hopping, or session window.

The resulting attribute is a rowtime attribute that can be used in subsequent time-based operations such as interval joins and group window or over window aggregations.

TUMBLE_PROCTIME(time_attr, interval)
HOP_PROCTIME(time_attr, interval, interval)
SESSION_PROCTIME(time_attr, interval)

Returns a proctime attribute that can be used in subsequent time-based operations such as interval joins and group window or over window aggregations.

Note: Auxiliary functions must be called with exactly same arguments as the group window function in the GROUP BY clause.

The following examples show how to specify SQL queries with group windows on streaming tables.

  1. StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
  2. StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env);
  3. // ingest a DataStream from an external source
  4. DataStream<Tuple3<Long, String, Integer>> ds = env.addSource(...);
  5. // register the DataStream as table "Orders"
  6. tableEnv.createTemporaryView("Orders", ds, $("user"), $("product"), $("amount"), $("proctime").proctime(), $("rowtime").rowtime());
  7. // compute SUM(amount) per day (in event-time)
  8. Table result1 = tableEnv.sqlQuery(
  9. "SELECT user, " +
  10. " TUMBLE_START(rowtime, INTERVAL '1' DAY) as wStart, " +
  11. " SUM(amount) FROM Orders " +
  12. "GROUP BY TUMBLE(rowtime, INTERVAL '1' DAY), user");
  13. // compute SUM(amount) per day (in processing-time)
  14. Table result2 = tableEnv.sqlQuery(
  15. "SELECT user, SUM(amount) FROM Orders GROUP BY TUMBLE(proctime, INTERVAL '1' DAY), user");
  16. // compute every hour the SUM(amount) of the last 24 hours in event-time
  17. Table result3 = tableEnv.sqlQuery(
  18. "SELECT product, SUM(amount) FROM Orders GROUP BY HOP(rowtime, INTERVAL '1' HOUR, INTERVAL '1' DAY), product");
  19. // compute SUM(amount) per session with 12 hour inactivity gap (in event-time)
  20. Table result4 = tableEnv.sqlQuery(
  21. "SELECT user, " +
  22. " SESSION_START(rowtime, INTERVAL '12' HOUR) AS sStart, " +
  23. " SESSION_ROWTIME(rowtime, INTERVAL '12' HOUR) AS snd, " +
  24. " SUM(amount) " +
  25. "FROM Orders " +
  26. "GROUP BY SESSION(rowtime, INTERVAL '12' HOUR), user");
  1. val env = StreamExecutionEnvironment.getExecutionEnvironment
  2. val tableEnv = StreamTableEnvironment.create(env)
  3. // read a DataStream from an external source
  4. val ds: DataStream[(Long, String, Int)] = env.addSource(...)
  5. // register the DataStream under the name "Orders"
  6. tableEnv.createTemporaryView("Orders", ds, $"user", $"product", $"amount", $"proctime".proctime, $"rowtime".rowtime)
  7. // compute SUM(amount) per day (in event-time)
  8. val result1 = tableEnv.sqlQuery(
  9. """
  10. |SELECT
  11. | user,
  12. | TUMBLE_START(rowtime, INTERVAL '1' DAY) as wStart,
  13. | SUM(amount)
  14. | FROM Orders
  15. | GROUP BY TUMBLE(rowtime, INTERVAL '1' DAY), user
  16. """.stripMargin)
  17. // compute SUM(amount) per day (in processing-time)
  18. val result2 = tableEnv.sqlQuery(
  19. "SELECT user, SUM(amount) FROM Orders GROUP BY TUMBLE(proctime, INTERVAL '1' DAY), user")
  20. // compute every hour the SUM(amount) of the last 24 hours in event-time
  21. val result3 = tableEnv.sqlQuery(
  22. "SELECT product, SUM(amount) FROM Orders GROUP BY HOP(rowtime, INTERVAL '1' HOUR, INTERVAL '1' DAY), product")
  23. // compute SUM(amount) per session with 12 hour inactivity gap (in event-time)
  24. val result4 = tableEnv.sqlQuery(
  25. """
  26. |SELECT
  27. | user,
  28. | SESSION_START(rowtime, INTERVAL '12' HOUR) AS sStart,
  29. | SESSION_END(rowtime, INTERVAL '12' HOUR) AS sEnd,
  30. | SUM(amount)
  31. | FROM Orders
  32. | GROUP BY SESSION(rowtime(), INTERVAL '12' HOUR), user
  33. """.stripMargin)

Pattern Recognition

OperationDescription
MATCH_RECOGNIZE
Streaming

Searches for a given pattern in a streaming table according to the MATCH_RECOGNIZE ISO standard. This makes it possible to express complex event processing (CEP) logic in SQL queries.

For a more detailed description, see the dedicated page for detecting patterns in tables.

  1. SELECT T.aid, T.bid, T.cid
  2. FROM MyTable
  3. MATCH_RECOGNIZE (
  4. PARTITION BY userid
  5. ORDER BY proctime
  6. MEASURES
  7. A.id AS aid,
  8. B.id AS bid,
  9. C.id AS cid
  10. PATTERN (A B C)
  11. DEFINE
  12. A AS name = a’,
  13. B AS name = b’,
  14. C AS name = c
  15. ) AS T