Presto

This documentation is a guide for using Paimon in Presto.

Version

Paimon currently supports Presto 0.236 and above.

Preparing Paimon Jar File

Download from master: https://paimon.apache.org/docs/master/project/download/

You can also manually build a bundled jar from the source code.

To build from the source code, clone the git repository.

Build presto connector plugin with the following command.

mvn clean install -DskipTests

After the packaging is complete, you can choose the corresponding connector based on your own Presto version:

Of course, we also support different versions of Hive and Hadoop. But note that we utilize Presto-shaded versions of Hive and Hadoop packages to address dependency conflicts. You can check the following two links to select the appropriate versions of Hive and Hadoop:

hadoop-apache2

hive-apache

Both Hive 2 and 3, as well as Hadoop 2 and 3, are supported.

For example, if your presto version is 0.274, hive and hadoop version is 2.x, you could run:

mvn clean install -DskipTests -am -pl paimon-presto-0.273 -Dpresto.version=0.274 -Dhadoop.apache2.version=2.7.4-9 -Dhive.apache.version=1.2.2-2

Tmp Dir

Paimon will unzip some jars to the tmp directory for codegen. By default, Presto will use '/tmp' as the temporary directory, but '/tmp' may be periodically deleted.

You can configure this environment variable when Presto starts:

-Djava.io.tmpdir=/path/to/other/tmpdir

Let Paimon use a secure temporary directory.

Configure Paimon Catalog

Install Paimon Connector

tar -zxf paimon-presto-${PRESTO_VERSION}/target/paimon-presto-${PRESTO_VERSION}-${PAIMON_VERSION}-plugin.tar.gz -C ${PRESTO_HOME}/plugin

Note that, the variable PRESTO_VERSION is module name, must be one of 0.236, 0.268, 0.273.

Configuration

cd ${PRESTO_HOME}
mkdir -p etc/catalog

connector.name=paimon
# set your filesystem path, such as hdfs://namenode01:8020/path and s3://${YOUR_S3_BUCKET}/path
warehouse=${YOUR_FS_PATH}

If you are using HDFS FileSystem, you will also need to do one more thing: choose one of the following ways to configure your HDFS:

• set environment variable HADOOP_HOME.
• set environment variable HADOOP_CONF_DIR.
• configure hadoop-conf-dir in the properties.

If you are using S3 FileSystem, you need to add paimon-s3-${PAIMON_VERSION}.jar in ${PRESTO_HOME}/plugin/paimon and additionally configure the following properties in paimon.properties:

s3.endpoint=${YOUR_ENDPOINTS}
s3.access-key=${YOUR_AK}
s3.secret-key=${YOUR_SK}

Query HiveCatalog table:

vim etc/catalog/paimon.properties

and set the following config:

connector.name=paimon
# set your filesystem path, such as hdfs://namenode01:8020/path and s3://${YOUR_S3_BUCKET}/path
warehouse=${YOUR_FS_PATH}
metastore=hive
uri=thrift://${YOUR_HIVE_METASTORE}:9083

Kerberos

You can configure kerberos keytab file when using KERBEROS authentication in the properties.

security.kerberos.login.principal=hadoop-user
security.kerberos.login.keytab=/etc/presto/hdfs.keytab

Keytab files must be distributed to every node in the cluster that runs Presto.

Create Schema

CREATE SCHEMA paimon.test_db;

Create Table

CREATE TABLE paimon.test_db.orders (
    order_key bigint,
    order_status varchar,
    total_price decimal(18,4),
    order_date date
)
WITH (
    file_format = 'ORC',
    primary_key = ARRAY['order_key','order_date'],
    partitioned_by = ARRAY['order_date'],
    bucket = '2',
    bucket_key = 'order_key',
    changelog_producer = 'input'
)

Add Column

CREATE TABLE paimon.test_db.orders (
    order_key bigint,
    orders_tatus varchar,
    total_price decimal(18,4),
    order_date date
)
WITH (
    file_format = 'ORC',
    primary_key = ARRAY['order_key','order_date'],
    partitioned_by = ARRAY['order_date'],
    bucket = '2',
    bucket_key = 'order_key',
    changelog_producer = 'input'
)
ALTER TABLE paimon.test_db.orders ADD COLUMN "shipping_address varchar;

Query

SELECT * FROM paimon.default.MyTable

Presto to Paimon type mapping

This section lists all supported type conversion between Presto and Paimon. All Presto’s data types are available in package com.facebook.presto.common.type.