Alibaba Cloud MaxCompute

MaxCompute (previously known as ODPS) is a data warehousing solution that can process terabytes or petabytes of data.

What is MaxCompute

Instructions

  1. Max Compute Catalog is developed based on Max Compute Tunnel SDK, and Query performance is limited.
  2. During a query, Max Compute DownloadSession objects are created for each Scan. Therefore, performance will degrade when Max Compute is accessed in parallel. It is recommended that minimize the number of partitions and the size of the queried data when using the Max Compute Catalog.

Connect to MaxCompute

  1. CREATE CATALOG mc PROPERTIES (
  2.   "type" = "max_compute",
  3.   "mc.region" = "cn-beijing",
  4.   "mc.default.project" = "your-project",
  5.   "mc.access_key" = "ak",
  6.   "mc.secret_key" = "sk"
  7. );
  • mc.region: MaxCompute Region. Can Get the Region From Endpoints.
  • mc.default.project: MaxCompute Project. See Your MaxCompute Projects.
  • mc.access_key: AccessKey, which you can create and manage on the Alibaba Cloud console.
  • mc.secret_key: SecretKey, which you can create and manage on the Alibaba Cloud console.
  • mc.public_access: You can enable public network access for test, when set "mc.public_access"="true".

Quotas

Pay-as-you-go quota has limited concurrency and usage. For additional resources, please refer to the documentation. See Manage quotas.

Column type mapping

Consistent with Hive Catalog, please refer to the column type mapping section in Hive Catalog.

User-defined service address

The region property is specified to generate a default endpoint of public network.

In addition to default endpoint addresses, Max Compute Catalog also supports custom service addresses in properties.

Use the following properties:

  • mc.odps_endpoint：Max Compute Endpoint。
  • mc.tunnel_endpoint: Tunnel Endpoint，Max Compute Catalog uses the Tunnel SDK to obtain data.

For more information about Max Compute Endpoint and Tunnel Endpoint that are used in different regions and network connection modes, see Endpoint

For example:

  1. CREATE CATALOG mc PROPERTIES (
  2.   "type" = "max_compute",
  3.   "mc.region" = "cn-beijing",
  4.   "mc.default.project" = "your-project",
  5.   "mc.access_key" = "ak",
  6.   "mc.secret_key" = "sk"
  7.   "mc.odps_endpoint" = "http://service.cn-beijing.maxcompute.aliyun-inc.com/api",
  8.   "mc.tunnel_endpoint" = "http://dt.cn-beijing.maxcompute.aliyun-inc.com"
  9. );