Ceph

Slack Docker Pulls GitHub edit source

This guide describes how to configure Alluxio with Ceph as the under storage system. Alluxio supports two different clients APIs to connect to Ceph Object Storage using Rados Gateway:

Initial Setup

The Alluxio binaries must be on your machine. You can either compile Alluxio, or download the binaries locally.

Configuring Alluxio

You need to configure Alluxio to use under storage systems by modifying conf/alluxio-site.properties. If it does not exist, create the configuration file from the template.

  1. cp conf/alluxio-site.properties.template conf/alluxio-site.properties

Option 1: S3A Interface (preferred)

Modify conf/alluxio-site.properties to include:

  1. alluxio.underfs.address=s3a://<bucket>/<folder>
  2. aws.accessKeyId=<access-key>
  3. aws.secretKey=<secret-key>
  4. alluxio.underfs.s3.endpoint=http://<rgw-hostname>:<rgw-port>
  5. alluxio.underfs.s3.disable.dns.buckets=true
  6. alluxio.underfs.s3a.inherit_acl=<inherit-acl>

If using a Ceph release such as hammer (or older) specify alluxio.underfs.s3a.signer.algorithm=S3SignerType to use v2 S3 signatures. To use GET Bucket (List Objects) Version 1 specify alluxio.underfs.s3a.list.objects.v1=true.

Option 2: Swift Interface

Modify conf/alluxio-site.properties to include:

  1. alluxio.underfs.address=swift://<container>/<folder>
  2. fs.swift.user=<swift-user>
  3. fs.swift.tenant=<swift-tenant>
  4. fs.swift.password=<swift-user-password>
  5. fs.swift.auth.url=<swift-auth-url>
  6. fs.swift.use.public.url=<swift-use-public>
  7. fs.swift.auth.method=<swift-auth-model>

Replace <container>/<folder> with an existing Swift container location. Possible values of <swift-use-public> are true, false. Specify <swift-auth-model> as swiftauth if using native Ceph RGW authentication and <swift-auth-url> as http://<rgw-hostname>:<rgw-port>/auth/1.0.

Running Alluxio Locally with Ceph

Start up Alluxio locally to see that everything works.

  1. ./bin/alluxio format
  2. ./bin/alluxio-start.sh local

This should start an Alluxio master and an Alluxio worker. You can see the master UI at http://localhost:19999.

Run a simple example program:

  1. ./bin/alluxio runTests

Visit your bucket/container to verify the files and directories created by Alluxio exist.

If using the S3A connector, you should see files named like:

  1. <bucket>/<folder>/default_tests_files/Basic_CACHE_THROUGH

If using the Swift connector, you should see files named like:

  1. <container>/<folder>/default_tests_files/Basic_CACHE_THROUGH

To stop Alluxio, run:

  1. ./bin/alluxio-stop.sh local

Access Control

If Alluxio security is enabled, Alluxio enforces the access control inherited from underlying Ceph Object Storage. Depending on the interace used, refer to S3A Access Control or Swift Access Control for more information.