CephFS

This guide describes how to configure Alluxio with CephFS as the under storage system.

The Ceph File System (CephFS) is a POSIX-compliant file system built on top of Ceph’s distributed object store, RADOS. CephFS endeavors to provide a state-of-the-art, multi-use, highly available, and performant file store for a variety of applications, including traditional use-cases like shared home directories, HPC scratch space, and distributed workflow shared storage.

Alluxio supports two different implementations of under storage system for CephFS. Fore more information, please read its documentation:

Prerequisites

If you haven’t already, please see Prerequisites before you get started.

In preparation for using CephFS with Alluxio:

`<CEPHFS_CONF_FILE>`	Local path to Ceph configuration file ceph.conf
`<CEPHFS_NAME>`	Ceph URI that is used to identify dameon instances in the ceph.conf
`<CEPHFS_DIRECTORY>`	The directory you want to use, either by creating a new directory or using an existing one
`<CEPHFS_AUTH_ID>`	Ceph user id
`<CEPHFS_KEYRING_FILE>`	Ceph keyring file that stores one or more Ceph authentication keys

Install Dependencies

Follow Ceph packages install to install below packages:

cephfs-java
libcephfs_jni
libcephfs2

Make symbolic links

$ ln -s /usr/lib64/libcephfs_jni.so.1.0.0 /usr/lib64/libcephfs_jni.so
$ ln -s /usr/lib64/libcephfs.so.2.0.0 /usr/lib64/libcephfs.so
$ java_path=`which java | xargs readlink | sed 's#/bin/java##g'`
$ ln -s /usr/share/java/libcephfs.jar $java_path/jre/lib/ext/libcephfs.jar

Download CephFS Hadoop jar

$ curl -o $java_path/jre/lib/ext/hadoop-cephfs.jar -s https://download.ceph.com/tarballs/hadoop-cephfs.jar

Basic Setup

To use CephFS as the UFS of Alluxio root mount point, you need to configure Alluxio to use under storage systems by modifying conf/alluxio-site.properties. If it does not exist, create the configuration file from the template.

$ cp conf/alluxio-site.properties.template conf/alluxio-site.properties
$ cp conf/core-site.xml.template conf/core-site.xml

Set the following property to define CephFS as the root mount

alluxio.dora.client.ufs.root=cephfs://mon1\;mon2\;mon3/

cephfs
cephfs-hadoop

Modify conf/alluxio-site.properties to include:

alluxio.underfs.cephfs.conf.file=<CEPHFS_CONF_FILE>
alluxio.underfs.cephfs.mds.namespace=<CEPHFS_NAME>
alluxio.underfs.cephfs.mount.point=<CEPHFS_DIRECTORY>
alluxio.underfs.cephfs.auth.id=<CEPHFS_AUTH_ID>
alluxio.underfs.cephfs.auth.keyring=<CEPHFS_KEYRING_FILE>

Modify conf/alluxio-site.properties to include:

alluxio.underfs.hdfs.configuration=${ALLUXIO_HOME}/conf/core-site.xml

Modify conf/core-site.xml to include:

<configuration>
  <property>
    <name>fs.default.name</name>
    <value>ceph://mon1,mon2,mon3/</value>
  </property>
  <property>
    <name>fs.defaultFS</name>
    <value>ceph://mon1,mon2,mon3/</value>
  </property>
  <property>
    <name>ceph.data.pools</name>
    <value>${data-pools}</value>
  </property>
  <property>
    <name>ceph.auth.id</name>
    <value>${client-id}</value>
  </property>
  <property>
    <name>ceph.conf.options</name>
    <value>client_mount_gid=${gid},client_mount_uid=${uid},client_mds_namespace=${ceph-fs-name}</value>
  </property>
  <property>
    <name>ceph.root.dir</name>
    <value>${ceph-fs-dir}</value>
  </property>
  <property>
    <name>ceph.mon.address</name>
    <value>mon1,mon2,mon3</value>
  </property>
  <property>
    <name>fs.AbstractFileSystem.ceph.impl</name>
    <value>org.apache.hadoop.fs.ceph.CephFs</value>
  </property>
  <property>
    <name>fs.ceph.impl</name>
    <value>org.apache.hadoop.fs.ceph.CephFileSystem</value>
  </property>
  <property>
    <name>ceph.auth.keyring</name>
    <value>${client-keyring-file}</value>
  </property>
</configuration>

Running Alluxio Locally with CephFS

Once you have configured Alluxio to CephFS, try running Alluxio locally to see that everything works.

$ ./bin/alluxio init format
$ ./bin/alluxio process start local

Run a simple example program:

$ ./bin/alluxio exec basicIOTest

Visit your cephfs to verify the files and directories created by Alluxio exist. You should see files named like:

${cephfs-dir}/default_tests_files/Basic_CACHE_THROUGH

In cephfs, you can visit cephfs with ceph-fuse or mount by POSIX APIs. Mounting CephFS

In Alluxio, you can visit the nested directory in the Alluxio. Alluxio’s Command Line Interface can be used for this purpose.

/mnt/cephfs/default_tests_files/Basic_CACHE_THROUGH

Contributed by the Alluxio Community

CephFS and CephFS-Hadoop UFS integration is contributed and maintained by the Alluxio community. The source code for CephFS is located here and for CephFS-Hadoop is located here. Feel free submit pull requests to improve the integration and update the documentation here if any information is missing or out of date.

Categories