- Basic setup
- Excample: Running Alluxio Locally with OBS
- Advanced Setup
- Contributed by the Alluxio Community
This guide describes how to configure Alluxio with Open Telecom OBS as the under storage system. Object Storage Service (OBS) is a massive, secure and highly reliable cloud storage service provided by Huawei Cloud.
To run an Alluxio cluster on a set of machines, you must deploy Alluxio binaries to each of these machines. You can either compile the binaries from Alluxio source code, or download the precompiled binaries directly.
OBS under storage is implemented as an under storage extension.
Clone the alluxio-extensions repo and run the following command under
$ mvn package -DskipTests
The built jar can be found under the
Execute the following command on master to install the extension to all masters and workers defined in
$ ./bin/alluxio extensions install /PATH/TO/OBS/jar
See here for more details on Alluxio extension management.
In preparation for using OBS with Alluxio, follow the OBS quick start guide to create a OBS bucket.
To configure Alluxio to use OBS as its under storage system, you will need to modify the configuration file
conf/alluxio-site.properties. If the file does not exist, create the configuration file from the template.
$ cp conf/alluxio-site.properties.template conf/alluxio-site.properties
conf/alluxio-site.properties file to set the under storage address to an existing OBS bucket and folder you want to mount to Alluxio.
For example, the under storage address can be
obs://alluxio-obs-bucket/ to mount the whole bucket to Alluxio
obs://alluxio-obs-bucket/alluxio/data if only the folder
/alluxio/data inside the bucket
is mapped to Alluxio.
Specify the Huawei Cloud credentials for OBS access. In
fs.obs.accessKey=<OBS_ACCESS_KEY> fs.obs.secretKey=<OBS_SECRET_KEY> fs.oss.endpoint=<OBS_ENDPOINT>
fs.obs.secretKey is access keys for Huawei cloud, please follow the
Access keys Guide
to create and manage your keys.
fs.obs.endpoint specifies the endpoint of the bucket.
Valid values are similar to
Available endpoints are listed in the Region and Endpoint list
under the Object Storage Service category.
Excample: Running Alluxio Locally with OBS
Start the Alluxio servers:
$ ./bin/alluxio format $ ./bin/alluxio-start.sh local
This should start an Alluxio master and an Alluxio worker. You can see the master UI at http://localhost:19999.
Run a simple example program:
$ ./bin/alluxio runTests
Visit your OBS folder
obs://<OBS_BUCKET>/<OBS_DIRECTORY> to verify the files
and directories created by Alluxio exist.
To stop Alluxio, you can run:
$ ./bin/alluxio-stop.sh local
An OBS location can be mounted at a nested directory in the Alluxio namespace to have unified access to multiple under storage systems. Alluxio’s Mount Command can be used for this purpose.
For example, the following command mounts a folder inside an OBS bucket into Alluxio
$ ./bin/alluxio fs mount --option fs.obs.accessKey=<OBS_ACCESS_KEY> \ --option fs.obs.secretKey=<OBS_SECRET_KEY> \ --option fs.obs.endpoint=<OBS_ENDPOINT> \ /obs obs://<OBS_BUCKET>/<OBS_DIRECTORY>/
Contributed by the Alluxio Community
OBS UFS integration is contributed and maintained by the Alluxio community. The source code is located here. Feel free submit pull requests to improve the integration and update the documentation here if any information is missing or out of date.