Aliyun Object Storage Service
This guide describes how to configure Aliyun OSS as Alluxio’s under storage system.
Aliyun Object Storage Service (OSS) is a massive, secure and highly reliable cloud storage service provided by Alibaba Cloud. OSS provides multiple storage classes to help you manage and reduce storage costs.
For more information about Aliyun OSS, please read its documentation
If you haven’t already, please see Prerequisites before you get started.
In preparation for using OSS with Alluxio:
||Create a a new bucket in the OSS console or use an existing bucket|
||The directory you want to use in the bucket, either by creating a new directory or using an existing one|
||ID used to identify a user. See How to Obtain AccessKey Pair|
||Secret is used to verify the identify of the user. See How to Obtain AccessKey Pair|
||Endpoints are the domain names that other services can use to access OSS. See Regions and OSS Endpoints in the Public Cloud|
To use Aliyun OSS as the UFS of Alluxio root mount point, you need to configure Alluxio to use under storage systems by modifying
conf/alluxio-site.properties. If it does not exist, create the configuration file from the template.
$ cp conf/alluxio-site.properties.template conf/alluxio-site.properties
Specify an OSS bucket and directory as the underfs address by modifying
For example, the under storage address can be
you want to mount the whole bucket to Alluxio, or
oss://alluxio-bucket/alluxio/data if only the directory
inside the oss bucket
alluxio-bucket is mapped to Alluxio.
Specify credentials for Aliyun OSS access by adding the following properties in
fs.oss.accessKeyId=<OSS_ACCESS_KEY_ID> fs.oss.accessKeySecret=<OSS_ACCESS_KEY_SECRET> fs.oss.endpoint=<OSS_ENDPOINT>
Running Alluxio Locally with Aliyun OSS
Once you have configured Alluxio to Aliyun OSS, try running Alluxio locally to see that everything works.
[Experimental] OSS multipart upload
The default upload method uploads one file completely from start to end in one go. We use multipart-upload method to upload one file by multiple parts, every part will be uploaded in one thread. It won’t generate any temporary files while uploading.
To enable OSS multipart upload, you need to modify
conf/alluxio-site.properties to include:
There are other parameters you can specify in
conf/alluxio-site.properties to make the process faster and better.
# Timeout for uploading part when using multipart upload. alluxio.underfs.object.store.multipart.upload.timeout
# Thread pool size for OSS multipart upload. alluxio.underfs.oss.multipart.upload.threads
# Multipart upload partition size for OSS. The default partition size is 64MB. alluxio.underfs.oss.multipart.upload.partition.size
Contributed by the Alluxio Community
Aliyun OSS UFS integration is contributed and maintained by the Alluxio community. The source code is located here. Feel free submit pull requests to improve the integration and update the documentation here if any information is missing or out of date.