Tencent COS

This guide describes the instructions to configure Tencent COS as Alluxio’s under storage system.

Alluxio support two different implementations of under storage system for Tencent COS:

COS

Tencent Cloud Object Storage (COS) is a distributed storage service offered by Tencent Cloud for unstructured data and accessible via HTTP/HTTPS protocols. It can store massive amounts of data and features imperceptible bandwidth and capacity expansion, making it a perfect data pool for big data computation and analytics.

For more information about Tencent COS, please read its documentation.
COSN, also known as Hadoop-COS

COSN is a client that makes the upper computing systems based on HDFS be able to use Tencent COS as its underlying storage system.

For more information about COSN, please read its documentation.

Prerequisites

If you haven’t already, please see Prerequisites before you get started.

In preparation for using COS or COSN with Alluxio:

COS
COSN

`<COS_BUCKET>`	Create a new bucket or use an existing bucket
`<COS_DIRECTORY>`	The directory you want to use in the bucket, either by creating a new directory or using an existing one
`<COS_ACCESS_KEY>`	A developer-owned access key used for the project. It can be obtained at Manage API Key
`<COS_SECRET_KEY>`	A developer-owned secret key used for the project. It can be obtained at Manage API Key
`<COS_REGION>`	Region information. For more information about the enumerated values, please see Regions and Access Endpoints
`<COS_APPID>`	A unique user-level resource identifier for COS access. It can be obtained at Manage API Key

`<COSN_BUCKET>`	Create a new bucket or use an existing bucket
`<COSN_DIRECTORY>`	The directory you want to use in the bucket, either by creating a new directory or using an existing one
`<COSN_SECRET_ID>`	ID used to authenticate user
`<COSN_SECRET_KEY>`	Key used to authenticate user
`<COSN_REGION>`	Region information, for more information about the enumerated values please see Regions and Access Endpoints

To use Tencent COS as the UFS of Alluxio root mount point, you need to configure Alluxio to use under storage systems by modifying conf/alluxio-site.properties. If it does not exist, create the configuration file from the template.

$ cp conf/alluxio-site.properties.template conf/alluxio-site.properties

Specify an existing COS bucket and directory as the underfs address by modifying conf/alluxio-site.properties to include:

alluxio.dora.client.ufs.root=cos://COS_BUCKET/COS_DIRECTORY/

Note that if you want to mount the whole cos bucket, add a trailing slash after the bucket name (e.g. cos://COS_BUCKET/).

Specify the COS credentials for COS access by setting fs.cos.access.key and fs.cos.secret.key in alluxio-site.properties.

fs.cos.access.key=<COS_ACCESS_KEY>
fs.cos.secret.key=<COS_SECRET_KEY>

Specify the COS region by setting fs.cos.region in alluxio-site.properties (e.g. ap-beijing) and fs.cos.app.id.

fs.cos.region=<COS_REGION>
fs.cos.app.id=<COS_APPID>

To use Tencent COSN as the UFS of Alluxio root mount point, you need to configure Alluxio to use under storage systems by modifying conf/alluxio-site.properties and conf/core-site.xml. If they do not exist, create the files from the template.

$ cp conf/alluxio-site.properties.template conf/alluxio-site.properties
$ cp conf/core-site.xml.template conf/core-site.xml

Specify an existing COSN bucket and directory as the underfs address by modifying conf/alluxio-site.properties to include:

alluxio.dora.client.ufs.root=cosn://COSN_BUCKET/COSN_DIRECTORY/

Specify COSN configuration information in order to access COSN by modifying conf/core-site.xml to include COSN_SECRET_KEY, COSN_SECRET_ID, AND COSN_REGION:

<property>
   <name>fs.cosn.impl</name>
   <value>org.apache.hadoop.fs.CosFileSystem</value>
</property>
<property>
  <name>fs.AbstractFileSystem.cosn.impl</name>
  <value>org.apache.hadoop.fs.CosN</value>
</property>
<property>
  <name>fs.cosn.userinfo.secretKey</name>
  <value>COSN_SECRET_KEY</value>
</property>
<property>
  <name>fs.cosn.userinfo.secretId</name>
  <value>COSN_SECRET_ID</value>
</property>
<property>
  <name>fs.cosn.bucket.region</name>
  <value>COSN_REGION</value>
</property>

The above is the most basic configuration. For more configuration please refer to here.

After these changes, Alluxio should be configured to work with COS or COSN as its under storage system.

Running Alluxio Locally with COS/COSN

Once you have configured Alluxio to Tencent COS or COSN, try running Alluxio locally to see that everything works.

Advanced Setup

[Experimental] COS multipart upload

The default upload method uploads one file completely from start to end in one go. We use multipart-upload method to upload one file by multiple parts, every part will be uploaded in one thread. It won’t generate any temporary files while uploading.

To enable COS multipart upload, you need to modify conf/alluxio-site.properties to include:

alluxio.underfs.cos.multipart.upload.enabled=true

There are other parameters you can specify in conf/alluxio-site.properties to make the process faster and better.

# Timeout for uploading part when using multipart upload.
alluxio.underfs.object.store.multipart.upload.timeout

# Thread pool size for COS multipart upload.
alluxio.underfs.cos.multipart.upload.threads

# Multipart upload partition size for COS. The default partition size is 64MB. 
alluxio.underfs.cos.multipart.upload.partition.size

Contributed by the Alluxio Community

Tencent COS and COSN UFS integration is contributed and maintained by the Alluxio community. The source code for COS is located here and for COSN is located here. Feel free submit pull requests to improve the integration and update the documentation here if any information is missing or out of date.