Tencent COS
This guide describes the instructions to configure Tencent COS as Alluxio’s under storage system.
Alluxio support two different implementations of under storage system for Tencent COS:
-
- COS
- Tencent Cloud Object Storage (COS) is a distributed storage service offered by Tencent Cloud for unstructured data and accessible via HTTP/HTTPS protocols. It can store massive amounts of data and features imperceptible bandwidth and capacity expansion, making it a perfect data pool for big data computation and analytics.
- For more information about Tencent COS, please read its documentation.
-
- COSN, also known as Hadoop-COS
- COSN is a client that makes the upper computing systems based on HDFS be able to use Tencent COS as its underlying storage system.
- For more information about COSN, please read its documentation.
Prerequisites
If you haven’t already, please see Prerequisites before you get started.
In preparation for using COS or COSN with Alluxio:
Basic Setup
After these changes, Alluxio should be configured to work with COS or COSN as its under storage system.
Running Alluxio Locally with COS/COSN
Once you have configured Alluxio to Tencent COS or COSN, try running Alluxio locally to see that everything works.
Advanced Setup
[Experimental] COS multipart upload
The default upload method uploads one file completely from start to end in one go. We use multipart-upload method to upload one file by multiple parts, every part will be uploaded in one thread. It won’t generate any temporary files while uploading.
To enable COS multipart upload, you need to modify conf/alluxio-site.properties
to include:
alluxio.underfs.cos.multipart.upload.enabled=true
There are other parameters you can specify in conf/alluxio-site.properties
to make the process faster and better.
# Timeout for uploading part when using multipart upload.
alluxio.underfs.object.store.multipart.upload.timeout
# Thread pool size for COS multipart upload.
alluxio.underfs.cos.multipart.upload.threads
# Multipart upload partition size for COS. The default partition size is 64MB.
alluxio.underfs.cos.multipart.upload.partition.size
Contributed by the Alluxio Community
Tencent COS and COSN UFS integration is contributed and maintained by the Alluxio community. The source code for COS is located here and for COSN is located here. Feel free submit pull requests to improve the integration and update the documentation here if any information is missing or out of date.