Storage Integrations Overview
This guide will cover general prerequisites and running Alluxio locally with your desired under storage system. To learn how to configure Alluxio with each individual storage system, please look at their respective pages.
Prerequisites
In preparation for using your chosen storage system with Alluxio, please be sure you have all the required location, credentials, and additional properties before you begin configuring Alluxio to your under storage system.
For the purposes of this guide, the following are placeholders.
| Storage System | Location | Credentials | Additional Properties |
|---|---|---|---|
| Amazon AWS S3 | S3_BUCKET, S3_DIRECTORY |
S3_ACCESS_KEY_ID, S3_SECRET_KEY |
|
| HDFS | HDFS_NAMENODE, HDFS_PORT |
Specify Hadoop version: HADOOP_VERSION |
|
| Aliyun Object Storage Service (OSS) | OSS_BUCKET, OSS_DIRECTORY |
OSS_ACCESS_KEY_ID, OSS_ACCESS_KEY_SECRET, OSS_ENDPOINT |
|
| Azure Blob Store | AZURE_CONTAINER, AZURE_DIRECTORY |
AZURE_ACCOUNT, AZURE_ACCOUNT_KEY |
|
| Azure Data Lake Storage Gen1 | AZURE_DIRECTORY |
AZURE_ACCOUNT |
OAuth credentials: CLIENT_ID, AUTHENTICATION_KEY, TENANT_ID |
| Azure Data Lake Storage Gen2 | AZURE_CONTAINER, AZURE_DIRECTORY |
AZURE_ACCOUNT, AZURE_SHARED_KEY |
OAuth credentials: OAUTH_ENDPOINT, CLIENT_ID, CLIENT_SECRET, MSI_ENDPOINT, MSI_TENANT |
| CephFS | CEPHFS_CONF_FILE, CEPHFS_NAME, CEPHFS_DIRECTORY, CEPHFS_AUTH_ID, CEPHFS_KEYRING_FILE |
||
| CephObjectStorage | CEPH_BUCKET, CEPH_DIRECTORY |
S3_ACCESS_KEY_ID, S3_SECRET_KEY_ID
|
RGW_HOSTNAME, RGW_PORT, INHERIT_ACL
|
| Google Cloud Storage (GCS) | GCS_BUCKET, GCS_DIRECTORY |
For GCS Version 1: GCS_ACCESS_KEY_ID, GCS_SECRET_ACCESS_KEY |
|
| Huawei Object Storage Service (OBS) | OBS_BUCKET, OBS_DIRECTORY |
OBS_ACCESS_KEY, OBS_SECRET_KEY, OBS_ENDPOINT |
|
| MinIO | MINIO_BUCKET, MINIO_DIRECTORY |
S3_ACCESS_KEY_ID, S3_SECRET_KEY, MINIO_ENDPOINT |
|
| NFS | |||
| Ozone |
o3fs: OZONE_BUCKET, OZONE_VOLUME ofs: OZONE_MANAGER, OZONE_BUCKET, OZONE_DIRECTORY, OZONE_VOLUME |
OM_SERVICE_IDS |
Mount specific version: OZONE_VERSION |
| Tencent Cloud Object Storage (COS) | COS_BUCKET, COS_DIRECTORY |
COS_ACCESS_KEY, COS_SECRET_KEY |
Specify COS region: COS_REGION, COS_APPID
|
| Tencent Cloud Object Storage in Hadoop (COSN) | COSN_BUCKET, COSN_DIRECTORY |
COSN_SECRET_ID, COSN_SECRET_KEY |
Specify COSN region: COSN_REGION
|
Running Alluxio Locally
Once you have configured Alluxio to your desired under storage system, start up Alluxio locally to see that everything works.
$ ./bin/alluxio init format
$ ./bin/alluxio process start local
This should start an Alluxio master and an Alluxio worker. You can see the master UI at http://localhost:19999.
Run a simple example program:
$ ./bin/alluxio exec basicIOTest
Visit your container <CONTAINER>/<DIRECTORY> or bucket <BUCKET>/<DIRECTORY> to verify the files and directories created by Alluxio exist. If there are no errors, then you have successfully configured your storage system!
To stop Alluxio, you can run:
$ ./bin/alluxio process stop local