WEB
This guide describes the instructions to configure WEB as Alluxio’s under storage system.
Initial Setup
The Alluxio binaries must be on your machine. You can either compile Alluxio, or download the binaries locally.
Configuring Alluxio
Alluxio unifies access to different storage systems through the unified namespace feature. An WEB location can be either mounted at the root of the Alluxio namespace or at a nested directory.
ROOT MOUNT
Configure Alluxio to use under storage systems by modifying
conf/alluxio-site.properties. If it does not exist, create the configuration file from the
template.
$ cp conf/alluxio-site.properties.template conf/alluxio-site.properties
the following environment variable assignment needs to be added to
conf/alluxio-site.properties, and we actually support both http:// and https:// protocols.
alluxio.master.hostname=localhost
# alluxio.master.mount.table.root.ufs=[https|http]://<HOSTNAME>:<PORT>/DIRECTORY/
# A sample for this article
alluxio.master.mount.table.root.ufs=https://downloads.alluxio.io/downloads/files/
Specify the settings for parsing a WEB page(Optional):
alluxio.underfs.web.connnection.timeout=<WEB_CONNECTION_TIMEOUT>
alluxio.underfs.web.header.last.modified=<WEB_HEADER_LAST_MODIFIED>
alluxio.underfs.web.parent.names=<WEB_PARENT_NAMES>
alluxio.underfs.web.titles=<WEB_TITLES>
Here, alluxio.underfs.web.connnection.timeout is the timeout setting for an http connection
(unit: second, default: 60s).
alluxio.underfs.web.header.last.modified represents the format to parse the last modified field for a directory
or a file from an http response header (default: “EEE, dd MMM yyyy HH:mm:ss zzz”).
alluxio.underfs.web.parent.names indicates the start row index of the files list,
which can be set as multiple flags separated by commas (default: “Parent Directory,..,../”).
alluxio.underfs.web.titles is a flag that can be used to check if a web page is a directory.
It can also be set with multiple values separated by commas (default: “Index of ,Directory listing for “).
NESTED MOUNT
An WEB location can be mounted at a nested directory in the Alluxio namespace to have unified access to multiple under storage systems. Alluxio’s Mount Command can be used for this purpose.
For example, the following command mounts a directory inside an WEB directory into Alluxio directory
/web:
$ ./bin/alluxio fs mount \
--option alluxio.underfs.web.connnection.timeout=<WEB_CONNECTION_TIMEOUT> \
--option alluxio.underfs.web.header.last.modified=<WEB_HEADER_LAST_MODIFIED> \
--option alluxio.underfs.web.parent.names=<WEB_PARENT_NAMES> \
--option alluxio.underfs.web.titles=<WEB_TITLES> \
/web [https|http]://<HOSTNAME>:<PORT>/DIRECTORY/
Running Alluxio with WEB
Run the following command to start Alluxio filesystem.
$ ./bin/alluxio format
$ ./bin/alluxio-start.sh local
To verify that Alluxio is running, you can visit
http://localhost:19999, or see the log in the logs folder.
Visit your WEB volume by running the following command:
$ ./bin/alluxio fs ls /
After waiting for a while, you can see the following result:
dr--r----- 0 PERSISTED 05-21-2019 12:53:22:000 DIR /1.4.0
dr--r----- 0 PERSISTED 05-21-2019 12:54:23:000 DIR /1.5.0
dr--r----- 0 PERSISTED 05-21-2019 12:55:06:000 DIR /1.6.0
dr--r----- 0 PERSISTED 05-21-2019 12:55:38:000 DIR /1.6.1
dr--r----- 0 PERSISTED 05-21-2019 12:57:00:000 DIR /1.7.0
dr--r----- 0 PERSISTED 05-21-2019 12:57:57:000 DIR /1.7.1
dr--r----- 0 PERSISTED 05-21-2019 13:00:25:000 DIR /1.8.0
dr--r----- 0 PERSISTED 05-21-2019 13:02:07:000 DIR /1.8.1
dr--r----- 0 PERSISTED 05-24-2019 05:16:31:000 DIR /2.0.0
dr--r----- 0 PERSISTED 05-21-2019 13:02:11:000 DIR /2.0.0-preview
Stop Alluxio by running:
$ ./bin/alluxio-stop.sh local