We are thrilled to announce the release of Alluxio Enterprise 2.8.0-1.0! This is the first release on the Alluxio Enterprise 2.8.X line.
Alluxio 2.8 further enhances the S3 API functionality, job service observability, data encryption, and system scalability.
Enhanced S3 API with Metadata Tagging
Alluxio 2.8 commences a slew of ongoing improvements to the S3-compatible REST API. Object and bucket tagging APIs have been added (f3d2477) which lay the groundwork for managing end-to-end file metadata through the Alluxio S3 API.
For example the “x-amz-tagging” header is now supported by the PutObject REST endpoint. You can specify user-defined tags to attach to the uploaded object through this request header as a query-parameterized string:
curl -H "x-amz-tagging: key1=value1&key2=value2&flag" -XPUT "http://<host>:<port>/api/v1/s3/<bucket>/<object>" --data="..."
Object tags can be retrieved via:
curl -XGET "http://<host>:<port>/api/v1/s3/<bucket>/<object>?tagging"
See the Alluxio S3 documentation for full details on supported tagging operations. Tags are currently limited to 10 user-defined tags per file in accordance with the S3 specifications.
This release aims to also further compatibility with S3 client applications through request header additions and improvements (e913945).
Stabilize and Scale Alluxio in Deep Training Workloads
Alluxio 2.8 improves the FUSE mount/unmount mechanism. FUSE can be mounted through the Alluxio CLI or via configuration properties (e5f53a9). Fuse unmount mechanism is improved to reduce the chance of leaving Fuse unmounted in the host machines (5393b6). FUSE3 (6f3fe6f) integration is newly supported, enabling future performance and scalability optimizations. Although FUSE2 is the default version, Alluxio will eventually phase out FUSE2 in favor of FUSE3.
Several critical issues were identified in production workloads and fixed to improve the stability of Alluxio when integrating with training workloads. Examples include a RocksDB core crash with high memory consumption (918e73), FUSE segment fault error (issue), and FUSE statfs potential OOM (fabcf47).
More optimizations were added to support a large number of small files and highly concurrent access, including supporting worker registration with millions of blocks (htf8e5e) and improving the performance by 10x and reducing memory overhead when preloading a large number of small files (bc104a9).
Exception Handling for Data Movement
In Alluxio 2.8, we improved the exception handling of data movement, specifically the distributedCp and distributedMv operations. The improvements include removing 0-byte files if the task failed or was canceled (2c83498, 75210e0) and preserving the destination file for overwrites until the write operation succeeds, which previously would have been deleted at the start of the job (2fa2683).
Asynchronous Job Service Execution
Distributed commands for load and copy can now be submitted in asynchronous mode
The new mode can be turned on by specifying the
--async flag in the
distributedCp and distributedMv commands
This new feature simplifies the command submission and monitoring process for the end user. In previous versions, users needed to ensure their command console to remain active and not be interrupted while long-running commands were executing. After launching an asynchronous command, its progress can be reported by running the getCmdStatus CLI command. In the default sync mode, users will no longer see the progress information on files being loaded or copied as they complete. The output will now display the command JobControlId, successfully loaded or copied files, each separated on a new line, statistics of completed and failed files, and the failed file path information.
System Stability and Scalability
Multiple improvements were made to reduce the general memory usage of Alluxio master. In particular, we improved the performance of recursive deletion and reduced the memory consumption, both by roughly 20%. Based on our performance tests, we have also updated recommendations on how recursive deletion can be done more efficiently. You can find the recommendations in our User-CLI doc.
New metrics were added in various parts of the system to provide better observability into the system state, including but not limited to, metadata sync operations (f24c26), data I/O exceptions (250bb3), and distributed commands (696cb8).
Alluxio 2.8 also added the ability for standby masters to serve the Web UI and metrics (a10823).
StressBench Tool Usability
In 2.8 we improved the usability of the StressBench framework by making some parameters more user-friendly. We also added a quick start pipeline to batch run multiple stress tests.
A new workflow has been added under the name MaxFile. The goal of this benchmark is to evaluate the maximum number of files an Alluxio master can handle before becoming unresponsive. The benchmark can run for many hours depending on the metastore type and heap space allocated to the Alluxio master. You can find more information on this document.
Enterprise Edition Highlights
Data Encryption for Enterprise-grade Security
Alluxio 2.8 supports encryption at rest of data in Alluxio managed storage. In conjunction with SSL, this new feature provides full server-side encryption, ensuring end-to-end data security. Data stored on the Alluxio worker is encrypted on write and the data is decrypted when it is read and sent to the client or UFS.
The feature introduces the concept of an encryption zone. Each encryption zone maps to an Alluxio URI, encapsulating files prefixed by the URI. The mapping of paths in the file system enables users to specify which parts of the namespace will be secured by encryption. This provides the ability to balance between security when within an encryption zone and performance for the unspecified paths.
Alluxio creates encryption keys for each encryption zone. The credentials can be stored in Hashicorp Vault or in the master journal. The latter option should only be used for testing or validation.
Alluxio supports different encryption algorithms:
AES/CTR/NoPadding is the default algorithm.
Read the encryption at rest documentation for more details.
Alluxio 2.8 introduces a new under storage access token framework. Instead of storing credentials on each worker, workers request for access tokens from the master as needed. These tokens can be permanent or set to expire after a certain duration.
In AWS S3 specific scenarios, users can configure the worker to request for a unique AssumeRole token for either each individual user or for each user and path combination to access S3 objects. The obtained token will expire in an hour and the worker will automatically refresh the token as needed.
When compared to the scenario where each worker independently requests for and manages its own access tokens, the AssumeRole framework can improve the performance by up to 45%.
Read more about configuring the AssumeRole feature for S3 in the documentation.
Policy Driven Data Management Observability
In Alluxio 2.8, we improved the observability of policy driven data management to enable users to diagnose problems when policy initiated jobs fail. A user CLI (eff3d75) has been added and metrics (3a22ce6) are exposed to report the status of the policy engine, policy execution status, and action status. Check out Policy-Driven Data Management for a detailed description of the new CLI.
IMPROVEMENTS AND FIXES
See improvements and fixes since 2.7.0 in their respective release notes:
Enterprise edition changelog
- Add metrics to policy scan (71ccb79ca2)
- Add client cert, client key, CA cert and verify configs (e11561b487)
- Correct the type of vault version (f74953556c)
- Fix the secure server handler if the encryption is not enabled (efb977bab4)
- Split encryption meta from block meta (3f555b4f46)
- Update PDDM Cli for ActionInfo and report (eff3d75aad)
- Add encryption for tier storage (e815c664db)
- Fix alluxio.web.login.session.timeout prop key type (52e62fbdbf)
- Add raft info
fsadmin reportcommand (acb5779cd9)
- Add license check for worker stream register (defd1fa056)
- Add a config for worker secure rpc port (3bbb9e5a09)
- Fix the user name in the cache of opa (54179befdb)
- Modify the capability key to temporary encryption key (3aeafd44bb)
- Copy ranger-2.1 auth module with generic name (f4fb93cf41)
- Add counter for assume role related operations (ed315645cd)
- Add refresh logic for S3 AssumeRole tokens (1afdb11720)
- Fix the AssumeRole Token generation for Worker S3A UFS instance (26d924875a)
- Support assume role credential process (d2b4c70af3)
- Fix space in file name bug (79a8ea68e0)
- Support the AssumeRole token per resource and operation (fd5ef89086)