Release Notes

Slack Docker Pulls

2.4.0-1.0

We are excited to announce the release of Alluxio Enterprise 2.4.0-1.0! This is the first release on the Alluxio Enterprise 2.4.X line. It contains a variety of feature enhancements, bug fixes, and performance improvements over the existing Alluxio Enterprise 2.3.X line.

Organizations leverage Alluxio at enormous scale, both in data size and number of files. With this release, users will be able to manage namespaces with billions of files while breaking away from dependencies on traditional Hadoop components. We have further bolstered support for cloud native and container based deployments with simplified system monitoring, DevOps tooling, and integrations for secure credential management.

Highlights

Embedded Journal Scalability

Alluxio 2.4 allows users to manage namespaces with billions of files with the embedded journal.

The embedded journal is a fully contained, distributed state machine which uses the RAFT consensus algorithm. This allows users to have a fault tolerant and performant medium for journal storage, independent of 3rd party storage systems and Zookeeper. Users running only on object stores will be able to deploy Alluxio in high availability mode without incurring a large metadata performance penalty or relying on another distributed storage. See the docs for more details.

Note: the embedded journal format in 2.4 is not compatible with embedded journal format in lower versions. Please read here regarding how to upgrade the journals.

ADLS and Apache Ozone Under Filesystem Integrations

Alluxio 2.4 comes with under filesystem implementations for ADLS v1 and Apache Ozone.

Azure Data Lake Storage is a hyper scale, Hadoop compatible data lake on Microsoft Azure. Users can now connect Alluxio to ADLS with the ADLS v1 connector to enable compute to access data in ADLS. See the docs for more details.

Apache Ozone is a scalable, redundant, and distributed object store for Hadoop. Users can now connect Alluxio to Ozone with the Ozone connector to enable compute to access data in Ozone. See the docs for more details.

1-command Log Collection

Alluxio 2.4 improves the built in log collection process so administrators can gather all relevant cluster logs in a single file with one command.

The standardized log collection command greatly simplifies the administrator’s life when submitting logs for troubleshooting. See the docs for more details.

Cluster Performance Metrics

Alluxio 2.4 allows users to track key cluster performance metrics in the web interface (/metrics) and/or programmatically.

Alluxio 2.4 includes timeseries for aggregated cluster throughput and cumulative API calls served per mount point. Cluster throughput is essential for determining the effectiveness of the Alluxio cache. API calls served is a strong metric for quantifying the latency and potential cost savings provided by Alluxio’s namespace virtualization.

Java 11 Support

Alluxio 2.4 client and server artifacts support Java 11 in addition to Java 8 environments.

Java 11 can improve the performance and stability of Big Data applications including Alluxio, Presto, and Spark. Applications relying on the Alluxio client library can be run in a Java 11 or Java 8 environment. The Alluxio servers can be run in Java 11 or Java 8 environments. See the docs for more details.

JVM Pause Monitoring

Alluxio 2.4 enables users to better monitor cluster health by logging a warning when a long JVM pause is detected.

JVM pauses can cause unexpected application slowdowns or failures. The root cause of a JVM pause can be from several sources such as extensive garbage collection or underlying machine instability. Being able to monitor and track JVM pauses experienced by the Alluxio servers greatly helps system administrators in troubleshooting and maintaining cluster health.

Enterprise Edition Highlights

Hashicorp Vault Support

Alluxio 2.4 integrates with secret stores like Hashicorp Vault, allowing users to store sensitive secrets in a trusted secret store and only provide secret store authentication information in Alluxio.

Users with secret stores can manage the lifecycle of their access tokens and passwords without needing to reconfigure Alluxio after rotations. Administrators no longer need to provide sensitive information directly to the Alluxio system. This feature is currently in private beta, contact us if you would like to participate.

Job Failure History Server

Alluxio 2.4 allows programmatic access to job failure history through a REST endpoint.

Users can query the job failures and provide custom logic for handling failures based on failure type and cause. See the docs for more details.

1-click GCP Deployment

Alluxio 2.4 is integrated with Terraform deployment scripts that allow for easy deployment on GCP with Dataproc.

Users of Alluxio in GCP can easily customize and spin up a cluster with the deployment scripts provided. See the docs for more details.

Other Improvements

  • Stabilize Alluxio delegation backup functionality
  • Improve error message and reduce spamminess
  • Add job failure history rest API endpoint
  • Support HDFS 2.8, 2.9, 3.2 versions by default for EMR and Dataproc
  • Change Alluxio default HDFS version to 3.3 instead of 2.7
  • Add metadata operations saved metrics
  • Support java related commands like jmap, jstack in Alluxio docker containers
  • Support du -g command showing capacity information grouped by worker
  • Support dynamic users in Alluxio docker fuse container
  • Add Azure Data Lake Gen 1 UFS
  • Add Alluxio kubernetes operator
  • Add S3 connection TTL
  • Improve hdfs, hms, and environment validation tools
  • Increase master serving timeout for slower machines
  • Support older hive metastores in Alluxio structured data service
  • Support Glue column statistics in structured data service
  • Add proxy support for Glue udb of structured data service
  • Improve credentials protection
  • Simplify UfsStatus handling in loadMetadata
  • Support customized environment variables for helm chart
  • Optimize UFS access on missing paths
  • Support customize mount point in docker fuse container
  • Support RPC cancellation
  • Improve worker streaming read/write tracking
  • (Enterprise Edition) Use certificate from TLS keystore for SecureRpcServer
  • (Enterprise Edition) Add logging for capability master keys
  • (Enterprise Edition) Support compute cluster connecting to HA hdfs/hive for hybrid cloud
  • (Enterprise Edition) Improve terraform scripts for hybrid cloud deployment

Bug Fixes

  • Fix and improve Alluxio Dataproc and EMR scripts
  • Update sync status after processing the entire sync
  • Fix cp command with wildcard and special characters in path
  • Auto close worker client when file read finished to avoid resources exhausted
  • Avoid Active Sync manager connecting to UFS when becoming secondary mater
  • Fix NullPointerException in worker tier promote task
  • Fix AbstractWriteHandler abort
  • Cancel in-progress checkpoints when thread is shutdown
  • Pass options to DelegatingFileSystem
  • Fix java opts passing in docker and kubernetes environment
  • Fix comma-separated medium types parsing in Helm chart
  • Fix block location iteration with rocksdb
  • Fix Object UFS edge case
  • Resolve null values in structured data service with Glue
  • (Enterprise Edition) Fix presto apache bugs
  • (Enterprise Edition) Avoid excessive querying of block size in union FS listStatus