We are excited to announce the release of Alluxio Enterprise 2.4.1-1.0! This is a patch release on the Alluxio Enterprise 2.4.X line. It contains a variety of bug fixes over the existing Alluxio Enterprise 2.4.0. It’s recommended to upgrade from 2.4.0 to 2.4.1 as soon as possible.
- Embedded journal bug fixes and stabilization
- Improve HDFS,Hive, and other validation tools
- Update Oshi version to 5.3.5 to solve JVM crash issue
- Use longer remote IO timeouts to account for slow operations
- Remove unneeded gRPC connection
- Better support dynamic non-root user in Alluxio docker integration
- Use lower tiers in workers based on tier usage
- Implement concurrent UFS input stream cache
- Implement access API for Hadoop client
- Add ETag headers in S3 Proxy API to support reading fils with Python s3fs
- Fix active sync path check
- Fix and improve UFS IO test
- Add logs for slow remote reads and writes
- Support setting folder in Alluxio performance test
- (Enterprise Edition) Add AWS dynamic secret support
- (Enterprise Edition) Add login to REST endpoints
- (Enterprise Edition) Fix Hub Dataproc init action
- (Enterprise Edition) Run the Hub automatically for data proc and open ports
- (Enterprise Edition) Add correct checks in reverse URI translation
- (Enterprise Edition) Add support for removing mountpoint in Hub
- (Enterprise Edition) Add speed check support in Hub for UFS with hidden credentials
- (Enterprise Edition) Prevent root mount deletes in Hub UI
- (Enterprise Edition) Various Improvement for Hub UI and HMS
We’re pleased to announce the general availability of Alluxio Enterprise 2.4.0-2.0, including the launch of Data Orchestration Hub! The data ecosystem is complex with the separation of storage and compute across data centers and cloud providers. With this release we’ve made great strides towards simplifying data access and management across multiple environments.
Data Orchestration Hub, or the Hub, is a management console that makes it easy to manage analytics or AI clusters with Alluxio, and connect them with multiple data sources to unify data lakes. The service provides an easy to use unified management view for configuration and monitoring, and wizard based curation of deployment workflows. See the docs for more details.
As part of the first release of the Hub, the following tools are available immediately.
Connect to Data Sources
The data storage console provides wizards to configure and validate connectivity with data sources. The initial release supports adding and editing mount points for the following storage systems:
- Amazon S3
- Google Cloud Storage.
Additionally once a connection is established via the wizards, a “speed check” tool is provided to run distributed IO tests to measure the connection speed between the Alluxio cluster and a data source.
The process management console allows a user to monitor the status of all processes in the cluster, and also start or stop processes. Group or individual actions are available to start, stop or restart processes across the cluster, sequentially or in parallel.
Configure Data Catalogs
The data catalog console provides wizards to configure and validate connectivity with data catalogs. In this release, the Hub only supports configuring a Hive catalog for Presto without having to re-define table locations for Alluxio.
The advanced management console allows a user to make arbitrary configuration changes to customize the deployment without SSH access. This includes site properties, log levels, and environment variables. Additionally, files can be distributed to the cluster using this tool.
- Improve backup memory management
- Improve polling client for embedded journal
- Upgrade Ozone UFS to 1.0.0 version
- (Enterprise Edition) Improve TLS support
- (Enterprise Edition) Fix trailing slash truncation in Union UFS
We are excited to announce the release of Alluxio Enterprise 2.4.0-1.0! This is the first release on the Alluxio Enterprise 2.4.X line. It contains a variety of feature enhancements, bug fixes, and performance improvements over the existing Alluxio Enterprise 2.3.X line.
Organizations leverage Alluxio at enormous scale, both in data size and number of files. With this release, users will be able to manage namespaces with billions of files while breaking away from dependencies on traditional Hadoop components. We have further bolstered support for cloud native and container based deployments with simplified system monitoring, DevOps tooling, and integrations for secure credential management.
Embedded Journal Scalability
Alluxio 2.4 allows users to manage namespaces with billions of files with the embedded journal.
The embedded journal is a fully contained, distributed state machine which uses the RAFT consensus algorithm. This allows users to have a fault tolerant and performant medium for journal storage, independent of 3rd party storage systems and Zookeeper. Users running only on object stores will be able to deploy Alluxio in high availability mode without incurring a large metadata performance penalty or relying on another distributed storage. See the docs for more details.
Note: the embedded journal format in 2.4 is not compatible with embedded journal format in lower versions. Please read here regarding how to upgrade the journals.
ADLS and Apache Ozone Under Filesystem Integrations
Alluxio 2.4 comes with under filesystem implementations for ADLS v1 and Apache Ozone.
Azure Data Lake Storage is a hyper scale, Hadoop compatible data lake on Microsoft Azure. Users can now connect Alluxio to ADLS with the ADLS v1 connector to enable compute to access data in ADLS. See the docs for more details.
Apache Ozone is a scalable, redundant, and distributed object store for Hadoop. Users can now connect Alluxio to Ozone with the Ozone connector to enable compute to access data in Ozone. See the docs for more details.
1-command Log Collection
Alluxio 2.4 improves the built in log collection process so administrators can gather all relevant cluster logs in a single file with one command.
The standardized log collection command greatly simplifies the administrator’s life when submitting logs for troubleshooting. See the docs for more details.
Cluster Performance Metrics
Alluxio 2.4 allows users to track key cluster performance metrics in the web interface (
Alluxio 2.4 includes timeseries for aggregated cluster throughput and cumulative API calls served per mount point. Cluster throughput is essential for determining the effectiveness of the Alluxio cache. API calls served is a strong metric for quantifying the latency and potential cost savings provided by Alluxio’s namespace virtualization.
Java 11 Support
Alluxio 2.4 client and server artifacts support Java 11 in addition to Java 8 environments.
Java 11 can improve the performance and stability of Big Data applications including Alluxio, Presto, and Spark. Applications relying on the Alluxio client library can be run in a Java 11 or Java 8 environment. The Alluxio servers can be run in Java 11 or Java 8 environments. See the docs for more details.
JVM Pause Monitoring
Alluxio 2.4 enables users to better monitor cluster health by logging a warning when a long JVM pause is detected.
JVM pauses can cause unexpected application slowdowns or failures. The root cause of a JVM pause can be from several sources such as extensive garbage collection or underlying machine instability. Being able to monitor and track JVM pauses experienced by the Alluxio servers greatly helps system administrators in troubleshooting and maintaining cluster health.
Enterprise Edition Highlights
Hashicorp Vault Support
Alluxio 2.4 integrates with secret stores like Hashicorp Vault, allowing users to store sensitive secrets in a trusted secret store and only provide secret store authentication information in Alluxio.
Users with secret stores can manage the lifecycle of their access tokens and passwords without needing to reconfigure Alluxio after rotations. Administrators no longer need to provide sensitive information directly to the Alluxio system. This feature is currently in private beta, contact us if you would like to participate.
Job Failure History Server
Alluxio 2.4 allows programmatic access to job failure history through a REST endpoint.
Users can query the job failures and provide custom logic for handling failures based on failure type and cause. See the docs for more details.
1-click GCP Deployment
Alluxio 2.4 is integrated with Terraform deployment scripts that allow for easy deployment on GCP with Dataproc.
Users of Alluxio in GCP can easily customize and spin up a cluster with the deployment scripts provided. See the docs for more details.
- Stabilize Alluxio delegation backup functionality
- Improve error message and reduce spamminess
- Add job failure history rest API endpoint
- Support HDFS 2.8, 2.9, 3.2 versions by default for EMR and Dataproc
- Change Alluxio default HDFS version to 3.3 instead of 2.7
- Add metadata operations saved metrics
- Support java related commands like jmap, jstack in Alluxio docker containers
- Support du -g command showing capacity information grouped by worker
- Support dynamic users in Alluxio docker fuse container
- Add Azure Data Lake Gen 1 UFS
- Add Alluxio kubernetes operator
- Add S3 connection TTL
- Improve hdfs, hms, and environment validation tools
- Increase master serving timeout for slower machines
- Support older hive metastores in Alluxio structured data service
- Support Glue column statistics in structured data service
- Add proxy support for Glue udb of structured data service
- Improve credentials protection
- Simplify UfsStatus handling in loadMetadata
- Support customized environment variables for helm chart
- Optimize UFS access on missing paths
- Support customize mount point in docker fuse container
- Support RPC cancellation
- Improve worker streaming read/write tracking
- (Enterprise Edition) Use certificate from TLS keystore for SecureRpcServer
- (Enterprise Edition) Add logging for capability master keys
- (Enterprise Edition) Support compute cluster connecting to HA hdfs/hive for hybrid cloud
- (Enterprise Edition) Improve terraform scripts for hybrid cloud deployment
- Fix and improve Alluxio Dataproc and EMR scripts
- Update sync status after processing the entire sync
- Fix cp command with wildcard and special characters in path
- Auto close worker client when file read finished to avoid resources exhausted
- Avoid Active Sync manager connecting to UFS when becoming secondary mater
- Fix NullPointerException in worker tier promote task
- Fix AbstractWriteHandler abort
- Cancel in-progress checkpoints when thread is shutdown
- Pass options to DelegatingFileSystem
- Fix java opts passing in docker and kubernetes environment
- Fix comma-separated medium types parsing in Helm chart
- Fix block location iteration with rocksdb
- Fix Object UFS edge case
- Resolve null values in structured data service with Glue
- (Enterprise Edition) Fix presto apache bugs
- (Enterprise Edition) Avoid excessive querying of block size in union FS listStatus