Overview

Welcome to Alluxio Documentation! You will find resources regarding deploying Alluxio, integrations with various tech stacks, API references, and more! If you have any questions, join our Alluxio Community Slack → alluxio.io/slack

Alluxio Enterprise AI Overview

Alluxio Enterprise AI is a high-performance data platform designed to significantly enhance machine learning training and data access through advanced software architecture and intelligent caching capabilities. It bridges the gap between compute and storage, offering a high performance and cost efficient solution for seamless data access with billions objects scalability. Our platform redefines the way AI training and inference accesses data, providing users with a streamlined and efficient method to leverage data wherever it resides.

Epic Performance

Model Training: Achieve up to 20x I/O performance on top of your data lake by leveraging enhanced distributed caching tailored to AI workloads. Alluxio improves IO performance throughout various stages of the training workflow, from reading dataset to writing checkpoints, eliminating GPU idle time.

Model Serving: Up to 10x acceleration for serving models from offline training clusters to offline and online inference nodes compared to serving models directly from object stores. The architecture can be easily scaled to serve thousands of inference nodes without worrying about untimely model updates.

Seamless Data Access

Quickly deploy Alluxio alongside your GPU cluster with Kubernetes and connect it to your storage clusters. Immediately start training jobs with increased performance without needing to explicitly migrate data. Minimize the time to production for machine learning platforms across different cloud and on premise clusters.

High Scalability

Our distributed system architecture handles up to 100 billion objects using commodity hardware on the cloud without sacrificing latency.

Cost Efficiency

Rather than utilizing specialized storage hardware to increase performance, our solution eliminates the need to purchase additional hardware. Alluxio integrates seamlessly with existing data lakes and storage solutions and is deployed on the same hardware as the GPU cluster to self-serve the data with high I/O throughput.

Integration with AI Frameworks

Alluxio Enterprise AI supports various APIs, including POSIX (FUSE-based), S3, and FSSpec, to run workloads via PyTorch, TensorFlow, Apache Ray, or Spark.

Alluxio Enterprise AI is a comprehensive solution designed to meet the demands of modern AI and ML workloads. It delivers exceptional performance, seamless data access, and scalability, making it an essential tool for enterprises aiming to scale their AI operations efficiently.

Deploying with Kubernetes Operator

See Install Alluxio on Kubernetes on how to install Alluxio on Kubernetes via Helm, a kubernetes package manager, and Operator, a kubernetes extension for managing applications.