FUSE SDK Overview

Slack Docker Pulls GitHub edit source

The Alluxio POSIX API is a feature that allows mounting training datasets in specific storage services (e.g. S3, HDFS) to the local filesystem and provides caching capabilities to speed up I/O access to frequently used data.

Local Cache vs Distributed Cache

There are two kinds of caching capabilities: 1. local caching only 2. local caching + distributed caching.

Differences between the two solutions are listed below, choose your desired solution based on training requirements and available resources.

Category Local Caching Distributed Caching
Prerequisite N/A Require a running Alluxio cluster (master + worker)
Caching Capability Bounded by local storage size Bounded by Alluxio cluster storage size
Suited Workloads Single node training with large dataset. Distributed training with no data shuffle between nodes Multiple training nodes or training tasks share the same dataset

Local Caching Solution

See Local Cache Quick Start to quickly setup your FUSE SDK local cache solution which can connects to your desired storage services.

Local Cache Tuning provides different local cache capabilities to speed up your workloads and reduce the pressure of storage services.

Advanced Tuning Guide provides advanced FUSE SDK tuning tips for performance optimization or debugging.

Distributed Caching Solution

FUSE SDK can connect to a shared distributed caching service. For more information, please refer to Distributed Cache Quick Start