FUSE SDK Overview
The Alluxio POSIX API is a feature that allows mounting training datasets in specific storage services (e.g. S3, HDFS) to the local filesystem and provides caching capabilities to speed up I/O access to frequently used data.
Local Cache vs Distributed Cache
There are two kinds of caching capabilities: 1. local caching only 2. local caching + distributed caching.
Differences between the two solutions are listed below, choose your desired solution based on training requirements and available resources.
| Category | Local Caching | Distributed Caching |
|---|---|---|
| Prerequisite | N/A | Require a running Alluxio cluster (master + worker) |
| Caching Capability | Bounded by local storage size | Bounded by Alluxio cluster storage size |
| Suited Workloads | Single node training with large dataset. Distributed training with no data shuffle between nodes | Multiple training nodes or training tasks share the same dataset |
Local Caching Solution
See Local Cache Quick Start to quickly setup your FUSE SDK local cache solution which can connects to your desired storage services.
Local Cache Tuning provides different local cache capabilities to speed up your workloads and reduce the pressure of storage services.
- Local Kernel Data Cache Configuration
- Local Userspace Data Cache Configuration
- Local Kernel Metadata Cache Configuration
- Local Userspace Metadata Cache Configuration
Advanced Tuning Guide provides advanced FUSE SDK tuning tips for performance optimization or debugging.
Distributed Caching Solution
FUSE SDK can connect to a shared distributed caching service. For more information, please refer to Distributed Cache Quick Start