alluxio.master.file

Class InodeSyncStream

java.lang.Object

alluxio.master.file.InodeSyncStream

public class InodeSyncStream
extends Object

This class is responsible for maintaining the logic which surrounds syncing metadata between Alluxio and its UFSes. This implementation uses a BFS-based approach to crawl the inode tree. In order to speed up the sync process we use an ExecutorService which we submit inode paths to using processSyncPath(AlluxioURI). The processing of inode paths will discover new paths to sync depending on the mDescendantType. Syncing is finished when all submitted tasks are completed and there are no new inodes left in the queue. Syncing inode metadata requires making calls to the UFS. This implementation will schedule UFS RPCs with the UfsStatusCache.prefetchChildren(AlluxioURI, MountTable). Then, once the inode begins processing, it can retrieve the results. After processing, it can then remove its UfsStatus from the cache. This strategy helps reduce memory pressure on the master while performing a sync for a large tree. Additionally, by using a prefetch mechanism we can concurrently process other inodes while waiting for UFS RPCs to complete. With regards to locking, this class expects to be able to take a write lock on any inode, and then subsequently downgrades or unlocks after the sync is finished. Even though we use ReentrantReadWriteLock, because we concurrently process inodes on separate threads, we cannot utilize the reetrnant behavior. The implications of that mean the caller of this class must not hold a write while calling sync(). A user of this class is expected to create a new instance for each path that they would like to process. This is because the Lock on the mRootScheme may be changed after calling sync().

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	InodeSyncStream.SyncStatus Return status of a sync result.

Constructor Summary

Constructors

Constructor and Description
InodeSyncStream(LockingScheme rootScheme, DefaultFileSystemMaster fsMaster, RpcContext rpcContext, DescendantType descendantType, FileSystemMasterCommonPOptions options, boolean isGetFileInfo, boolean forceSync, boolean loadOnly, boolean loadAlways) Create a new instance of InodeSyncStream without any audit or permission checks.
InodeSyncStream(LockingScheme rootPath, DefaultFileSystemMaster fsMaster, RpcContext rpcContext, DescendantType descendantType, FileSystemMasterCommonPOptions options, FileSystemMasterAuditContext auditContext, java.util.function.Function<LockedInodePath,Inode> auditContextSrcInodeFunc, alluxio.master.file.DefaultFileSystemMaster.PermissionCheckFunction permissionCheckOperation, boolean isGetFileInfo, boolean forceSync, boolean loadOnly, boolean loadAlways) Create a new instance of InodeSyncStream.

Method Summary

Modifier and Type	Method and Description
static List<Journal.JournalEntry>	mergeCreateComplete(List<Journal.JournalEntry> entries) Merge inode entry with subsequent update inode and update inode file entries.
InodeSyncStream.SyncStatus	sync() Sync the metadata according the the root path the stream was created with.
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

InodeSyncStream

public InodeSyncStream(LockingScheme rootPath,
                       DefaultFileSystemMaster fsMaster,
                       RpcContext rpcContext,
                       DescendantType descendantType,
                       FileSystemMasterCommonPOptions options,
                       @Nullable
                       FileSystemMasterAuditContext auditContext,
                       @Nullable
                       java.util.function.Function<LockedInodePath,Inode> auditContextSrcInodeFunc,
                       @Nullable
                       alluxio.master.file.DefaultFileSystemMaster.PermissionCheckFunction permissionCheckOperation,
                       boolean isGetFileInfo,
                       boolean forceSync,
                       boolean loadOnly,
                       boolean loadAlways)

Create a new instance of InodeSyncStream. The root path should be already locked with InodeTree.LockPattern.WRITE_EDGE unless the user is only planning on loading metadata. The desired pattern should always be InodeTree.LockPattern.READ. It is an error to initiate sync without a WRITE_EDGE lock when loadOnly is false. If loadOnly is set to true, then the the root path may have a read lock.

Parameters:

rootPath - The root path to begin syncing

fsMaster - the FileSystemMaster calling this method

rpcContext - the caller's RpcContext

descendantType - determines the number of descendant inodes to sync

options - the RPC's FileSystemMasterCommonPOptions

auditContext - the audit context to use when loading

auditContextSrcInodeFunc - the inode to set as the audit context source

permissionCheckOperation - the operation to use to check permissions

isGetFileInfo - whether the caller is FileSystemMaster.getFileInfo(long)

forceSync - whether to sync inode metadata no matter what

loadOnly - whether to only load new metadata, rather than update existing metadata

loadAlways - whether to always load new metadata from the ufs, even if a file or directory has been previous found to not exist

InodeSyncStream

public InodeSyncStream(LockingScheme rootScheme,
                       DefaultFileSystemMaster fsMaster,
                       RpcContext rpcContext,
                       DescendantType descendantType,
                       FileSystemMasterCommonPOptions options,
                       boolean isGetFileInfo,
                       boolean forceSync,
                       boolean loadOnly,
                       boolean loadAlways)

Create a new instance of InodeSyncStream without any audit or permission checks.

Parameters:

rootScheme - The root path to begin syncing

fsMaster - the FileSystemMaster calling this method

rpcContext - the caller's RpcContext

descendantType - determines the number of descendant inodes to sync

options - the RPC's FileSystemMasterCommonPOptions

isGetFileInfo - whether the caller is FileSystemMaster.getFileInfo(long)

forceSync - whether to sync inode metadata no matter what

loadOnly - whether to only load new metadata, rather than update existing metadata

loadAlways - whether to always load new metadata from the ufs, even if a file or directory has been previous found to not exist

Method Detail

sync

public InodeSyncStream.SyncStatus sync()
                                throws AccessControlException,
                                       InvalidPathException

Sync the metadata according the the root path the stream was created with.

Returns:

SyncStatus object

Throws:

AccessControlException

InvalidPathException

mergeCreateComplete

public static List<Journal.JournalEntry> mergeCreateComplete(List<Journal.JournalEntry> entries)

Merge inode entry with subsequent update inode and update inode file entries.

Parameters:

entries - list of journal entries

Returns:

a list of compacted journal entries

toString

public String toString()

Overrides:

toString in class Object