alluxio.worker.dora

Class PagedDoraWorker

java.lang.Object

alluxio.worker.AbstractWorker

alluxio.worker.dora.PagedDoraWorker

All Implemented Interfaces:

Server<WorkerNetAddress>, DataWorker, DoraWorker, SessionCleanable, Worker

public class PagedDoraWorker
extends AbstractWorker
implements DoraWorker

Page store based dora worker.

Field Summary

Fields

Modifier and Type	Field and Description
static long	DUMMY_BLOCK_SIZE
protected AlluxioConfiguration	mConf
protected DoraUfsManager	mUfsManager

Constructor Summary

Constructors

Modifier	Constructor and Description
	PagedDoraWorker(AtomicReference<Long> workerId, WorkerIdentity identity, AlluxioConfiguration conf, CacheManager cacheManager, MembershipManager membershipManager) Constructor.
protected	PagedDoraWorker(AtomicReference<Long> workerId, WorkerIdentity identity, AlluxioConfiguration conf, CacheManager cacheManager, MembershipManager membershipManager, BlockMasterClientPool blockMasterClientPool, FileSystemContext fileSystemContext)

Method Summary

Modifier and Type	Method and Description
FileInfo	buildFileInfoFromUfsStatus(UfsStatus status, String ufsFullPath) Build FileInfo from UfsStatus and UFS full Path.
DoraMeta.FileStatus	buildFileStatusFromUfsStatus(UfsStatus status, String ufsFullPath) Build FileStatus from UfsStatus and UFS full Path.
protected void	checkCopyPermission(String srcPath, String dstPath)
protected void	checkMovePermission(String srcPath, String dstPath)
void	cleanupSession(long sessionId) Cleans up the session.
void	close() Closes the server.
void	completeFile(String path, CompleteFilePOptions options, String uuid) Complete writing a file.
com.google.common.util.concurrent.ListenableFuture<List<RouteFailure>>	copy(List<Route> routes, UfsReadOptions readOptions, WriteOptions writeOptions) Copies files from src to dst.
void	createDirectory(String path, CreateDirectoryPOptions options) Create a directory.
OpenFileHandle	createFile(String path, CreateFilePOptions options) Create File.
BlockReader	createFileReader(String fileId, long offset, boolean positionShort, Protocol.OpenUfsBlockOptions options) Creates the file reader to read from Alluxio dora.
BlockWriter	createFileWriter(String fileId, String ufsPath) Creates the file writer to write to Alluxio dora.
void	delete(String path, DeletePOptions options) Delete a file.
boolean	exists(String path, ExistsPOptions options) Check existence of a file or dir.
WorkerNetAddress	getAddress() Get the address of the Dora Worker.
protected int	getCachedPercentage(FileInfo fi, String ufsFullPath)
Set<Class<? extends Server>>	getDependencies()
FileInfo	getFileInfo(String ufsFullPath, GetStatusPOptions options) Gets the file information.
protected FileInfo	getGrpcFileInfo(String ufsFullPath, long syncIntervalMs)
String	getName()
protected DoraOpenFileHandleContainer	getOpenFileHandleContainer()
Map<ServiceType,GrpcService>	getServices()
protected UnderFileSystem	getUfsInstance(String ufsUriStr)
protected UnderFileSystem	getUnderFileSystem(String ufsPath)
AtomicReference<Long>	getWorkerId()
UfsStatus[]	listStatus(String path, ListStatusPOptions options) List status from Under File System.
com.google.common.util.concurrent.ListenableFuture<LoadFileResponse>	load(boolean loadData, boolean skipIfExists, List<UfsStatus> ufsStatuses, UfsReadOptions options) Loads the metadata and data of files from UFS to Alluxio.
protected void	loadData(String ufsPath, long mountId, long length)
com.google.common.util.concurrent.ListenableFuture<List<RouteFailure>>	move(List<Route> routes, UfsReadOptions readOptions, WriteOptions writeOptions) Moves files from src to dst.
void	rename(String src, String dst, RenamePOptions options) Rename src to dst.
void	setAttribute(String path, SetAttributePOptions options) Set attribute for this file/dir.
void	start(WorkerNetAddress address) Starts the Alluxio server.
void	stop() Stops the Alluxio server.

Methods inherited from class alluxio.worker.AbstractWorker

getExecutorService

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DUMMY_BLOCK_SIZE

public static final long DUMMY_BLOCK_SIZE

See Also:

Constant Field Values

mUfsManager

protected final DoraUfsManager mUfsManager

mConf

protected final AlluxioConfiguration mConf

Constructor Detail

PagedDoraWorker

@Inject
public PagedDoraWorker(@Named(value="workerId")
                               AtomicReference<Long> workerId,
                               WorkerIdentity identity,
                               AlluxioConfiguration conf,
                               CacheManager cacheManager,
                               MembershipManager membershipManager)

Constructor.

Parameters:

workerId -

identity -

conf -

cacheManager -

membershipManager -

PagedDoraWorker

protected PagedDoraWorker(AtomicReference<Long> workerId,
                          WorkerIdentity identity,
                          AlluxioConfiguration conf,
                          CacheManager cacheManager,
                          MembershipManager membershipManager,
                          BlockMasterClientPool blockMasterClientPool,
                          FileSystemContext fileSystemContext)

Method Detail

getUfsInstance

protected UnderFileSystem getUfsInstance(String ufsUriStr)

getDependencies

public Set<Class<? extends Server>> getDependencies()

Specified by:

getDependencies in interface Server<WorkerNetAddress>

Returns:

the server dependencies

getName

public String getName()

Specified by:

getName in interface Server<WorkerNetAddress>

Returns:

the server's name

getServices

public Map<ServiceType,GrpcService> getServices()

Specified by:

getServices in interface Server<WorkerNetAddress>

Returns:

a map from service names to gRPC serviced that serve RPCs for this server

start

public void start(WorkerNetAddress address)
           throws IOException

Description copied from interface: Server

Starts the Alluxio server.

Specified by:

start in interface Server<WorkerNetAddress>

Overrides:

start in class AbstractWorker

Parameters:

address - the start options

Throws:

IOException

stop

public void stop()
          throws IOException

Description copied from interface: Server

Stops the Alluxio server. Here, anything created or started in #start(T) should be cleaned up and shutdown.

Specified by:

stop in interface Server<WorkerNetAddress>

Overrides:

stop in class AbstractWorker

Throws:

IOException

close

public void close()
           throws IOException

Description copied from interface: Server

Closes the server.

Specified by:

close in interface Server<WorkerNetAddress>

Overrides:

close in class AbstractWorker

Throws:

IOException

getWorkerId

public AtomicReference<Long> getWorkerId()

Specified by:

getWorkerId in interface DataWorker

Returns:

the worker id

listStatus

@Nullable
public UfsStatus[] listStatus(String path,
                                        ListStatusPOptions options)
                                 throws IOException,
                                        AccessControlException

Description copied from interface: DoraWorker

List status from Under File System. Please refer to UnderFileSystem.listStatus().

Specified by:

listStatus in interface DoraWorker

Parameters:

path - the path of a dir or file

options - the option for listStatus()

Returns:

An array with the statuses of the files and directories in the directory denoted by this abstract pathname. The array will be empty if the directory is empty. Returns null if this abstract pathname does not denote a directory.

Throws:

IOException

AccessControlException

getFileInfo

public FileInfo getFileInfo(String ufsFullPath,
                            GetStatusPOptions options)
                     throws IOException,
                            AccessControlException

Description copied from interface: DoraWorker

Gets the file information.

Specified by:

getFileInfo in interface DoraWorker

Parameters:

ufsFullPath - the file id

options - the options for the GetStatusPRequest

Returns:

the file info

Throws:

IOException

AccessControlException

getGrpcFileInfo

protected FileInfo getGrpcFileInfo(String ufsFullPath,
                                   long syncIntervalMs)
                            throws IOException

Throws:

IOException

getCachedPercentage

protected int getCachedPercentage(FileInfo fi,
                                  String ufsFullPath)

buildFileInfoFromUfsStatus

public FileInfo buildFileInfoFromUfsStatus(UfsStatus status,
                                           String ufsFullPath)

Build FileInfo from UfsStatus and UFS full Path.

Parameters:

status -

ufsFullPath -

Returns:

a FileInfo

buildFileStatusFromUfsStatus

public DoraMeta.FileStatus buildFileStatusFromUfsStatus(UfsStatus status,
                                                        String ufsFullPath)

Build FileStatus from UfsStatus and UFS full Path.

Parameters:

status - the ufs status

ufsFullPath - the full ufs path

Returns:

the file status

createFileReader

public BlockReader createFileReader(String fileId,
                                    long offset,
                                    boolean positionShort,
                                    Protocol.OpenUfsBlockOptions options)
                             throws IOException,
                                    AccessControlException

Description copied from interface: DoraWorker

Creates the file reader to read from Alluxio dora. Owner of this block reader must close it or lock will leak.

Specified by:

createFileReader in interface DoraWorker

Parameters:

fileId - the ID of the UFS file

offset - the offset within the block

positionShort - whether the operation is using positioned read to a small buffer size

options - the options

Returns:

a block reader to read data from

Throws:

IOException - if it fails to get block reader

AccessControlException

createFileWriter

public BlockWriter createFileWriter(String fileId,
                                    String ufsPath)
                             throws AccessControlException,
                                    IOException

Description copied from interface: DoraWorker

Creates the file writer to write to Alluxio dora. Owner of this block writer must close it or lock will leak.

Specified by:

createFileWriter in interface DoraWorker

Parameters:

fileId - the ID of the UFS file

ufsPath - the path of UFS file

Returns:

the block writer for the local file

Throws:

AccessControlException

IOException

load

public com.google.common.util.concurrent.ListenableFuture<LoadFileResponse> load(boolean loadData,
                                                                                 boolean skipIfExists,
                                                                                 List<UfsStatus> ufsStatuses,
                                                                                 UfsReadOptions options)
                                                                          throws AccessControlException,
                                                                                 IOException

Description copied from interface: DoraWorker

Loads the metadata and data of files from UFS to Alluxio.

Specified by:

load in interface DoraWorker

Parameters:

loadData - true if data should also be loaded, otherwise metadata only

skipIfExists - true if data loading should be skipped if it's already loaded

ufsStatuses - the files to load

Returns:

a list of failed files

Throws:

AccessControlException

IOException

loadData

protected void loadData(String ufsPath,
                        long mountId,
                        long length)
                 throws AccessControlException,
                        IOException

Throws:

AccessControlException

IOException

copy

public com.google.common.util.concurrent.ListenableFuture<List<RouteFailure>> copy(List<Route> routes,
                                                                                   UfsReadOptions readOptions,
                                                                                   WriteOptions writeOptions)

Description copied from interface: DoraWorker

Copies files from src to dst.

Specified by:

copy in interface DoraWorker

Parameters:

routes - the files to copy

readOptions - the options for reading

writeOptions - the options for writing

Returns:

a list of failed files

getUnderFileSystem

protected UnderFileSystem getUnderFileSystem(String ufsPath)

move

public com.google.common.util.concurrent.ListenableFuture<List<RouteFailure>> move(List<Route> routes,
                                                                                   UfsReadOptions readOptions,
                                                                                   WriteOptions writeOptions)

Description copied from interface: DoraWorker

Moves files from src to dst.

Specified by:

move in interface DoraWorker

Parameters:

routes - the files to move

readOptions - the options for reading

writeOptions - the options for writing

Returns:

a list of failed files

createFile

public OpenFileHandle createFile(String path,
                                 CreateFilePOptions options)
                          throws AccessControlException,
                                 IOException

Description copied from interface: DoraWorker

Create File.

Specified by:

createFile in interface DoraWorker

Parameters:

path - the path of this file

options - the options for this operation

Returns:

a open file handle including a unique uuid

Throws:

AccessControlException

IOException

completeFile

public void completeFile(String path,
                         CompleteFilePOptions options,
                         String uuid)
                  throws IOException,
                         AccessControlException

Description copied from interface: DoraWorker

Complete writing a file. This operation will verify the supplied uuid matching its open handle.

Specified by:

completeFile in interface DoraWorker

Parameters:

path - the path of this file

options - the options for this operation

uuid - the uuid string of its file open handle

Throws:

IOException

AccessControlException

delete

public void delete(String path,
                   DeletePOptions options)
            throws IOException,
                   AccessControlException

Description copied from interface: DoraWorker

Delete a file.

Specified by:

delete in interface DoraWorker

Parameters:

path - the path of this file

options - the options for this operation

Throws:

IOException

AccessControlException

rename

public void rename(String src,
                   String dst,
                   RenamePOptions options)
            throws IOException,
                   AccessControlException

Description copied from interface: DoraWorker

Rename src to dst.

Specified by:

rename in interface DoraWorker

Parameters:

src - the source file/dir

dst - the destination file/dir

options - the options for this operations

Throws:

IOException

AccessControlException

createDirectory

public void createDirectory(String path,
                            CreateDirectoryPOptions options)
                     throws IOException,
                            AccessControlException

Description copied from interface: DoraWorker

Create a directory.

Specified by:

createDirectory in interface DoraWorker

Parameters:

path - the directory name

options - the options for this operations

Throws:

IOException

AccessControlException

exists

public boolean exists(String path,
                      ExistsPOptions options)
               throws IOException

Description copied from interface: DoraWorker

Check existence of a file or dir.

Specified by:

exists in interface DoraWorker

Parameters:

path - the path of this file or dir

options - the options of this operation

Returns:

true if the file or dir exists, false otherwise

Throws:

IOException

setAttribute

public void setAttribute(String path,
                         SetAttributePOptions options)
                  throws IOException

Set attribute for this file/dir. Please note, at this moment, the options::recursive is ignored. TODO(hua)

Specified by:

setAttribute in interface DoraWorker

Parameters:

path - the pth of this file or dir

options - the options of this operation

Throws:

IOException

cleanupSession

public void cleanupSession(long sessionId)

Description copied from interface: SessionCleanable

Cleans up the session.

Specified by:

cleanupSession in interface SessionCleanable

Parameters:

sessionId - the session id

checkCopyPermission

protected void checkCopyPermission(String srcPath,
                                   String dstPath)
                            throws AccessControlException,
                                   IOException

Throws:

AccessControlException

IOException

checkMovePermission

protected void checkMovePermission(String srcPath,
                                   String dstPath)
                            throws AccessControlException,
                                   IOException

Throws:

AccessControlException

IOException

getOpenFileHandleContainer

protected DoraOpenFileHandleContainer getOpenFileHandleContainer()

getAddress

public WorkerNetAddress getAddress()

Description copied from interface: DoraWorker

Get the address of the Dora Worker.

Specified by:

getAddress in interface DoraWorker

Returns:

worker address