alluxio.master.journal.raft

Class RaftJournalSystem

java.lang.Object

alluxio.master.journal.AbstractJournalSystem

alluxio.master.journal.raft.RaftJournalSystem

All Implemented Interfaces:

JournalSystem

@ThreadSafe
public final class RaftJournalSystem
extends AbstractJournalSystem

System for multiplexing many logical journals into a single raft-based journal. This class embeds a CopycatServer which implements the raft algorithm, replicating entries across a majority of servers before applying them to the state machine. To make the Copycat system work as an Alluxio journal system, we implement two non-standard behaviors: (1) pre-applying operations on the primary and (2) tightly controlling primary snapshotting.

Pre-apply

Unlike the Copycat framework, Alluxio updates state machine state *before* writing to the journal. This lets us avoid journaling operations which do not result in state modification. To make this work in the Copycat framework, we allow RPCs to modify state directly, then write an entry to Copycat afterwards. Once the entry is journaled, Copycat will attempt to apply the journal entry to each master. The entry has already been applied on the primary, so we treat all journal entries applied to the primary as no-ops to avoid double-application.

Correctness of pre-apply

There are two cases to worry about: (1) incorrectly ignoring an entry and (2) incorrectly applying an entry.

Avoid incorrectly ignoring entries

This could happen if a server thinks it is the primary, and ignores a journal entry served from the real primary. To prevent this, primaries wait for a quiet period before serving requests. During this time, the previous primary will go through at least two election cycles without successfully sending heartbeats to the majority of the cluster. If the old primary successfully sent a heartbeat to a node which elected the new primary, the old primary would realize it isn't primary and step down. Therefore, the old primary will step down and no longer ignore requests by the time the new primary begins sending entries.

Avoid incorrectly applying entries

Entries can never be double-applied to a primary's state because as long as it is the primary, it will ignore all entries, and once it becomes secondary, it will completely reset its state and rejoin the cluster.

Snapshot control

The way we apply journal entries to the primary makes it tricky to perform primary state snapshots. Normally Copycat would decide when it wants a snapshot, but with the pre-apply protocol we may be in the middle of modifying state when the snapshot would happen. To manage this, we inject an AtomicBoolean into Copycat which decides whether it will be allowed to take snapshots. Normally, snapshots are prohibited on the primary. However, we don't want the primary's log to grow unbounded, so we allow a snapshot to be taken once a day at a user-configured time. To support this, all state changes must first acquire a read lock, and snapshotting requires the corresponding write lock. Once we have the write lock for all state machines, we enable snapshots in Copycat through our AtomicBoolean, then wait for any snapshot to complete.

Nested Class Summary

Nested classes/interfaces inherited from interface alluxio.master.journal.JournalSystem

JournalSystem.Builder, JournalSystem.Mode

Method Summary

Modifier and Type	Method and Description
void	checkpoint() Creates a checkpoint in the primary master journal system.
static RaftJournalSystem	create(RaftJournalConfiguration conf) Creates and initializes a raft journal system.
Journal	createJournal(Master master) Creates a journal for the given state machine.
static io.atomix.catalyst.serializer.Serializer	createSerializer()
void	format() Formats the journal system.
void	gainPrimacy() Transitions the journal to primary mode.
PrimarySelector	getPrimarySelector()
List<QuorumServerInfo>	getQuorumServerInfoList() Used to get information of internal RAFT quorum.
boolean	isEmpty() Returns whether the journal is formatted and has not had any entries written to it yet.
boolean	isFormatted()
void	losePrimacy() Transitions the journal to secondary mode.
void	removeQuorumServer(NetAddress serverNetAddress) Removes from RAFT quorum, a server with given address.
void	startInternal() Starts the journal system.
void	stopInternal() Stops the journal system.

Methods inherited from class alluxio.master.journal.AbstractJournalSystem

addJournalSink, getJournalSinks, removeJournalSink, start, stop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

create

public static RaftJournalSystem create(RaftJournalConfiguration conf)

Creates and initializes a raft journal system.

Parameters:

conf - raft journal configuration

Returns:

the created raft journal system

createSerializer

public static io.atomix.catalyst.serializer.Serializer createSerializer()

Returns:

the serializer for commands in the StateMachine

createJournal

public Journal createJournal(Master master)

Description copied from interface: JournalSystem

Creates a journal for the given state machine. The returned journal can create journal contexts for writing journal entries. However, no entries may be written until the journal system has been started, and entries may only be written when the journal system is in PRIMARY mode. When the journal is started in secondary mode, it will call Journaled#processJournalEntry(JournalEntry) and Journaled.resetState() to keep the state machine's state in sync with the entries written to the journal.

Parameters:

master - the master to create the journal for

Returns:

a new instance of Journal

gainPrimacy

public void gainPrimacy()

Description copied from interface: JournalSystem

Transitions the journal to primary mode.

losePrimacy

public void losePrimacy()

Description copied from interface: JournalSystem

Transitions the journal to secondary mode.

checkpoint

public void checkpoint()
                throws IOException

Description copied from interface: JournalSystem

Creates a checkpoint in the primary master journal system.

Throws:

IOException

startInternal

public void startInternal()
                   throws InterruptedException,
                          IOException

Description copied from class: AbstractJournalSystem

Starts the journal system.

Specified by:

startInternal in class AbstractJournalSystem

Throws:

InterruptedException

IOException

stopInternal

public void stopInternal()
                  throws InterruptedException,
                         IOException

Description copied from class: AbstractJournalSystem

Stops the journal system.

Specified by:

stopInternal in class AbstractJournalSystem

Throws:

InterruptedException

IOException

getQuorumServerInfoList

public List<QuorumServerInfo> getQuorumServerInfoList()

Used to get information of internal RAFT quorum.

Returns:

list of information for participating servers in RAFT quorum

removeQuorumServer

public void removeQuorumServer(NetAddress serverNetAddress)
                        throws IOException

Removes from RAFT quorum, a server with given address. For server to be removed, it should be in unavailable state in quorum.

Parameters:

serverNetAddress - address of the server to remove from the quorum

Throws:

IOException

isEmpty

public boolean isEmpty()

Description copied from interface: JournalSystem

Returns whether the journal is formatted and has not had any entries written to it yet. This can only be determined when the journal system is in primary mode because entries are written to the primary first.

Returns:

whether the journal system is freshly formatted

isFormatted

public boolean isFormatted()

Returns:

whether the journal system has been formatted

format

public void format()
            throws IOException

Description copied from interface: JournalSystem

Formats the journal system.

Throws:

IOException

getPrimarySelector

public PrimarySelector getPrimarySelector()

Returns:

a primary selector backed by leadership within the Raft cluster