Timer bounded arbitration protocol for resource control
View Patent ↗Example apparatus, methods, and computers prevent a split brain scenario in a pair of high availability servers by maintaining single writer access to a resource by controlling the resource according to a timer bounded arbitration protocol that controls self-termination of a writer process. One example method includes monitoring control of an arbitration (ARB) block by an active file system manager (FSM) and selectively causing a selection of a standby metadata controller (MDC) when control of the ARB block does not satisfy the timer bounded ARB protocol. The example method also includes selectively forcing a hardware reset of an apparatus running the active FSM and selectively establishing an FSM on a selected redundant MDC as a replacement FSM.
1. An apparatus, comprising:
a processor;
a memory; and
an interface to connect the processor, memory, and a set of logics, the set of logics comprising:
a metadata controller (MDC) logic configured to selectively terminate an active File System Manager (FSM) process running on the apparatus and to selectively activate a standby FSM process available on a redundant MDC upon determining that the active FSM process has violated a timer bounded arbitration (ARB) protocol, where the timer bounded ARB protocol relies on an ARB block acting as a resource shared between the active FSM process and the standby FSM process; and
an ARB logic configured to maintain single writer access to file system metadata using the ARB block, and configured to control the active FSM process and the standby FSM process to synchronize control of the ARB block by using the ARB block,
where an MDC housing the active FSM comprises:
an active file system portmapper (FSMPM);
an active file system manager (FSM) connected to the active FSMPM by a socket; and
an active timer,
where the active FSMPM is configured to selectively force an election of an FSM process to replace the active FSM process upon the metadata controller logic determining that the active FSM process has not maintained control according to the timer bounded ARB protocol,
where the active FSM process is configured to maintain ownership of the ARB by periodically writing the ARB according to the timer bounded ARB protocol and to restart the active timer upon successfully writing the ARB,
where an MDC housing the standby FSM comprises:
a standby FSMPM;
a standby FSM process connected to the standby FSMPM by a socket; and
a standby timer,
where the standby FSMPM is configured to activate the standby FSM process after being elected to replace the active FSM process, and
where the standby FSM process is configured to acquire ownership of the ARB, to start the standby timer, to maintain ownership of the ARB by periodically writing the ARB according to the timer bounded ARB protocol, and to restart the standby timer upon successfully writing the ARB.
2. The apparatus of claim 1 , the ARB logic being configured to communicate with a data store, where the data store stores the ARB, at least a portion of a file system associated with the file system metadata, and the file system metadata.
3. The apparatus of claim 2 , where the active FSM process and the standby FSM process reside on separate pieces of computer hardware, and where the active FSM process and the standby FSM process communicate over a computer network.
4. The apparatus of claim 3 , where the apparatus, the data store, and a client computer are connected by a computer network, and where the client computer is configured to write the file system data under control of the active FSM process.
5. The apparatus of claim 1 , the timer bounded arbitration protocol comprising:
the active FSM process writing the ARB once per FSM write period to indicate continued ownership of the ARB and, upon successfully writing the ARB, restarting the active timer to a reset threshold period;
the active timer causing a hardware reset of the apparatus upon expiring after an amount of time exceeding the reset threshold period;
the FSMPM of the redundant FSM process selectively activating the standby FSM process upon determining that the standby FSM process has been selected to take control of the ARB; and
upon being activated, the standby FSM process, selectively taking control of the ARB by writing the ARB, and by waiting a delay time period before writing metadata.
6. The apparatus of claim 5 , where the FSM write period is 0.5 seconds, where the reset threshold period is 5 seconds, and where the delay time period is the reset threshold period plus twice the FSM write period.
7. The apparatus of claim 1 , where the metadata controller logic is configured to control a kernel thread to reset an MDC housing the active FSM.
8. The apparatus of claim 1 , where there is a one to one relationship between the ARB and the active FSM process, and where there is a one to one relationship between the active FSM process and a file system associated with the file system metadata.
9. The apparatus of claim 1 , where the metadata controller logic is configured to terminate the active FSM process before the standby FSM process writes to the metadata.
10. The apparatus of claim 1 , where the active timer is one of, a kernel timer, an operating system timer, and a timer associated with computer hardware operatively connected to the interface, where there is one timer per active FSM process, where the standby timer is one of, a kernel timer, an operating system timer, and a timer associated with computer hardware operatively connected to the interface, and where there is one timer per standby FSM process.
11. The apparatus of claim 1 , the metadata controller logic being configured to force a hardware reset of the apparatus in response to the closing of an operating system special file associated with the active timer, where the operating system special file is closed upon the termination of the active FSM process.
12. The apparatus of claim 11 , the metadata controller logic being configured to selectively not force the hardware reset of the apparatus in response to the closing of an operating system special file associated with the active timer, upon determining that an operating system unmount of a file system associated with the file system metadata has completed successfully leaving no active data writers on the MDC.
13. A non-transitory computer readable medium storing computer executable instructions that when executed by a computer control the computer to perform a method, the method comprising:
monitoring control of an arbitration (ARB) block by an active file system manager (FSM);
selectively causing a selection of a redundant metadata controller (MDC) when control of the ARB block does not satisfy a timer bounded ARB protocol;
selectively forcing a hardware reset of an apparatus running the active FSM upon expiration of a timer; and
selectively activating an FSM on a redundant MDC as a replacement FSM,
where control of the ARB block by the active FSM comprises:
the active FSM reading the ARB block;
the active FSM writing the ARB block; and
the active FSM restarting a bounding timer upon successfully writing the ARB block.
14. The computer readable medium of claim 13 ,
where forcing the hardware reset is performed without a communication with the redundant MDC and without an action by the active FSM.
15. The computer readable medium of claim 13 , where selectively establishing the FSM on the redundant MDC as the replacement FSM comprises:
sending an activation command to the replacement FSM;
controlling the replacement FSM to monitor the ARB block for a safety period of time; and
controlling the replacement FSM to write the ARB block after determining that the active FSM has not written the ARB during the safety period of time; and
controlling the replacement FSM to start a timer on the replacement FSM after writing the ARB block.
16. The computer readable medium of claim 15 , where selectively establishing the FSM on the redundant MDC as the replacement FSM also comprises controlling the FSM on the redundant MDC to write the ARB block after determining that a state on the FSM on the redundant MDC exceeds the state of other FSMs available to write the ARB block, and controlling the FSM on the redundant MDC to start its timer after writing the ARB block.