Automatic recovery of locking filesystem shared between virtual tape library nodes
A virtual tape library system includes multiple copies of a LockFS that are mounted on all nodes. One of the LockFSs is designated as the active copy. A primary node manages recovery when a secondary node signals loss of connectivity with the active copy by posting a sync request file to at least one of the LockFSs. The primary node posts status request files to the LockFSs and the secondary nodes respond by writing status to the status request files. A LockFS that is reachable by all nodes is selected as the new active LockFS. Switchover is accomplished by the primary node posting a disable request file to the selected LockFS and waiting for all secondary nodes to acknowledge by writing to the disable request file. The primary node then posts an activate request file to the selected LockFS and waits for all secondary nodes to acknowledge by writing to the activate request file.
1. A virtual tape library system, comprising:
a primary virtual tape server comprising a non-transitory memory with program code configured to:
select an active copy of a shared locking filesystem from a plurality of copies of the shared locking filesystem, the active copy of the shared locking filesystem and the plurality of copies of the shared locking filesystem each being stored on different non-volatile drives;
receive tape drive utility access commands;
responsive to the tape drive utility access commands, use the active copy of the shared locking filesystem to generate corresponding input-output commands to access storage resources; and
a plurality of secondary virtual tape servers comprising non-transitory memory with program code configured to:
receive tape drive utility access commands;
responsive to the tape drive utility access commands, use the active copy of the shared locking filesystem selected by the primary virtual tape server to generate corresponding input-output commands to access storage resources; and
signal to the primary virtual tape server responsive to the active copy of the shared locking filesystem becoming unreachable from the non-volatile drive on which the active copy of the shared locking filesystem is stored, the signaling prompting one of the plurality of copies of the shared locking filesystem to replace the active copy of the shared locking filesystem to generate input-output commands to access storage resources;
wherein ones of the secondary virtual tape servers are configured to signal that the active copy of the shared locking filesystem has become unreachable by posting a synchronization request to at least one of the copies of the shared locking filesystem that is reachable;
wherein the primary virtual tape server is responsive to the synchronization request to post status request messages to all reachable copies of the shared locking filesystem; and
wherein the secondary virtual tape servers are responsive to the status request messages to post status messages to reachable copies of the shared locking filesystem indicating availability, including use as the active copy of the shared locking filesystem.
2. The virtual tape library system of claim 1 wherein the primary node is responsive to the status messages to select a different one of the copies of the shared locking filesystem as the active copy.
3. The virtual tape library system of claim 2 wherein the primary node posts a disable request to the different one of the copies of the shared locking filesystem.
4. The virtual tape library system of claim 3 wherein the secondary nodes are responsive to the disable request to enter a limited functionality node and cease use of the active copy of the shared locking filesystem.
5. The virtual tape library system of claim 4 wherein the primary node posts an activate request to the different one of the copies of the shared locking filesystem.
6. The virtual tape library system of claim 5 wherein the secondary nodes are responsive to the activate request to exit the limited functionality node and begin use of the different one of the copies of the shared locking filesystem as a new active copy of the shared locking filesystem.
7. A method for shared locking filesystem recovery in a virtual tape library system, comprising:
a primary virtual tape server comprising a non-transitory memory with program code:
selecting an active copy of a shared locking filesystem from a plurality of copies of the shared locking filesystem, the active copy of the shared locking filesystem and the plurality of copies of the shared locking filesystem each being stored on different non-volatile drives;
receiving tape drive utility access commands;
responsive to the tape drive utility access commands, using the active copy of the shared locking filesystem for generating corresponding input-output commands to access storage resources; and
a plurality of secondary virtual tape servers comprising non-transitory memory with program code:
receiving tape drive utility access commands; and
responsive to the tape drive utility access commands, using the active copy of the shared locking filesystem selected by the primary virtual tape server for generating corresponding input-output commands to access storage resources;
at least one of the secondary virtual tape servers signaling to the primary virtual tape server responsive to the active copy of the shared locking filesystem becoming unreachable by posting a synchronization request to at least one of the copies of the shared locking filesystem that is reachable; and
in response to the signaling by the at least one of the secondary virtual tape servers, the primary virtual tape server posting status request messages to all reachable copies of the shared locking filesystem responsive to the synchronization request, the secondary virtual tape servers posting status messages to reachable copies of the shared locking filesystem indicating availability, including use as the active copy of the shared locking filesystem, and the primary virtual tape server selecting a different one of the copies of the shared locking filesystem as a new active copy of the shared locking filesystem to replace the active copy of the shared locking filesystem for generating input-output commands to access storage resources.
8. The method of claim 7 comprising the primary node posting a disable request to the different one of the copies of the shared locking filesystem and the secondary nodes entering a limited functionality node and ceasing use of the active copy of the shared locking filesystem responsive to the disable request.
9. The method of claim 8 comprising the primary node posting an activate request to the different one of the copies of the shared locking filesystem and the secondary nodes exiting the limited functionality node and beginning use of the different one of the copies of the shared locking filesystem as a new active copy of the shared locking filesystem.