Systems and methods for high-availability witness sleds
Systems and methods provide failover operations for witness sled hardware installed in a chassis. The chassis may include data storage devices and Information Handling Systems (IHSs), each storing data redundantly to the storage devices, such as in a vSAN. A primary witness sled provides witness functions, such as arbitrating discrepancies in the redundantly stored data. The chassis includes a chassis management controller (CMC) that connects with the primary witness sled via a management signaling pathway and maintains a copy of witness transactions stored by the primary witness sled based on transactions transmitted via the management signaling pathway. The CMC generates an inventory of co-located witness sleds. Upon detecting the primary witness sled is not operational, the CMC selects a failover witness sled from the inventory of co-located witness sleds and migrates witness operations of the primary witness sled to the selected failover witness sled.
1. A first chassis comprising:
a plurality of data storage devices;
one or more Information Handling Systems (IHSs), each comprising one or more CPUs (Central Processing Units) and further comprising one or more memory devices having instructions stored thereon that, upon execution by a CPU, respectively cause each IHS to store data redundantly to the plurality of data storage devices;
a primary witness sled comprising a logic unit and a memory device having instructions stored thereon that, upon execution by the logic unit, cause the primary witness sled to arbitrate discrepancies in the data redundantly stored in the plurality of storage devices; and
a chassis management controller comprising a logic unit and a memory device having instructions stored thereon, wherein execution of the instructions causes the logic unit to:
initiate a connection with the primary witness sled via a management signaling pathway;
maintain a copy of witness transactions stored by the primary witness sled based on witness transactions transmitted by the primary witness sled to the chassis management controller via the management signaling pathway;
generate an inventory of co-located witness sleds;
detect that the primary witness sled is not operational;
select a failover witness sled from the inventory of co-located witness sleds; and
migrate witness operations of the primary witness sled to the selected failover witness sled using the copy of witness transactions maintained by the chassis management controller.
2. The chassis of claim 1 , wherein the IHSs store data redundantly to the plurality of data storage devices according to instructions of a vSAN application.
3. The chassis of claim 1 , wherein the management signaling pathway comprises a management connection between the chassis management controller and a microcontroller of the primary witness sled.
4. The chassis of claim 3 , wherein the management signaling pathway comprises a signaling pathway used for I2C communications between the chassis management controller and the microcontroller of the primary witness sled.
5. The chassis of claim 4 , wherein the initiated connection emulates a USB connection via the management signaling pathway.
6. The chassis of claim 5 , wherein the witness data is transmitted by the microcontroller of the primary witness sled via the emulated USB connection.
7. The chassis of claim 1 , wherein the witness transactions comprise a report of a modification to a data element of the redundantly stored data.
8. The chassis of claim 1 , wherein the witness transactions comprise a request to arbitrate a discrepancy in the redundantly stored data.
9. The chassis of claim 8 , wherein the selected failover witness sled utilizes the migrated copy of the witness transactions to arbitrate the discrepancy once the primary witness sled is detected is not operational.
10. The chassis of claim 1 , wherein the chassis management controller tracks the witness transactions processed from the one or more IHSs while the primary witness sled is not operational.
11. The chassis of claim 10 , wherein the chassis management controller notifies the primary witness sled of the tracked witness transactions upon detecting that the primary witness sled has resumed normal operations.
12. The chassis of claim 1 , wherein the detected fault is identified by the chassis management controller based on a lapse in a heartbeat signal from the primary witness sled.
13. A chassis management controller installed within a chassis comprising one or more IHSs (Information Handling Systems) and further comprising a plurality of data storage devices, the chassis management controller comprising:
a logic unit; and
a memory device having instructions stored thereon that, upon execution by the logic unit, cause the chassis management controller to:
initiate a connection with a primary witness sled via a management signaling pathway, wherein the primary witness sled comprises a logic unit and a memory device having instructions stored thereon that, upon execution by the logic unit, cause the primary witness sled to arbitrate discrepancies in data redundantly stored in the plurality of storage devices;
maintain a copy of witness transactions stored by the primary witness sled based on witness transactions transmitted by the primary witness sled to the chassis management controller via the management signaling pathway;
generate an inventory of co-located witness sleds;
detect that the primary witness sled is not operational;
select a failover witness sled from the inventory of co-located witness sleds; and
migrate witness operations of the primary witness sled to the selected failover witness sled using the copy of witness transactions maintained by the chassis management controller.
14. The chassis management controller of claim 13 , wherein the IHSs store data redundantly to the plurality of data storage devices according to instructions of a vSAN application.
15. The chassis management controller of claim 13 , wherein the management signaling pathway comprises a management connection between the chassis management controller and a microcontroller of the primary witness sled.
16. The chassis management controller of claim 13 , wherein the chassis management controller tracks the witness transactions processed from the one or more IHSs while the primary witness sled is not operational.
17. A computer-readable storage device having instructions stored thereon for management of a chassis comprising one or more IHSs (Information Handling Systems) and further comprising a plurality of data storage devices, wherein execution of the instructions by one or more processors the chassis causes the one or more processors to:
initiate a connection with a primary witness sled via a management signaling pathway, wherein the primary witness sled comprises a logic unit and a memory device having instructions stored thereon that, upon execution by the logic unit, cause the primary witness sled to arbitrate discrepancies in data redundantly stored in the plurality of storage devices;
maintain a copy of witness transactions stored by the primary witness sled based on witness transactions transmitted by the primary witness sled to the chassis management controller via the management signaling pathway;
generate an inventory of co-located witness sleds;
detect the primary witness sled is not operational;
select a failover witness sled from the inventory of co-located witness sleds; and
migrate witness operations of the primary witness sled to the selected failover witness sled using the copy of witness transactions maintained by the chassis management controller.
18. The computer-readable storage device of claim 17 , wherein the IHSs store data redundantly to the plurality of data storage devices according to instructions of a vSAN application.
19. The computer-readable storage device of claim 17 , wherein the management signaling pathway comprises a management connection between the chassis management controller and a microcontroller of the primary witness sled.
20. The computer-readable storage device of claim 17 , wherein the chassis management controller tracks the witness transactions processed from the one or more IHSs while the primary witness sled is not operational.