System and method of drainless link repair for increased accelerator utilization during failure

1 . A system for repair of an accelerator architecture comprising:

a plurality of accelerator hosts;

a plurality of accelerator elements in each host;

a plurality of high-speed interconnects between accelerator elements;

a processor of each host executing a link health monitor service in each host configured to determine a status of each interconnect associated with the host and communicate the link status of each interconnect;

a pod manager for dividing the plurality of accelerator elements in the plurality of hosts into a plurality of slices comprising a number of the plurality of accelerator elements, the pod manager organizing the plurality of slices into a hierarchical tree structure, wherein the pod manager receives the communicated link status of each interconnect, and identifies one or more slices in direct connection with an interconnect with a failure status; and

a scheduler in communication with the pod manager, the scheduler configured to schedule accelerator tasks to other slices than the one or more identified slices in direct connection with the interconnect having the failure status.

2 . The system of claim 1 , the plurality of high-speed interconnects comprising:

at least one internal interconnect between a first accelerator element and a second accelerator element within a same accelerator host; and

at least one external interconnect between an accelerator element in a first accelerator host, and another accelerator element in a second accelerator host.

3 . The system of claim 2 , the link health monitor service configured to detect link status of the at least one internal interconnect and the at least one external interconnect associated with a given accelerator host.

4 . The system of claim 1 ,

wherein the scheduler configured to receive user job requests and map the received user request to a slice of the accelerator architecture for execution of the user request.

5 . The system of claim 4 , the pod manager communicatively connected between the scheduler and each accelerator host.

6 . The system of claim 5 , the pod manager configured to:

responsive to a message indicating a failure in an interconnect from one or more link health monitor service, determining a least common ancestor of the hierarchical tree of slices of the accelerator architecture, the least common ancestor defining a slice containing a first accelerator element and a second accelerator unit connected by the failed interconnect; and

flagging the least common ancestor and all parent slices of the least common ancestor in the hierarchical tree of slices.

7 . The system of claim 6 , the pod manager further configured to:

communicating all flagged slices to the scheduler to be drained by the scheduler.

8 . The system of claim 6 , the pod manager further configured to:

automatically determine a corrective action to repair the failed interconnect; and

communicate the determined corrective action to a technician.

9 . The system of claim 8 , the pod manager further configured to:

validate a repair performed by the technician; and

reactivate slices associated with the repaired interconnect.

10 . The system of claim 9 , wherein if validation indicates the repair was unsuccessful, the pod manager configured to automatically re-determine a second corrective action to repair the failed interconnect.

11 . The system of claim 7 , the scheduler configured to:

identify any uncompleted tasks assigned to a flagged slice; and

reschedule the uncompleted tasks to a remaining active slice.

12 . The system of claim 1 , each accelerator host comprising:

a compute processor; and

a firmware memory in communication with the compute processor, the firmware memory storing instructions that when executed by the compute processor, cause the compute processor to allow the accelerator host to continue running when an interconnect is removed or plugged into the accelerator host.

13 . A method for troubleshooting an accelerator architecture comprising a plurality of accelerator hosts, the method comprising:

automatically detecting a failure in an interconnect connecting two accelerator elements in the accelerator architecture in a health monitor service executed by a processor in each of the plurality of accelerator hosts;

in the link health monitor service, identifying a first accelerator element and a second accelerator element connected by the failed interconnect; and

identifying in a pod manager, a slice of accelerator elements, the slice including both the first accelerator element and the second accelerator element, the slice being organized in a hierarchical tree, and disabling accelerator elements in the slice of accelerator elements and all parent nodes of the identified slice of accelerator elements; and

in a scheduler, sending accelerator tasks to an accelerator element in a different slice than the slice including both the first and second accelerator elements.

14 . The method of claim 13 , further comprising:

marking, by the pod manager the identified slice as inactive; and

preventing scheduling of any processing task on the marked slice.

15 . The method of claim 14 , further comprising:

in the pod manager, automatically identifying a corrective action for the failed interconnect; and

communicating the corrective action to a technician.

16 . The method of claim 15 , further comprising:

upon completion of the corrective action, in the pod manager, validating a repair of the failed interconnect; and

on a condition that repair was successful, reactivating the inactive slice associated with the failed interconnect and on a condition that the repair was unsuccessful, identifying a second corrective action.

17 . The method of claim 16 , further comprising:

in the pod manager, notifying the scheduler of an updated status of the affected slice.

18 . The method of claim 17 , further comprising:

after identifying a slice of accelerator elements, the slice including both the first accelerator element and the second accelerator element, the slice being organized in a hierarchical tree, further identifying all parent slices containing the identified slice of accelerator elements in the hierarchical tree, and marking the parent slices as inactive.

19 . A non-transitory computer readable medium storing instructions executable by a computer processor, that when executed by the computer processor, cause the computer processor to:

in an accelerator architecture comprising a plurality of accelerator components connected to each other through high-speed interconnects, detecting a failure of at least one of the interconnects;

communicating the detected failure of the interconnect to a pod manager;

in the pod manager, determining a slice defining a group of connected accelerator components containing a first accelerator component and a second accelerator component connected by the failed interconnect, the slices arranged in a hierarchical tree;

marking the determined slice and all parent slices of the determined slice in the hierarchical tree as affected by the failed interconnect; and

identifying a scheduler of the affected slices, wherein the scheduler does not schedule any processing tasks to the affected slices.