Snapshot message
View Patent ↗A blockage is detected at a first link based on a delay and/or stoppage of transmission of a data message along the first link between first and second nodes of a plurality of nodes of a fabric. A snapshot message is sent along at least a second link between the first and second nodes in response to the blockage being detected. The second node may capture a fabric state at the second node in response to receiving the snapshot message, before a corrective action occurs.
1. A device, comprising:
a processor; and
a non-transitory storage medium storing instructions executable on the processor to:
detect a blockage at a first link based on at least one of a delay and stoppage of transmission of a data message along the first link between first and second nodes of a plurality of nodes of a fabric;
broadcast a snapshot message to the plurality of nodes along at least a second link between the first and second nodes in response to the blockage being detected, the snapshot message to cause the plurality of nodes to capture fabric states at the plurality of nodes in response to receiving the snapshot message, and before a corrective action occurs; and
debug a cause of the blockage based on the captured fabric states; and
trigger the corrective action at a threshold time from a time associated with the detecting of the blockage, the threshold time greater than or equal to a time for the snapshot message to propagate to the plurality of nodes.
2. The device of claim 1 , wherein each fabric state of the fabric states comprises at least one of flow control and arbitration values for at least one of entry and exit points of a respective node of the plurality of nodes.
3. The device of claim 2 , wherein the at least one of flow control and arbitration values includes information related to at least one of credits, queue depth, starvation, and backpressure.
4. The device of claim 1 , wherein the broadcasting of the snapshot message comprises:
the second node forwarding the snapshot message to a third node of the plurality of nodes before the corrective action is taken.
5. The device of claim 1 , wherein the threshold time is a first threshold time, and wherein the detecting of the blockage is based on at least one of a node endpoint and flow control timeout exceeding a second threshold time.
6. The device of claim 5 , wherein the triggering of the corrective action occurs at the first threshold time after the second threshold time.
7. The device of claim 6 , wherein the instructions are executable on the processor to
take the corrective action in response to the triggering, and wherein the corrective action includes at least one of containment and failover.
8. The device of claim 1 ,
wherein the debugging comprises reading values from logs to which information of the fabric states are written.
9. The device of claim 1 , wherein,
the second link is at least one a virtual channel and a sideband bus,
the second link provides a path redundant to the first link, and
the fabric includes a path between components of at least one of the device and a network.
10. The device of claim 1 , wherein the captured fabric states comprise information about at least one selected from among queue depth, starvation, and backpressure.
11. A method, comprising:
detecting a blockage at a first link between first and second nodes of a plurality of nodes of a fabric;
broadcasting a snapshot message to the plurality of nodes via a second link between the first and second nodes, the snapshot message to cause the plurality of nodes to capture respective fabric states of the fabric in response to receiving the snapshot message;
debugging a cause of the blockage based on the captured fabric states; and
triggering a corrective action for the blocked first link at a threshold time from a time associated with the detecting of the blockage, the threshold time greater than or equal to a time for the snapshot message to propagate to the plurality of nodes.
12. The method of claim 11 , wherein the detecting of the blockage includes detecting at least one of a node endpoint and flow control timeout, and wherein information of the fabric states includes at least one of flow control and arbitration values.
13. The method of claim 12 , wherein:
the plurality of nodes capture the fabric states by capturing data from one or more registers internal to each of the plurality of nodes, and
the data of the one or more registers changes after the corrective action is triggered.
14. The method of claim 11 , wherein each of the captured fabric states comprises information about a queue depth relating to a number of pending messages awaiting input or output at a respective node of the plurality of nodes.
15. The method of claim 11 , wherein each of the captured fabric states comprises information about starvation and backpressure at a respective node of the plurality of nodes.
16. A non-transitory computer-readable storage medium storing instructions that, if executed by a processor of a device, cause the processor to:
broadcast a snapshot message to a plurality of nodes via a second link between first and second nodes of the plurality of nodes, if blockage is detected at a first link between the first and second nodes; and
delay a corrective action for the first link until the snapshot message reaches the plurality of nodes, wherein
the plurality of nodes are to capture a fabric state in response to receiving the snapshot message.
17. The non-transitory computer-readable storage medium of claim 16 , wherein,
the plurality of nodes are to capture at least one of flow control and arbitration values when capturing the fabric state, and
the captured fabric state is stored in a log to be read out for debugging.
18. The non-transitory computer-readable storage medium of claim 16 , wherein the instructions if executed cause the processor to:
debug a cause of the blockage based on the captured fabric state.
19. The non-transitory computer-readable storage medium of claim 16 , wherein the instructions if executed cause the processor to:
trigger the corrective action at a threshold time after a time associated with the detecting of the blockage, the threshold time greater than or equal to a time for the snapshot message to propagate to the plurality of nodes.
20. The non-transitory computer-readable storage medium of claim 16 , wherein the captured fabric state comprises information about a queue depth relating to a number of pending messages awaiting input or output at each respective node of the plurality of nodes.
21. The non-transitory computer-readable storage medium of claim 16 , wherein the captured fabric state comprises information about starvation and backpressure at each respective node of the plurality of nodes.