Granted Patent B2

US 12,639,149 · App. 18/657,715 · Granted May 26, 2026

Inventors: Paul Jordan (Austin, TX); Manish K. Shah (Austin, TX)

Assignee: SambaNova Systems, Inc.

G06F11/0772G06F11/0757G06F11/0793H04L41/06H04L43/50

Abstract

Claims (127)

1 . A computing system comprising:

a host computer;

a first CGRP (coarse-grained reconfigurable architecture processor) coupled to the host computer through a first external communication link;

a second CGRP coupled to the host computer through a second external communication link;

the first CGRP and the second CGRP both respectively including:

an internal network;

a first external interface circuit coupled between the host computer and the internal network;

a second external interface circuit coupled between a third external communication link and the internal network;

a memory interface circuit coupled between an external memory and the internal network;

a first array of configurable units and a second array of configurable units, each coupled to the internal network and respectively comprising a plurality of processor configurable units and a plurality of memory configurable units coupled together with an array-level network;

a first hang detection circuit and a first force-quit controller associated with the first array of configurable units;

a second hang detection circuit and a second force-quit controller associated with the second array of configurable units; and

a network recovery circuit including one or more control registers that have a first application ID field associated with the first array of configurable units, a second application ID field associated with the second array of configurable units, a first hang syndrome field that indicate whether the first array of configurable units is hung, and a second hang syndrome field that indicate whether the second array of configurable units is hung;

the host computer programmed to:

send first configuration information for a first portion of a first application to the first array of configurable units in the first CGRP and set the first application ID field in the first CGRP to a value indicative of the first application;

send second configuration information for a second portion of the first application to the first array of configurable units in the second CGRP;

initiate execution of the first application using at least the first array of configurable units in both the first CGRP and the second CGRP; and

in response to receiving an interrupt from the first CGRP to the host computer indicating that the first array of configurable units in the first CGRP is hung, send a command to the second CGRP to send a force quit message to the first force-quit controller to halt execution of the first application on the first array of configurable units in the second CGRP; and

the network recovery circuit in the first CGRP, in response to receiving a hang message from the first hang detection circuit in the first CGRP over the internal network in the first CGRP, is configured to:

send a first force-quit message to the first force-quit controller in the first CGRP to halt execution by the first array of configurable units in the first CGRP;

set the first hang syndrome field to indicate that the first array of configurable units in the first CGRP is hung;

send the interrupt to the host computer to indicate that the first array of configurable units in the first CGRP is hung;

compare a first value of the first application ID field in the first CGRP to a second value of the second application ID field in the first CGRP; and

send a second force-quit message to the second force-quit controller in the first CGRP to halt execution by the second array of configurable units in the first CGRP in response to the first value being equal to the second value.

2 . The computing system of claim 1 , wherein the second external interface circuit in the first CGRP, in response to receiving a packet over the third external communication link from the second CGRP that includes information indicating a destination agent on the internal network of the first CGRP for the packet, is configured to:

in response to determining that the destination agent is associated with the first array of configurable units in the first CGRP and the first hang syndrome field in the first CGRP is not set, send the packet over the internal network in the first CGRP to the destination agent;

in response determining that the destination agent is associated with the first array of configurable units in the first CGRP and the first hang syndrome field in the first CGRP is set, discard the packet;

in response to determining that the destination agent is associated with the second array of configurable units in the first CGRP and that the second hang syndrome field in the first CGRP is not set, send the packet over the internal network in the first CGRP to the destination agent;

in response determining that the destination agent is associated with the second array of configurable units in the first CGRP and the second hang syndrome field in the first CGRP is set, discard the packet; and

in response determining that the destination agent is the memory interface circuit in the first CGRP, send the packet over the internal network in the first CGRP to memory interface circuit in the first CGRP regardless of a state of the first hang syndrome field or a state of the second hang syndrome field.

3 . A reconfigurable processor, comprising:

an internal network;

a first array of configurable units and a second array of configurable units, each coupled to the internal network and respectively comprising a plurality of processor configurable units and a plurality of memory configurable units coupled together with an array-level network;

a first hang detection circuit and a first force-quit controller associated with the first array of configurable units;

a second hang detection circuit and a second force-quit controller associated with the second array of configurable units; and

a network recovery circuit including one or more control registers that have a first application ID field associated with the first array of configurable units and a second application ID field associated with the second array of configurable units, the network recovery circuit configured to:

receive a hang message from the first hang detection circuit over the internal network;

send a first force-quit message to the first force-quit controller to halt execution by the first array of configurable units;

compare a first value of the first application ID field to a second value of the second application ID field; and

send a second force-quit message to the second force-quit controller to halt execution by the second array of configurable units in response to the first value being equal to the second value.

4 . The reconfigurable processor of claim 3 , further comprising a first external interface circuit coupled between the internal network and an external communication link, the first external interface circuit including one or more hang syndrome registers that indicate whether the first array of configurable units is hung and whether the second array of configurable units is hung, the network recovery circuit further configured to:

set the one or more hang syndrome registers to indicate that the first array of configurable units is hung in conjunction with sending the first force-quit message; and

set the one or more hang syndrome registers to indicate that the second array of configurable units is hung in conjunction with sending the second force-quit message.

5 . The reconfigurable processor of claim 4 , wherein the first external interface circuit configured to:

receive a packet over the external communication link that includes information indicating a destination agent on the internal network for the packet;

in response to determining that the destination agent is associated with the first array of configurable units and that one or more hang syndrome registers does not indicate that the first array of configurable units is hung, send the packet over the internal network to the destination agent;

in response to determining that the destination agent is associated with the first array of configurable units and that one or more hang syndrome registers indicates that the first array of configurable units is hung, discard the packet;

in response to determining that the destination agent is associated with the second array of configurable units and that one or more hang syndrome registers does not indicate that the second array of configurable units is hung, send the packet over the internal network to the destination agent; and

in response to determining that the destination agent is associated with the second array of configurable units and that one or more hang syndrome registers indicates that the second array of configurable units is hung, discard the packet.

6 . The reconfigurable processor of claim 4 , wherein the network recovery circuit further configured to detect whether the internal network is hung and to generate a network failure condition in response, and wherein the first external interface circuit configured to set the one or more hang syndrome registers to indicate that both the first array of configurable units and the second array of configurable units are hung in response to the network failure condition.

7 . The reconfigurable processor of claim 3 , further comprising:

a first external interface circuit coupled between the internal network and an external communication link, the first external interface circuit including one or more hang syndrome registers including a first bit field to indicate whether the first array of configurable units has been sent a force-quit message, a second bit field to indicate whether the second array of configurable units has been sent a force-quit message, and a third bit field to indicate that a network failure condition has been detected; and

a network health monitor circuit to detect whether the internal network is hung and to generate the network failure condition in response;

the network recovery circuit further configured to set the first bit field in conjunction with sending the first force-quit message, set the second bit field in conjunction with sending the second force-quit message, and set the third bit field in response to the network failure condition;

the first external interface circuit configured to:

receive a packet over the first external interface circuit that includes information indicating a destination agent on the internal network for the packet;

in response to determining that the destination agent is associated with the first array of configurable units and neither the first bit field nor the third bit field are set, send the packet over the internal network to the destination agent;

in response determining that the destination agent is associated with the first array of configurable units and either the first bit field or the third bit field are set, discard the packet;

in response to determining that the destination agent is associated with the second array of configurable units and neither the second bit field nor the third bit field are set, send the packet over the internal network to the destination agent; and

in response determining that the destination agent is associated with the second array of configurable units and either the second bit field or the third bit field are set, discard the packet.

8 . The reconfigurable processor of claim 3 , wherein the first force-quit controller also empties buffers in the first array of configurable units related to the internal network and allows any outstanding transactions involving the first array of configurable units to complete in response to receiving the first force-quit message.

9 . A method of halting execution of an application within a reconfigurable processor having a first array of configurable units and a second array of configurable units, each coupled to an internal network of the reconfigurable processor and respectively comprising a plurality of processor configurable units and a plurality of memory configurable units coupled together with an array-level network, the reconfigurable processor including one or more CSRs (control/status registers) that include a first application ID field and a second application ID field, the method comprising:

receiving, at a network recovery circuit in the reconfigurable processor, a hang message from a first hang detection circuit associated with the first array of configurable units of the reconfigurable processor;

sending, by the network recovery circuit, a first force-quit message to a first force-quit controller associated with the first array of configurable units in response to receiving the hang message;

halting, by the first force-quit controller, execution of a first application running on the first array of configurable units in response to receiving the first force-quit message;

comparing a first value of the first application ID field to a second value of the second application ID field;

sending, by the network recovery circuit, a second force-quit message to a second force-quit controller associated with the second array of configurable units in response to the first value being equal to the second value, which signifies that both the first array of configurable units and the second array of configurable units are assigned to the first application; and

halting, by the second force-quit controller, execution of the first application running on the second array of configurable units in response to receiving the second force-quit message.

10 . The method of claim 9 , further comprising:

sending first configuration information for a first portion of the first application from a host computer to the first array of configurable units in the reconfigurable processor and set the first application ID field in the reconfigurable processor to a value indicative of the first application;

send second configuration information for a second portion of the first application from the host computer to the second array of configurable units in the reconfigurable processor and set the second application ID field in the reconfigurable processor to the value indicative of the first application; and

initiate, from the host computer, execution of the first application using at least the first array of configurable units and the second array of configurable units in the reconfigurable processor.

11 . The method of claim 10 , wherein the reconfigurable processor is a first reconfigurable processor, the method further comprising:

send third configuration information for a third portion of the first application from the host computer to a third array of configurable units in a second reconfigurable processor and set a first application ID field in the second reconfigurable processor to the value indicative of the first application;

send fourth configuration information for a portion of a second application from the host computer to a fourth array of configurable units in the second reconfigurable processor and set a second application ID field in the second reconfigurable processor to a value indicative of the second application;

initiate, from the host computer, execution of the first application using the third array of configurable units in the second reconfigurable processor and execution of the second application using the fourth array of configurable units in the second reconfigurable processor;

send an interrupt from the first reconfigurable processor to the host computer to indicate that the first array of configurable units in the first reconfigurable processor is hung;

in response to receiving the interrupt from the first reconfigurable processor, determining, by the host computer, one or more other arrays of configurable units that are assigned to the first application, including the third array of configurable units in the second reconfigurable processor;

send a command to the second reconfigurable processor to send a force-quit message to a third force-quit controller associated with the third array of configurable units in the second reconfigurable processor; and

halting, by the third force-quit controller in the second reconfigurable processor, execution of the first application running on the third array of configurable units the second reconfigurable processor.

12 . The method of claim 9 , further comprising receiving the hang message at the network recovery circuit from the first hang detection circuit over the internal network.

13 . The method of claim 9 , further comprising allowing a second application to continue execution on the second array of configurable units in response to the first value being different than the second value, which signifies that the first array of configurable units and the second array of configurable units are assigned to different applications, wherein the second application continues to execute on the second array of configurable units concurrently with the halting of the execution of the first application on the first array of configurable units.

14 . The method of claim 9 , wherein the reconfigurable processor including an external interface circuit coupled between the internal network and an external communication link, the method further comprising:

setting a first hang syndrome field in the one or more CSRs to indicate that the first array of configurable units is hung in conjunction with sending the first force-quit message;

setting a second hang syndrome field in the one or more CSRs to indicate that the second array of configurable units is hung in conjunction with sending the second force-quit message;

receiving a packet over the external communication link at the external interface circuit, the packet including information indicating a destination agent on the internal network for the packet;

sending the packet over the internal network to the destination agent in response to determining that either:

the destination agent is associated with the first array of configurable units and that the first hang syndrome field does not indicate that the first array of configurable units is hung, or

the destination agent is associated with the second array of configurable units and that the second hang syndrome field does not indicate that the second array of configurable units is hung; and

discarding the packet in response to determining that either:

the destination agent is associated with the first array of configurable units and that the first hang syndrome field indicates that the first array of configurable units is hung, or

the destination agent is associated with the second array of configurable units and that the second hang syndrome field indicates that the second array of configurable units is hung.

15 . The method of claim 14 , further comprising:

detecting that the internal network is hung; and

setting both a first hang syndrome field and the second hang syndrome field in the one or more CSRs.

16 . The method of claim 9 , wherein the reconfigurable processor including an external interface circuit coupled between the internal network and an external communication link, the method further comprising:

setting a first hang syndrome field in the one or more CSRs to indicate that the first array of configurable units is hung in conjunction with sending the first force-quit message;

setting a second hang syndrome field in the one or more CSRs to indicate that the second array of configurable units is hung in conjunction with sending the second force-quit message;

detecting that the internal network is hung and setting a third hang syndrome field in the one or more CSRs to indicated that the internal network is hung in response;

receiving a packet over the external communication link at the external interface circuit, the packet including information indicating a destination agent on the internal network for the packet;

sending the packet over the internal network to the destination agent in response to determining that either:

the destination agent is associated with the first array of configurable units and that neither the first hang syndrome field nor the third hang syndrome field is set, or

the destination agent is associated with the second array of configurable units and that neither the second hang syndrome field nor the third hang syndrome field is set; and

discarding the packet in response determining that either:

the destination agent is associated with the first array of configurable units and that either the first hang syndrome field or the third hang syndrome field is set, or

the destination agent is associated with the second array of configurable units and that either the second hang syndrome field does or the third hang syndrome field is set.

17 . The method of claim 9 , further comprising emptying buffers in the first array of configurable units related to the internal network to allow any outstanding transactions involving the first array of configurable units to complete in response to the first force-quit controller receiving the first force-quit message.

18 . The method of claim 9 , wherein the reconfigurable processor including an external interface circuit coupled between the internal network and an external communication link, the method further comprising:

sending first configuration information for a first portion of the first application from a host computer through the external communication link, the external interface circuit, and the internal network, to the first array of configurable units;

setting the first application ID field to a value indicative of the first application in response to a command sent from the host computer;

sending second configuration information for a second portion of the first application from the host computer through the external communication link, the external interface circuit, and the internal network, to the second array of configurable units;

setting the second application ID field to a value indicative of the first application in response to a command sent from the host computer; and

executing the first application using at least both the first array of configurable units and the second array of configurable units.

19 . The method of claim 9 , wherein the reconfigurable processor including an external interface circuit coupled between the internal network and an external communication link, the method further comprising:

sending first configuration information for at least a portion of the first application from a host computer through the external communication link, the external interface circuit, and the internal network, to the first array of configurable units;

setting the first application ID field to a value indicative of the first application in response to a command sent from the host computer;

sending second configuration information for at least a portion of a second application from the host computer through the external communication link, the external interface circuit, and the internal network, to the second array of configurable units;

setting the second application ID field to a value indicative of the second application in response to a command sent from the host computer;

executing the first application using at least the first array of configurable units; and

executing the second application using at least the second array of configurable units.

20 . The method of claim 9 , wherein the reconfigurable processor including an external interface circuit coupled between the internal network and an external communication link, the method further comprising:

sending first configuration information for a first portion of the first application from a host computer through the external communication link, the external interface circuit, and the internal network, to the first array of configurable units;

setting the first application ID field to a value indicative of the first application in response to a command sent from the host computer;

sending second configuration information for a second portion of the first application from the host computer to a third array of configurable units in another reconfigurable processor;

executing the first application using at least both the first array of configurable units and the third array of configurable units;

sending an interrupt from the reconfigurable processor to the host computer in response to receiving the hang message; and

halting, by the host computer, execution of the first application running on the third array of configurable units in response to receiving the interrupt.

Assignments (2)

Continuity (5)

References Cited (23)