Methods for an AI accelerator integrated circuit chip with integrated cell-based fabric adapter
An integrated circuit formed on (i) a single semiconductor die or (ii) a plurality semiconductor dies that are integrated into a single package. The integrated circuit may include a communication interface including a serializer/deserializer (SerDes) interface; a fabric adapter communicatively coupled to the communication interface; a plurality of inference engine clusters, each inference engine cluster including a respective memory element and/or memory interface; and a data interconnect communicatively coupling each respective memory element and/or memory interfaces of the plurality of inference engine clusters to the fabric adapter. The fabric adapter may be configured to facilitate remote direct memory access (RDMA) read and write services and/or datagram communication over a cell-based switch fabric to and from the respective memory elements and/or memory interfaces of the plurality of inference engine clusters via the data interconnect.
1 . A method for transmitting data from a first artificial intelligence (AI) chip to a second AI chip over a cell-based switch fabric, the first AI chip being a leaf node of a Clos network and serving as a source endpoint, the first AI chip comprising a virtual output queue (VOQ) subsystem with a plurality of VOQs,
wherein the first AI chip comprises:
a communication interface;
a fabric adapter communicatively coupled to the communication interface, wherein the fabric adapter includes the VOQ subsystem;
an inference engine cluster including a memory element or a memory interface; and
a data interconnect communicatively coupling the memory element or the memory interface of the inference engine cluster to the fabric adapter,
the method comprising:
dividing, by the first AI chip, the data into packets; and
for each of the packets:
determining, by the first AI chip, a respective one of the VOQs to assign to the packet;
attaching, by the first AI chip, a packet header to the packet;
enqueuing, by the first AI chip, the packet to the respective one of the VOQs assigned to the packet; and
when selected for transport to the second AI chip, dequeuing, by the first AI chip, the packet from the respective one of the VOQs, cellifying, by the first AI chip, the packet, attaching, by the first AI chip, a cell header to each cell to indicate a destination OQ on the second AI chip corresponding to the assigned VOQ, and transmitting, by the first AI chip as the source endpoint, the cells to one or more cell-fabric switch chips.
2 . The method of claim 1 , wherein the packet header is used by the second AI chip to identify the data in the packet.
3 . The method of claim 1 , wherein cellifying the packet comprises dividing data contained in the packet into regular sized blocks of data.
4 . The method of claim 1 , further comprising maintaining a count of a total number of packets stored at each of the VOQs.
5 . The method of claim 1 , wherein enqueuing the packet to the respective one of the VOQs assigned to the packet comprises issuing, by a packet generator, an enqueue command to the VOQ subsystem, the enqueue command including an identifier of the packet, a length of the packet and a target VOQ number.
6 . The method of claim 1 , wherein enqueuing the packet to the respective one of the VOQs comprises communicating the packet from the memory element of the first AI chip to the respective one of the VOQs via a crossbar switch.
7 . A method for receiving data at a first artificial intelligence (AI) chip comprising an output queue (OQ) subsystem with a plurality of output queues (OQs), the first AI chip being a leaf node of a Clos network and serving as a destination endpoint, the OQs being associated with a VOQ subsystem on a second AI chip, wherein the first AI chip comprises:
a communication interface;
a fabric adapter communicatively coupled to the communication interface, wherein the fabric adapter includes the OQ subsystem;
an inference engine cluster including a memory element or a memory interface; and
a data interconnect communicatively coupling the memory element or the memory interface of the inference engine cluster to the fabric adapter,
the method comprising:
receiving, by the first AI chip, a plurality of packets from the second AI chip via one or more cell-fabric switch chips;
enqueueing, by the first AI chip, the plurality of packets to the OQs;
dequeuing, by the first AI chip, one or more of the packets from the OQs according to a scheduling policy based on a class of service assigned to each of the OQs; and
for each of the packets,
decellifying, by the first AI chip as the destination endpoint, respective cells of the packet to reconstitute packet data;
examining, by the first AI chip, a header of the packet to identify the packet data and determine its appropriate processing;
removing, by the first AI chip, the header of the packet; and
storing, by the first AI chip, the packet data to the memory element or communicating the packet data to the memory interface.
8 . The method of claim 7 , wherein the cells of a first one of the packets are decellified either before enqueueing the first packet to one of the OQs, or after the first packet has been dequeued from one of the OQs.
9 . The method of claim 7 , further comprising, prior to decellifying respective cells of the packet, reordering the respective cells of the packet.
10 . The method of claim 7 , wherein the scheduling policy is a round-robin policy, a weighted round-robin policy, a strict priority policy, or a combination of one or more of the aforementioned policies.
11 . The method of claim 7 , wherein the header of the packet includes one or more of receive and transmit remote direct memory access (RDMA) job identifiers which identify a specific RDMA transfer, a packet sequence number within a job, a last-packet-in-job flag, a packet type in order to indicate whether the packet contains RDMA data, datagram data, or a hardware message.
12 . The method of claim 7 , further comprising, prior to storing the packet data to the memory element or communicating the packet data to the memory interface, transmitting the packet data to the data interconnect configured to route the packet data to the memory element or memory interface, wherein the data interconnect comprises one or more of a crossbar switch, a ring, a torus interconnect or a mesh.