Architecture of crossbar of inference engine
A programmable hardware system for machine learning (ML) includes a core and an inference engine. The core receives commands from a host. The commands are in a first instruction set architecture (ISA) format. The core divides the commands into a first set for performance-critical operations, in the first ISA format, and a second set of performance non-critical operations, in the first ISA format. The core executes the second set to perform the performance non-critical operations of the ML operations and streams the first set to inference engine. The inference engine generates a stream of the first set of commands in a second ISA format based on the first set of commands in the first ISA format. The first set of commands in the second ISA format programs components within the inference engine to execute the ML operations to infer data.
1. A crossbar-based inference engine configured to perform a machine learning (ML) operation on an input data stream, comprising:
a plurality of on-chip memories (OCMs) coupled to a crossbar and each OCM is configured to load and maintain data from the input data stream for local access by components in the inference engine;
maintain and output results of the ML operation performed by the components in the inference engine as an output data stream;
a first plurality of processing units, wherein each processing unit of the first plurality of processing units is coupled to one OCM of the plurality of OCMs without going through the crossbar and configured to perform a dense and/or regular computation task of the ML operation on the data within the corresponding OCM;
a second plurality of processing units coupled to the first plurality of processing units and the plurality of OCMs through the crossbar, wherein each processing unit of the second plurality of processing units is configured to perform a sparse and/or irregular computation task of the ML operation on at least one of the data within the OCMs or from the first plurality of processing units; and
said crossbar configured to connect the second plurality of processing units to the plurality of OCMs to enable each processing unit of the second plurality of processing units to read data from and/or write data to the corresponding OCM.
2. The system of claim 1 , further comprising:
a plurality of OCM streamers each configured to stream data between each OCM and its corresponding processing unit from the first plurality of processing units.
3. The system of claim 1 , wherein:
the input data stream includes data to be analyzed and inferred by the inference engine and/or training data used to train the inference engine for the ML operation, wherein the training data includes a polynomial with their respective weights.
4. The system of claim 1 , wherein:
each processing unit of the second plurality of processing units is configured to receive and execute a set of programming instructions directly from a core, wherein the core is configured to coordinate and program the inference engine to perform the ML operation.
5. The system of claim 4 , wherein:
the plurality of OCMs and/or the first plurality of processing units are configured to receive and execute a set of programming instructions from the core via the crossbar.
6. The system of claim 1 , wherein:
each processing unit of the first plurality of processing units is configured to perform a matrix multiplication operation on the data in its corresponding OCM.
7. The system of claim 6 , wherein:
each processing unit of the first plurality of processing units is configured to perform one or more post matrix multiplication operation on the output of processing unit of the first plurality of processing units.
8. The system of claim 1 , wherein:
the crossbar is configured to accept one read request or one write request per processing unit of the second plurality of processing units to read data from and write data to one OCM of the plurality of OCMs, respectively.
9. The system of claim 8 , wherein:
the crossbar is configured to route the read or the write request through the plurality of OCMs in the inference engine until the request reaches the OCM associated with the request.
10. The system of claim 8 , wherein:
the crossbar is configured to merge a plurality of read and/or write requests to a same address in the same OCM.
11. A method to perform a machine learning (ML) operation on an input data stream via an inference engine, comprising:
loading and maintaining data from the input data stream for local access by components in the inference engine in each on-chip memory (OCM) of a plurality of OCMs, wherein the plurality of OCMs is coupled to a crossbar in the inference engine;
performing a dense and/or regular computation task of the ML operation on the data in a OCM of the plurality of OCMs via one processing unit of a first plurality of processing units that are coupled to the OCM without going through the crossbar;
performing a sparse and/or irregular computation task of the ML operation on the data in the plurality of OCMs and/or from the first plurality of processing units via one processing unit of a second plurality of second processing units/elements that is coupled to the plurality of OCMs and the first plurality of processing units through the crossbar;
connecting the second plurality of processing units to the plurality of OCMs via the crossbar to enable each processing unit of the second plurality of processing units to read data from and/or write data to the plurality of OCMs; and
maintaining and outputting results of the ML operation performed by a processing tile that comprises at least a processing unit from a first plurality of processing units, a processing unit from a second plurality of processing units, and an OCM, wherein the OCM is configured to output a data stream from the processing tile.
12. The method of claim 11 , further comprising:
streaming data between each OCM and its corresponding processing unit of the first plurality of processing units via an OCM streamers.
13. The method of claim 11 , further comprising:
including in the input data stream data to be analyzed and inferred by the inference engine and/or training data used to train the inference engine for the ML operation, wherein the training data includes a polynomial with their respective weights.
14. The method of claim 11 , further comprising:
receiving and executing by each processing unit of the second plurality of processing units a set of programming instructions directly from a core, wherein the core is configured to coordinate and program the inference engine to perform the ML operation.
15. The method of claim 14 , further comprising:
receiving and executing by the plurality of OCMs and/or the first plurality of processing units a set of programming instructions from the core via the crossbar.
16. The method of claim 11 , further comprising:
performing a matrix multiplication operation by each processing unit of the first plurality of processing units on the data in its corresponding OCM.
17. The method of claim 16 , further comprising:
perform one or more post matrix multiplication operations by each processing unit of the first plurality of processing units on the output from the matrix multiplication operation.
18. The method of claim 11 , further comprising:
accepting one read request or one write request per processing unit of the second plurality of processing units by the crossbar to read data from and write data to one OCM of the plurality of OCMs, respectively.
19. The method of claim 18 , further comprising:
routing the read or the write request through the plurality of OCMs in the inference engine until the request reaches the OCM associated with the request.
20. The method of claim 18 , further comprising:
merging a plurality of read and/or write requests to a same address in a same OCM.
21. A crossbar-based inference engine configured to perform a machine learning (ML) operation on an input data stream, comprising:
a plurality of on-chip memories (OCMs) coupled to a crossbar and each OCM is configured to load and maintain data from the input data stream for local access by components in the inference engine;
maintain and output results of the ML operation performed by the components in the inference engine as an output data stream;
a first plurality of processing units, wherein each processing unit of the first plurality of processing units is directly coupled to one OCM of the plurality of OCMs and configured to perform a dense and/or regular computation task of the ML operation on the data within the corresponding OCM;
a second plurality of processing units coupled to the first plurality of processing units and the plurality of OCMs through the crossbar, wherein each processing unit of the second plurality of processing units is configured to perform a sparse and/or irregular computation task of the ML operation on at least one of the data within the OCMs or from the first plurality of processing units; and
said crossbar configured to connect the second plurality of processing units to the plurality of OCMs to enable each processing unit of the second plurality of processing units to read data from and/or write data to the corresponding OCM.