Architecture for irregular operations in machine learning inference engine
A processing unit of an inference engine for machine learning (ML) includes a first data load steamer, a second data load streamer, an operator component, and a store streamer. The first data load streamer streams a first data stream from an on-chip memory (OCM) to the operator component. The second data load streamer streams a second data stream from the OCM to the operator component. The operator component performs a matrix operation on the first data stream and the second data stream. The store streamer receives a data output stream from the operator component and to store the data output stream in a buffer.
1. A processing unit of an inference engine for machine learning (ML), comprising:
a first data load streamer configured to stream a first data stream comprising a first plurality of data sections from an on-chip memory (OCM), using a single instruction, to an operator component by using an address of the OCM and a pattern of data to be loaded for the first data stream to be read and streamed;
a second data load streamer configured to stream a second data stream comprising a second plurality of data sections from the OCM, using a single instruction, to the operator component by using an address of the OCM and a pattern of data to be loaded for the second data stream to be read and streamed;
the operator component configured to perform a data operation on the first data stream and the second data stream; and
a store streamer configured to receive a data output stream from the operator component and to store the data output stream in a buffer,
wherein the pattern of data to be loaded for the first data stream includes a stride to a next block and a stride between lines,
wherein the first data stream pattern is specified by one or more of a starting address, number of lines to read for each operation, number of bytes per line, and a number of blocks to read.
2. The processing unit of claim 1 , wherein the data operation is a matrix multiplication operation and is selected from a group consisting of determining a maximum value, calculating an average value for a stream of data, calculating an addition of the first data stream to the second data stream, calculating a multiplication of the first data stream to the second data stream, rewriting the first data stream in a different pattern for matrix transformation, Tanh operation, Sigmoid operation, spatial batch normalization operation, and local response normalization.
3. The processing unit of claim 1 further comprising an instruction controller configured to store instructions received from a core engine.
4. The processing unit of claim 1 , wherein the buffer is configured to stream the data output stream to the OCM for storage thereof.
5. The processing unit of claim 1 , wherein the data output stream is specified by one or more of a starting address, a number of lines to write, line stride between lines, a number of bytes per line, and stride to a next block.
6. The processing unit of claim 1 , wherein the first data load streamer, the second data load streamer, the operator component, and the store streamer are configured to iteratively execute and process data until a termination condition is met.
7. A processing unit of an inference engine for machine learning (ML), comprising:
a first data load streamer configured to stream a first data stream comprising a first plurality of data sections from an on-chip memory (OCM), using a single instruction, to an operator component by using an address of the OCM and a pattern of data to be loaded for the first data stream to be read and streamed;
a second data load streamer configured to stream a second data stream comprising a second plurality of data sections from the OCM, using a single instruction, to the operator component by using an address of the OCM and a pattern of data to be loaded for the second data stream to be read and streamed;
the operator component configured to perform a matrix operation on the first data stream and the second data stream, wherein the matrix operation is performed by another processing unit that reads data within each matrix only once and wherein the another processing unit is configured to receive data within the each matrix as a data stream using a single instruction and further configured to operate on the each matrix as the data stream using a single instruction to generate an output matrix; and
a store streamer configured to receive a data output stream from the operator component and to store the data output stream in a buffer,
wherein the pattern of data to be loaded for the first data stream includes a stride to a next block and a stride between lines,
wherein the data output stream is specified by a starting address, a number of lines to write, line stride between lines, a number of bytes per line, and stride to a next block.
8. The processing unit of claim 7 , wherein the matrix operation is a matrix multiplication operation and is selected from a group consisting of determining a maximum value, calculating an average value for a stream of data, calculating an addition of the first data stream to the second data stream, calculating a multiplication of the first data stream to the second data stream, rewriting the first data stream in a different pattern for matrix transformation, Tanh operation, Sigmoid operation, spatial batch normalization operation, and local response normalization.
9. The processing unit of claim 7 further comprising an instruction controller configured to store instructions received from a core engine.
10. The processing unit of claim 7 , wherein the first data stream pattern is specified by a starting address, number of lines to read for each operation, number of bytes per line, and a number of blocks to read.
11. The processing unit of claim 7 , wherein the buffer is configured to stream the data output stream to the OCM for storage thereof.
12. The processing unit of claim 7 , wherein the first data load streamer, the second data load streamer, the operator component, and the store streamer are configured to iteratively execute and process data until a termination condition is met.
13. A method comprising:
streaming a first data stream comprising a first plurality of data sections from an on-chip memory (OCM), using a single instruction, to an operator component by using an address of the OCM and a pattern of data to be loaded for the first data stream to be read and streamed;
streaming a second data stream comprising a second plurality of data sections from the OCM to the operator component by using an address of the OCM and a pattern of data to be loaded for the second data stream to be read and streamed;
performing a data operation on the first data stream and the second data stream;
streaming a data output stream resulting from the performing; and
storing the data output stream,
wherein the pattern of data to be loaded for the first data stream includes a stride to a next block and a stride between lines,
wherein the first data stream pattern is specified by a starting address, number of lines to read for each operation, number of bytes per line, and a number of blocks to read.
14. The method of claim 13 , wherein the data operation is a matrix multiplication and is selected from a group consisting of determining a maximum value, calculating an average value for a stream of data, calculating an addition of the first data stream to the second data stream, calculating a multiplication of the first data stream to the second data stream, rewriting the first data stream in a different pattern for matrix transformation, Tanh operation, Sigmoid operation, spatial batch normalization operation, and local response normalization.
15. The method of claim 13 further comprising storing instructions received from a core engine.
16. The method of claim 13 , wherein the data output stream is specified by a starting address, a number of lines to write, line stride between lines, a number of bytes per line, and stride to a next block.
17. The method of claim 13 further comprising iteratively repeating the streaming the first data stream, the streaming the second data stream, the performing the post data operation, the streaming the data output stream, and the storing the data output stream until a termination condition is met.
18. The processing unit of claim 1 further comprising an instruction streamer configured to receive an instruction stream from a core, wherein the core is configured to track a head pointer and wherein the instruction streamer is configured to track a tail pointer to stream instructions from the core to the instruction streamer, and wherein the operator component is configured based on the streamed instructions.
19. The processing unit of claim 1 , wherein the first data stream is specified by a starting address and a number of blocks to read.
20. The processing unit of claim 1 , wherein the data output stream is specified by a starting address and stride to a next block.
21. A method comprising:
streaming a first data stream comprising a first plurality of data sections from an on-chip memory (OCM), using a single instruction, to an operator component by using an address of the OCM and a pattern of data to be loaded for the first data stream to be read and streamed;
streaming a second data stream comprising a second plurality of data sections from the OCM to the operator component by using an address of the OCM and a pattern of data to be loaded for the second data stream to be read and streamed;
performing a data operation on the first data stream and the second data stream;
streaming a data output stream resulting from the performing; and
storing the data output stream,
wherein the pattern of data to be loaded for the first data stream includes a stride to a next block and a stride between lines,
wherein the data output stream is specified by a starting address, a number of lines to write, line stride between lines, a number of bytes per line, and stride to a next block.