Artificial intelligence (AI)/machine learning (ML) tensor processor

1. A system for executing tensor operations, the system comprising:

a programmable tensor processor; and

a memory coupled to the programmable tensor processor, wherein the programmable tensor processor comprises:

one or more load address generation unit (AGU) circuits configured to generate a first sequence of addresses and read input tensor operands from the memory based on the first sequence of addresses;

a datapath circuit configured to perform the tensor operations on the input tensor operands based on receiving one or more instructions to determine output tensor operands, the one or more instructions being based on a loop iteration count and loop body micro-code instructions defining a loop body of a tensor program stored in the memory, the loop body micro-code instructions being executed in the programmable tensor processor; and

a store AGU circuit configured to generate a second sequence of addresses and write the output tensor operands to the memory based on the second sequence of addresses.

2. The system of claim 1 , wherein the one or more load AGU circuits are configured to generate the first sequence of addresses based on receiving one or more input address generation parameters based on the tensor program stored in the memory, and to generate the second sequence of addresses based on receiving one or more output address generation parameters based on the tensor program stored in the memory, the first sequence of addresses comprising one or more sequences of addresses.

3. The system of claim 1 , wherein the tensor operations comprise one or more selected from a group of arithmetic operations, look-up-table (LUT) based operations, multiply and multiply-accumulate operations, transpose and shuffle operations, and/or reduction operations.

4. The system of claim 1 , wherein the tensor program is decomposed into an input operand memory loads section, a loop sequencing section, a loop body microcode executing across functional circuits of the datapath circuit, and an output operand memory store section, wherein the loop sequencing section of the tensor program comprises the loop iteration count of the tensor program and a plurality of predicate instructions representing the loop body micro-code instructions, and wherein the loop iteration count is a set value.

5. The system of claim 4 , wherein the first sequence of addresses are generated based on the input operand memory loads section of the tensor program and the second sequence of addresses are generated based on the output operand memory store section of the tensor program.

6. The system of claim 1 , wherein the programmable tensor processor further comprises:

a nested loop sequence circuit configured to iterate loops in the tensor program based on the loop iteration count of the tensor program and to generate a stream of condition codes;

a predicate filter circuit configured to filter a subset of the loop body micro-code instructions, based on the stream of condition codes from the nested loop sequencer circuit, and to generate filtered instructions comprising corresponding predicate instructions; and

a pending instruction tracker circuit configured to track the filtered instructions, from the predicate filter circuit, in an instruction queue, and to issue the one or more instructions of the filtered instructions to the datapath circuit, the one or more instructions are at a head of the instruction queue.

7. The system of claim 6 , wherein the predicate filter circuit is configured to filter one or more instructions from the loop body micro-code instructions that fail a condition check based on the stream of condition codes received from the nested loop sequencer circuit.

8. The system of claim 6 , wherein the pending instruction tracker circuit is further configured to maintain a table of scheduled instructions comprising the instruction queue, to shift the table of scheduled instructions to remove the one or more instructions after the one or more instructions from the head of the instruction queue are issued to the datapath circuit, and to accept and adds a new iteration of the filtered instructions to the table of scheduled instructions, wherein the new iteration of the filtered instructions that are added to the table of scheduled instructions are free of hazards.

9. The system of claim 1 , wherein the datapath circuit comprises:

one or more load functional circuits configured to load the input tensor operands through the one or more load AGU circuits;

one or more arithmetic-logic functional circuits configured to perform the tensor operations;

one or more multiplier functional circuits;

at least one storage circuit configured to store the output tensor operands through the store AGU circuit and intermediate results of the tensor operations; and

at least one register files circuit to store intermediate results of the tensor operations.

10. A method for executing tensor operations in neural network, the method comprising:

generating, by a processor, a first sequence of addresses;

reading, by the processor, input tensor operands from a memory coupled to the processor based on the first sequence of addresses;

iterating loops in a tensor program stored in the memory and generating a stream of condition codes by the processor;

filtering, by the processor, a subset of micro-code instructions from loop body micro-code instructions of the tensor program, based on the stream of condition codes and generating filtered instructions;

performing, by the processor, the tensor operations on the input tensor operands based on one or more instructions of the filtered instructions to determine output tensor operands;

generating, by the processor, a second sequence of addresses; and

writing, by the processor, the output tensor operands to the memory based on the second sequence of addresses.

11. The method of claim 10 , further comprising:

tracking, by the processor, the filtered instructions in an instruction queue; and

issuing, by the processor, the one or more instructions to a datapath circuit of the processor, the one or more instructions are at a head of the instruction queue.

12. The method of claim 11 , further comprising:

maintaining, by the processor, a table of scheduled instructions comprising the instruction queue;

shifting, by the processor, the table to remove the one or more instructions after the one or more instructions from the head of the instruction queue are issued to the datapath circuit; and

accepting and adding, by the processor, a new iteration of the filtered instructions to the table of scheduled instructions, wherein the new iteration of the filtered instructions that are added to the table of scheduled instructions are free of hazards.

13. The method of claim 10 , wherein the processor is configured to iterate the loops in the tensor program based on a loop iteration count of the tensor program,

wherein the loop body micro-code instructions defines a loop body of the tensor program and are executed in the processor,

wherein the filtered instructions comprises a plurality of predicate instructions representing the subset of the micro-code instructions that are filtered, and

wherein two or more predicate instructions from among the plurality of predicate instructions are allocated to a same functional circuit at a same cycle based on predicates in the two or more predicate instructions being mutually exclusive.

14. The method of claim 10 , wherein the first sequence of addresses are generated based on receiving one or more input address generation parameters based on the tensor program, and wherein the second sequence of addresses are generated based on receiving one or more output address generation parameters based on the tensor program.

15. The method of claim 10 , wherein the tensor operations comprise one or more selected from a group of arithmetic operations, look-up-table (LUT) based operations, multiply and multiply-accumulate operations, transpose and shuffle operations, and/or reduction operations.

16. A non-transitory computer readable medium comprising stored instructions, which when executed by a processor, cause the processor to generate a digital representation of a tensor processor circuit comprising:

one or more load address generation unit (AGU) circuits configured to generate a first sequence of addresses and read input tensor operands from a memory coupled to the tensor processor circuit based on the first sequence of addresses;

a datapath circuit configured to perform tensor operations on the input tensor operands based on receiving one or more instructions to determine output tensor operands, the one or more instructions being based on a loop iteration count and loop body micro-code instructions defining a loop body of a tensor program stored in the memory, the loop body micro-code instructions being executed in the tensor processor circuit; and

a store AGU circuit configured to generate a second sequence of addresses and write the output tensor operands to the memory based on the second sequence of addresses.

17. The non-transitory computer readable medium of claim 16 , wherein the tensor processor circuit further comprises:

a nested loop sequence circuit configured to iterate loops in the tensor program based on the loop iteration count of the tensor program and to generate a stream of condition codes;

a predicate filter circuit configured to filter a subset of the loop body micro-code instructions, based on the stream of condition codes from the nested loop sequencer circuit, and to generate filtered instructions comprising corresponding predicate instructions; and

a pending instruction tracker circuit configured to track the filtered instructions, from the predicate filter circuit, in an instruction queue, and to issue the one or more instructions of the filtered instructions to the datapath circuit, the one or more instructions are at a head of the instruction queue.

18. The non-transitory computer readable medium of claim 16 , wherein the datapath circuit comprises:

one or more load functional circuits configured to load the input tensor operands through the one or more load AGU circuits;

one or more arithmetic-logic functional circuits configured to perform the tensor operations;

at least one storage circuit configured to store the output tensor operands through the store AGU circuit and intermediate results of the tensor operations; and

at least one register files circuit to store intermediate results of the tensor operations.

19. The non-transitory computer readable medium of claim 16 , wherein the tensor program is decomposed into an input operand memory loads section, a loop sequencing section, a loop body execution on across functional units section, and an output operand memory store section, wherein the loop sequencing section of the tensor program comprises the loop iteration count of the tensor program and a plurality of predicate instructions representing the loop body micro-code instructions, and wherein the loop iteration count is a set value.

20. The non-transitory computer readable medium of claim 16 , wherein the tensor operations comprise one or more selected from a group of arithmetic operations, look-up-table (LUT) based operations, multiply and multiply-accumulate operations, transpose and shuffle operations, and/or reduction operations.