Granted Patent B1

US 12,321,849 · App. 18/238,722 · Granted Jun 3, 2025

Inventors: Animesh Jain (Sunnyvale, CA); Tobias Joseph Kastulus Edler von Koch (Austin, TX); Yizhi Liu (Fremont, CA); Taemin Kim (Portland, OR); Jindrich Zejda (Saratoga, CA); Yida Wang (Palo Alto, CA); Vinod Sharma (Menlo Park, CA); Richard John Heaton (San Jose, CA); Randy Renfu Huang (Morgan Hill, CA)

Assignee: Amazon Technologies, Inc.

G06N3/063G06F9/30007G06F9/545

Abstract

Claims (73)

1. A compiler-implemented method of generating executable instructions for a computing system, comprising:

receiving a first set of instructions including a kernel of a first operator and a kernel of a second operator, the kernel of the first operator including instructions of the first operator and write instructions to a virtual data node, the kernel of the second operator including instructions of the second operator and read instructions to the virtual data node;

determining a mapping between the write instructions and read instructions based on common tensor addresses shared by the write instructions and the read instructions;

determining, based on a mapping between the write instructions and read instructions, instructions of data transfer operations between the first operator and the second operator, wherein the instructions of the data transfer operations are determined such that a duplication of a same write instruction from the write instructions is reduced; and

generating a second set of instructions representing a fused operator of the first operator and the second operator, the second set of instructions being executable by the computing system and including the instructions of the first operator, the instructions of the second operator, and the instructions of the data transfer operations.

2. The compiler-implemented method of claim 1 , further comprising:

wherein the virtual data node represents a logical tensor to store output data of the first operator and from which the second operator fetches input data.

3. The compiler-implemented method of claim 2 , further comprising:

wherein the logical tensor represented by the virtual data node includes entries each associated with a tensor address;

wherein the write instructions include the tensor addresses of the entries to store the output data;

wherein the read instructions include the tensor addresses of the entries from which the input data are to be fetched.

4. The compiler-implemented method of claim 1 , further comprising:

identifying the virtual data node based on identifying the write instructions to the virtual data node and the read instructions to the virtual data node; and

associating the write instructions and the read instructions with the virtual data node.

5. The compiler-implemented method of claim 4 , further comprising:

creating a plurality of access groups, each of the plurality of access groups including at least one of the read instructions and at least one corresponding write instruction mapped to the at least one of the read instructions; and

converting, for each of the plurality of access groups, the at least one of the read instructions to at least one memory read instruction at a memory and the least one corresponding write instruction to at least one memory write instruction at the memory.

6. The compiler-implemented method of claim 5 , further comprising:

determining, for one of the read instructions, that a first access group includes all of the corresponding write instructions mapped to the one of the read instructions; and

adding the one of the read instructions to the first access group.

7. The compiler-implemented method of claim 5 , further comprising:

determining, for one of the read instructions, that a first access group includes a first subset of corresponding write instructions mapped to the one of the read instructions but not the remaining subset of the corresponding write instructions;

determining that a total amount of data to be written by the write instructions of the first access group and by the remaining subset of the corresponding write instructions is below a threshold; and

responsive to determining that the total amount of data is below the threshold, adding the one of the read instructions and the remaining subset of the corresponding write instructions to the first access group.

8. The compiler-implemented method of claim 5 , further comprising:

for each access group, and based on determining that the first operator and the second operator are to be fused, converting the at least one of the read instructions to at least one memory read instruction at an on-chip memory internal to the computing system and the least one corresponding write instruction to at least one memory write instruction at the on-chip memory.

9. The compiler-implemented method of claim 5 , further comprising:

wherein the read instructions of the plurality of access groups are to be performed in parallel; and

wherein the write instructions of the plurality of access groups are to be performed in parallel.

10. The compiler-implemented method of claim 1 , further comprising:

determining that a total amount of data involved in the data transfer operations exceeds a threshold; and

responsive to determining that the total amount exceeds the threshold, generating the instructions of the data transfer operations as access instructions at an off-chip memory external to the computing system.

11. The compiler-implemented method of claim 1 , further comprising:

wherein the virtual data node is a first virtual data node;

wherein the kernel of the first operator further includes read instructions at a second virtual data node;

wherein the kernel of the second operator further includes write instructions at a third virtual data node; and

wherein the compiler-implemented method further comprises:

responsive to determining that read instructions at the second virtual data nodes d 0 not have corresponding write instructions, converting the read instructions at the second virtual data nodes to memory read instructions at an off-chip memory external to the computing system; and

responsive to determining that write instructions at the third virtual data nodes d 0 not have corresponding read instructions, converting the write instructions at the third virtual data nodes to memory write instructions at the off-chip memory.

12. The compiler-implemented method of claim 1 , further comprising:

wherein the computing system comprises a hardware accelerator.

13. The compiler-implemented method of claim 1 , further comprising:

wherein the first operator and the second operator are associated with, respectively, a first neural network layer and a second neural network layer of a neural network implemented by the computing system.

14. A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to execute a compiler by performing operations comprising:

receiving a first set of instructions including a kernel of a first operator and a kernel of a second operator, the kernel of the first operator including instructions of the first operator and write instructions to a virtual data node, the kernel of the second operator including instructions of the second operator and read instructions to the virtual data node;

determining a mapping between the write instructions and read instructions based on common tensor addresses shared by the write instructions and the read instructions;

determining, based on a mapping between the write instructions and read instructions, instructions of data transfer operations between the first operator and the second operator, wherein the instructions of the data transfer operations are determined such that a duplication of a same write instruction from the write instructions is reduced; and

generating a second set of instructions representing a fused operator of the first operator and the second operator, the second set of instructions being executable by a computing system and including the instructions of the first operator, the instructions of the second operator, and the instructions of the data transfer operations.

15. The non-transitory computer-readable medium of claim 14 , further comprising:

wherein the virtual data node represents a logical tensor to store output data of the first operator and from which the second operator fetches input data.

16. The non-transitory computer-readable medium of claim 15 , further comprising:

wherein the logical tensor represented by the virtual data node includes entries each associated with a tensor address;

wherein the write instructions include the tensor addresses of the entries to store the output data;

wherein the read instructions include the tensor addresses of the entries from which the input data are to be fetched.

17. The non-transitory computer-readable medium of claim 14 , further comprising:

wherein the operations further comprise:

identifying the virtual data node based on identifying the write instructions to the virtual data node and the read instructions to the virtual data node; and

associating the write instructions and the read instructions with the virtual data node.

18. The non-transitory computer-readable medium of claim 17 , further comprising:

wherein the operations further comprise:

creating a plurality of access groups, each of the plurality of access groups including at least one of the read instructions and at least one corresponding write instruction mapped to the at least one of the read instructions; and

converting, for each of the plurality of access groups, the at least one of the read instructions to at least one memory read instruction at a memory and the least one corresponding write instruction to at least one memory write instruction at the memory.

19. The non-transitory computer-readable medium of claim 18 , further comprising:

wherein the operations further comprise:

determining, for one of the read instructions, that a first access group includes all of the corresponding write instructions mapped to the one of the read instructions; and

adding the one of the read instructions to the first access group.

20. An apparatus comprising:

one or more processors; and

a non-transitory computer-readable medium storing a compiler, wherein the one or more processors are configured to execute the compiler by performing operations comprising:

receiving a first set of instructions including a kernel of a first operator and a kernel of a second operator, the kernel of the first operator including instructions of the first operator and write instructions to a virtual data node, the kernel of the second operator including instructions of the second operator and read instructions to the virtual data node;

determining a mapping between the write instructions and read instructions based on common tensor addresses shared by the write instructions and the read instructions;

determining, based on a mapping between the write instructions and read instructions, instructions of data transfer operations between the first operator and the second operator, wherein the instructions of the data transfer operations are determined such that a duplication of a same write instruction from the write instructions is reduced; and

generating a second set of instructions representing a fused operator of the first operator and the second operator, the second set of instructions being executable by a computing system and including the instructions of the first operator, the instructions of the second operator, and the instructions of the data transfer operations.

Assignments (1)

Continuity (1)

References Cited (14)