Neural network operation reordering for parallel execution
Techniques are disclosed for reordering operations of a neural network to improve runtime efficiency. In some examples, a compiler receives a description of the neural network comprising a plurality of operations. The compiler may determine which execution engine of a plurality of execution engines is to perform each of the plurality of operations. The compiler may determine an order of performance associated with the plurality of operations. The compiler may identify a runtime inefficiency based on the order of performance and a hardware usage for each of the plurality of operations. An operation may be reordered to reduce the runtime inefficiency. Instructions may be compiled based on the plurality of operations, which include the reordered operation.
1. A method comprising:
receiving, by a compiler, a description of a neural network comprising a plurality of operations;
determining an order of performance for the plurality of operations;
determining a hardware usage for each of the plurality of operations, wherein determining the hardware usage for each of the plurality of operations includes:
determining which execution engine of a plurality of execution engines is to perform each of the plurality of operations;
identifying a runtime inefficiency based on the order of performance and the hardware usage for each of the plurality of operations, wherein identifying the runtime inefficiency is further based on which execution engine of the plurality of execution engines is to perform each of the plurality of operations;
reordering an operation of the plurality of operations to modify the order of performance and to reduce the runtime inefficiency; and
generating a set of compiled instructions based on the plurality of operations, wherein the plurality of operations includes the reordered operation.
2. The method of claim 1 , wherein identifying the runtime inefficiency includes identifying:
a first operation performed by a first execution engine;
a second operation performed by a second execution engine, wherein the second operation is performed after the first operation; and
a third operation performed by the first execution engine, wherein the third operation is performed after the second operation.
3. The method of claim 2 , wherein identifying the runtime inefficiency includes identifying:
a fourth operation performed by a third execution engine, wherein the fourth operation is performed before the first operation.
4. The method of claim 2 , wherein:
the second operation is performed on an output of the first operation; and
the third operation is performed on an output of the second operation.
5. The method of claim 2 , wherein reordering the operation includes:
reordering the first operation to be performed after the second operation.
6. The method of claim 5 , wherein the reordered first operation is performed on an output of the second operation.
7. A non-transitory computer-readable medium comprising instructions that, when executed by a processor, cause the processor to perform operations comprising:
receiving, by a compiler, a description of a neural network comprising a plurality of operations;
determining an order of performance for the plurality of operations;
determining a hardware usage for each of the plurality of operations, wherein determining the hardware usage for each of the plurality of operations includes:
determining which execution engine of a plurality of execution engines is to perform each of the plurality of operations;
identifying a runtime inefficiency based on the order of performance and the hardware usage for each of the plurality of operations, wherein identifying the runtime inefficiency is further based on which execution engine of the plurality of execution engines is to perform each of the plurality of operations;
reordering an operation of the plurality of operations to modify the order of performance and to reduce the runtime inefficiency; and
generating a set of compiled instructions based on the plurality of operations, wherein the plurality of operations includes the reordered operation.
8. The non-transitory computer-readable medium of claim 7 , wherein identifying the runtime inefficiency includes identifying:
a first operation performed by a first execution engine;
a second operation performed by a second execution engine, wherein the second operation is performed after the first operation; and
a third operation performed by the first execution engine, wherein the third operation is performed after the second operation.
9. The non-transitory computer-readable medium of claim 8 , wherein identifying the runtime inefficiency includes identifying:
a fourth operation performed by a third execution engine, wherein the fourth operation is performed before the first operation.
10. The non-transitory computer-readable medium of claim 8 , wherein reordering the operation includes:
reordering the first operation to be performed after the second operation.
11. The non-transitory computer-readable medium of claim 10 , wherein the reordered first operation is performed on an output of the second operation.
12. The non-transitory computer-readable medium of claim 7 , wherein determining the hardware usage for the plurality of operations includes:
determining a memory usage for each of the plurality of operations, wherein identifying the runtime inefficiency is further based on the memory usage for each of the plurality of operations.
13. A method comprising:
receiving, by a compiler, a description of a neural network comprising a plurality of operations;
determining an order of performance for the plurality of operations;
determining a hardware usage for each of the plurality of operations, wherein determining the hardware usage for each of the plurality of operations includes:
determining a memory usage for each of the plurality of operations;
identifying a runtime inefficiency based on the order of performance and the hardware usage for each of the plurality of operations, wherein identifying the runtime inefficiency is further based on the memory usage for each of the plurality of operations;
reordering an operation of the plurality of operations to modify the order of performance and to reduce the runtime inefficiency; and
generating a set of compiled instructions based on the plurality of operations, wherein the plurality of operations includes the reordered operation.