Algorithmic architecture co-design and exploration

1. A computer-implemented method for matching neural network layouts to hardware architectures, the method comprising:

using a number of processors to perform the steps of:

iteratively:

holding a number of neural network parameters constant while changing a number of hardware architecture parameters;

calculating a first loss value for a combination of the neural network parameters and hardware architecture parameters according to a gradient-based differentiable function within specified resource constraints;

holding the hardware architecture parameters constant while changing the neural network parameters;

calculating a second loss value for a new combination of neural network parameters and hardware architecture parameters according to the gradient-based differentiable function within the specified resource constraints;

combining the first loss value and the second loss value to calculate a combined loss value;

stopping the above iterative steps when the combined loss value reaches a specified threshold;

determining an optimal combination of neural network parameters and hardware architecture parameters according to the combined loss value; and

generating a neural network algorithm tailored to a target hardware architecture according to the optimal combination of neural network parameters and hardware architecture parameters.

2. The method of claim 1 , wherein the gradient-based differential function comprises a resource aware progressive differentiable search function.

3. The method of claim 1 , wherein the specified resource constraints comprise at least one of:

size;

weight;

power;

accuracy;

energy of computation;

minimum throughout;

maximum latency; or

manufacturing budget.

4. The method of claim 1 , wherein calculating the first loss value or second loss value for a combination of neural network parameters and hardware architecture parameters comprises performing a number of primitive operations in parallel, wherein output of the primitive operations are additively conformable.

5. The method of claim 4 , further comprising combining the primitive operations to form a mixed operation, wherein the first loss value is calculated from an output prediction of the mixed operation compared to an expected answer.

6. The method of claim 5 , further comprising:

calculating resource costs of each primitive operation according to respective cost functions; and

combining the resource costs of each primitive operation, wherein the second loss value comprises an expected resource cost of the mixed operation.

7. The method of claim 6 , wherein the expected cost of the mixed operation is differentiable with respect to the hardware architecture parameters.

8. A system for matching neural network layouts to hardware architectures, the system comprising:

a storage device configured to store program instructions; and

one or more processors operably connected to the storage device and configured to execute the program instructions to cause the system to:

iteratively:

hold a number of neural network parameters constant while changing a number of hardware architecture parameters;

calculate a first loss value for a combination of the neural network parameters and hardware architecture parameters according to a gradient-based differentiable function within specified resource constraints;

hold the hardware architecture parameters constant while changing the neural network parameters;

calculate a second loss value for a new combination of neural network parameters and hardware architecture parameters according to the gradient-based differentiable function within the specified resource constraints;

combine the first loss value and the second loss value to calculate a combined loss value;

stop the above iterative steps when the combined loss value reaches a specified threshold;

determine an optimal combination of neural network parameters and hardware architecture parameters according to the combined loss value; and

generate a neural network algorithm tailored to a target hardware architecture according to the optimal combination of neural network parameters and hardware architecture parameters.

9. The system of claim 8 , wherein the gradient-based differential function comprises a resource aware progressive differentiable search function.

10. The system of claim 8 , wherein the specified resource constraints comprise at least one of:

size;

weight;

power;

accuracy;

energy of computation;

minimum throughout;

maximum latency; or

manufacturing budget.

11. The system of claim 8 , wherein calculating the first loss value or second loss value for a combination of neural network parameters and hardware architecture parameters comprises performing a number of primitive operations in parallel, wherein output of the primitive operations are additively conformable.

12. The system of claim 11 , wherein the processors further execute instructions to combine the primitive operations to form a mixed operation, wherein the first loss value is calculated from an output prediction of the mixed operation compared to an expected answer.

13. The system of claim 12 , wherein the processors further execute instructions to:

calculate resource costs of each primitive operation according to respective cost functions; and

combine the resource costs of each primitive operation, wherein the second loss value comprises an expected resource cost of the mixed operation.

14. A computer program product for matching neural network layouts to hardware architectures, the computer program product comprising:

a computer-readable storage medium having program instructions embodied thereon to perform the steps of:

iteratively:

holding a number of neural network parameters constant while changing a number of hardware architecture parameters;

calculating a first loss value for a combination of the neural network parameters and hardware architecture parameters according to a gradient-based differentiable function within specified resource constraints;

holding the hardware architecture parameters constant while changing the neural network parameters;

calculating a second loss value for a new combination of neural network parameters and hardware architecture parameters according to the gradient-based differentiable function within the specified resource constraints;

combining the first loss value and the second loss value to calculate a combined loss value;

stopping the above iterative steps when the combined loss value reaches a specified threshold;

determining an optimal combination of neural network parameters and hardware architecture parameters according to the combined loss value; and

generating a neural network algorithm tailored to a target hardware architecture according to the optimal combination of neural network parameters and hardware architecture parameters.

15. The computer program product of claim 14 , wherein the gradient-based differential function comprises a resource aware progressive differentiable search function.

16. The computer program product of claim 14 , wherein the specified resource constraints comprise at least one of:

size;

weight;

power;

accuracy;

energy of computation;

minimum throughout;

maximum latency; or

manufacturing budget.

17. The computer program product of claim 14 , wherein calculating the first loss value or second loss value for a combination of neural network parameters and hardware architecture parameters comprises performing a number of primitive operations in parallel, wherein output of the primitive operations are additively conformable.

18. The computer program product of claim 17 , further comprising combining the primitive operations to form a mixed operation, wherein the first loss value is calculated from an output prediction of the mixed operation compared to an expected answer.

19. The computer program product of claim 18 , further comprising:

calculating resource costs of each primitive operation according to respective cost functions; and

combining the resource costs of each primitive operation, wherein the second loss value comprises an expected resource cost of the mixed operation.

20. The computer program product of claim 19 , wherein the expected cost of the mixed operation is differentiable with respect to the hardware architecture parameters.