Methods and apparatus for dynamic batching of data for neural network workloads

1. An apparatus comprising:

interface circuitry;

machine-readable instructions; and

at least one processor circuit to be programmed by the machine-readable instructions to;

determine a first layer ratio between a first number of computation operations for a first layer of a model and a first number of weights of the first layer;

determine a second layer ratio between a second number of computation operations for a second layer of the model and a second number of weights of the second layer, the second layer ratio different from the first layer ratio;

compare the first layer ratio to a number of operations performed by a compute engine per unit of data;

compare the second layer ratio to the number of operations performed by the compute engine per the unit of data;

based on the comparisons, select a first dynamic batch size of the first layer of the model to be different than a second dynamic batch size of the second layer of the model after a determination that the first dynamic batch size provides better cache usage in a hardware circuit than a reference batch size of the model;

parallel batch process first inputs of the first layer of the model based on the first dynamic batch size; and

parallel batch process second inputs of the second layer of the model based on the second dynamic batch size.

2. The apparatus of claim 1 , wherein one or more of the at least one processor circuit is to:

determine a number of operations that an accelerator can perform; and

determine the operations per second of the compute engine based on the number of operations that the accelerator can perform.

3. The apparatus of claim 1 , wherein the model is a neural network model, the one or more of the at least one processor circuit to intercept the neural network model before a compiler compiles the neural network model, the neural network model including the first layer and the second layer.

4. The apparatus of claim 3 , wherein the one or more of the at least one processor circuit is to extract parameter data from the first layer, the parameter data including the weights of the first layer and activation tensors of the first layer.

5. The apparatus of claim 1 , wherein the one or more of the at least one processor circuit is to determine the first number of computation operations of the first layer by combining a plurality of bit sizes corresponding to activation tensors.

6. The apparatus of claim 1 , wherein the one or more of the at least one processor circuit is to determine the second dynamic batch size of the second layer is equal to one after a determination that the second layer ratio is greater than the number of operations performed by the compute engine per the unit of data.

7. The apparatus of claim 1 , wherein the one or more of the at least one processor circuit is to generate a dynamic batch schedule, the dynamic batch schedule including batch sizes for ones of a plurality of layers in the model, the dynamic batch schedule using the first dynamic batch size of the first layer.

8. At least one non-transitory machine-readable medium comprising machine-readable instructions to cause at least one processor circuit to at least:

determine a first layer ratio between a first number of computation operations for a first layer of a model and a first number of weights of the first layer;

determine a second layer ratio between a second number of computation operations for a second layer of the model and a second number of weights of the second layer, the second layer ratio different from the first layer ratio;

compare the first layer ratio to a number of operations performed by a compute engine per unit of data;

compare the second layer ratio to the number of operations performed by the compute engine per the unit of data;

based on the comparisons, select a first dynamic batch size of the first layer of the model to be different than a second dynamic batch size of the second layer of the model after a determination that the first dynamic batch size provides better cache usage in a hardware circuit than a reference batch size of the model;

parallel batch process first inputs of the first layer of the model based on the first dynamic batch size; and

parallel batch process second inputs of the second layer of the model based on the second dynamic batch size.

9. The at least one non-transitory machine-readable medium of claim 8 , wherein the machine-readable instructions are to cause one or more of the at least one processor circuit to:

determine a number of operations that an accelerator can perform; and

determine the operations per second of the compute engine based on the number of operations that the accelerator can perform.

10. The at least one non-transitory machine-readable medium of claim 8 , wherein the machine-readable instructions are to cause one or more of the at least one processor circuit to intercept the model from a compiler, a neural network of the model including the first layer and the second layer.

11. The at least one non-transitory machine-readable medium of claim 10 , wherein the machine-readable instructions are to cause one or more of the at least one processor circuit to extract parameter data from the first layer of the model, the parameter data including the weights of the first layer and activation tensors of the first layer.

12. The at least one non-transitory machine-readable medium of claim 11 , wherein the machine-readable instructions are to cause one or more of the at least one processor circuit to determine the first number of computation operations of the first layer by combining a plurality of bit sizes corresponding to the activation tensors.

13. The at least one non-transitory machine-readable medium of claim 8 , wherein the machine-readable instructions are to cause one or more of the at least one processor circuit to determine the second dynamic batch size of the second layer is equal to one after a determination that the second layer ratio is greater than the number of operations performed by the compute engine per the unit of data.

14. The at least one non-transitory machine-readable medium of claim 8 , wherein the machine-readable instructions are to cause one or more of the at least one processor circuit to generate a dynamic batch schedule, the dynamic batch schedule including batch sizes for ones of a plurality of layers in a neural network of the model, the dynamic batch schedule using the first dynamic batch size of the first layer.

15. A method comprising:

selecting a first layer in a neural network, the first layer trained by hyperparameters to generate a prediction about input data;

determining a first layer ratio between a first number of computation operations of the first layer and a first number of weights of the first layer of the neural network;

determine a second layer ratio between a second number of computation operations for a second layer of the neural network and a second number of weights of the second layer, the second layer ratio different from the first layer ratio;

comparing the first layer ratio to a number of operations performed by a compute engine per unit of data;

comparing the second layer ratio to the number of operations performed by the compute engine per the unit of data; and

based on the comparisons, selecting, by at least one processor circuit programmed by at least one instruction, a first dynamic batch size of the first layer of the neural network to be different than a second dynamic batch size of the second layer of the neural network after a determination that the first dynamic batch size provides better cache usage in a hardware circuit than a maximum batch size of the neural network;

parallel batch processing first inputs of the first layer of the neural network based on the first dynamic batch size; and

parallel batch processing second inputs of the second layer of the neural network based on the second dynamic batch size.

16. The method of claim 15 , further including:

determining a number of operations that an accelerator can perform; and

determining the operations per second of the compute engine based on the number of operations that the accelerator can perform.

17. The method of claim 15 , further including intercepting the neural network from a compiler.

18. The method of claim 15 , further including extracting parameter data from the first layer of the neural network, the parameter data including the weights of the first layer and activation tensors of the first layer.

19. The method of claim 18 , further including determining the first number of computation operations of the first layer by combining a plurality of bit sizes corresponding to the activation tensors.

20. The method of claim 15 , further including generating a dynamic batch schedule, the dynamic batch schedule including batch sizes for ones of a plurality of layers in the neural network, the dynamic batch schedule including the first dynamic batch size of the first layer.

21. An apparatus comprising:

layer operations circuitry to determine a first layer ratio between a first number of computation operations of a first layer of a model and a first number of weights of the first layer, and a second layer ratio between a second number of computation operations for a second layer of the model and a second number of weights of the second layer, the second layer ratio different from the first layer ratio;

comparator circuitry to:

compare the first layer ratio to a number of operations performed by a compute engine per unit of data; and

compare the second layer ratio to the number of operations performed by the compute engine per the unit of data; and

batch size determination circuitry to:

based on the comparisons, select a first dynamic batch size of the first layer of the model to be different than a second dynamic batch size of the second layer of the model after a determination that the first dynamic batch size provides better cache usage in a hardware circuit than a reference batch size of the model;

generate first instructions to cause parallel batch processing of first inputs of the first layer of the model based on the first dynamic batch size; and

generate second instructions to cause parallel batch processing of second inputs of the second layer of the model based on the second dynamic batch size.

22. The apparatus of claim 21 , including engine characteristics circuitry to:

determine a number of operations that an accelerator can perform; and

determine the operations per second of the compute engine based on the number of operations that the accelerator can perform.

23. The apparatus of claim 21 , further including extraction circuitry to extract parameter data from the first layer of a neural network of the model, the parameter data including the weights of the first layer and activation tensors of the first layer.

24. The apparatus of claim 21 , wherein the layer operations circuitry is to determine the first number of computation operations of the first layer by combining a plurality of bit sizes corresponding to activation tensors.

25. The apparatus of claim 21 , wherein the batch size determination circuitry to is to determine the second dynamic batch size of the second layer is equal to one after a determination that the second layer ratio is greater than the number of operations performed by the compute engine per the unit of data.