QUANTIZED ARCHITECTURE SEARCH FOR MACHINE LEARNING MODELS

1 . A method of determining an architecture of a machine learning model that optimizes the machine learning model, the method comprising:

using a processor to perform:

obtaining the machine learning model configured with a first architecture of a plurality of architectures, the machine learning model comprising a first set of parameters;

determining a second architecture of the plurality of architectures using a quantization of the first set of parameters; and

updating the machine learning model to obtain the machine learning model configured with the second architecture.

2 . The method of claim 1 , further comprising obtaining the quantization of the first set of parameters.

3 . The method of claim 2 , wherein:

each of the first set of parameters is encoded with a first representation; and

obtaining the quantization of the first set of parameters comprises, for each of the first set of parameters, transforming the parameter to a second number representation.

4 . The method of claim 1 , wherein determining the second architecture using the quantization of the first set of parameters comprises:

determining an indication of an architecture gradient using the quantization of first set of parameters; and

determining the second architecture using the indication of the architecture gradient.

5 . The method of claim 4 , wherein determining the indication of the architecture gradient for the first architecture comprises determining a partial derivative of a loss function using the quantization of the first set of parameters.

6 . The method of claim 1 , further comprising updating the first set of parameters of the machine learning model to obtain a second set of parameters.

7 . The method of claim 6 , wherein updating the first set of parameters comprises using gradient descent to obtain the second set of parameters.

8 . The method of claim 1 , further comprising encoding an architecture of the machine learning model as a plurality of weights for respective architecture parameters, the architecture parameters representing the plurality of architectures.

9 . The method of claim 8 , wherein:

determining the second architecture comprises determining an update to at least some weights of the plurality of weights; and

updating the machine learning model comprises applying the update to the at least some weights.

10 . The method of claim 1 , wherein determining the second architecture using the quantization of the first set of parameters comprises:

combining each of the first set of parameters with a respective quantization of the parameter to obtain a set of blended parameter values; and

determining the second architecture using the set of blended parameter values.

11 . The method of claim 10 , wherein combining the parameter with the quantization of the parameter comprises determining a linear combination of the parameter and the quantization of the parameter.

12 . The method of claim 1 , wherein the machine learning model comprises a neural network.

13 . The method of claim 12 , wherein the neural network comprises a convolutional neural network (CNN).

14 . The method of claim 12 , wherein the neural network comprises a recurrent neural network (RNN).

15 . The method of claim 12 , wherein the neural network comprises a transformer neural network.

16 . The method of claim 1 , further comprising training the machine learning model configured with the second architecture to obtain a trained machine learning model configured with the second architecture.

17 . The method of claim 16 , further comprising quantizing parameters of the trained machine learning model configured with the second architecture to obtain a machine learning model with quantized parameters.

18 . The method of claim 17 , wherein the processor has a first word size and the method further comprises transmitting the machine learning model with quantized parameters to a device comprising a processor with a second word size, wherein the second word size is smaller than the first word size.

19 . A system for determining an architecture of a machine learning model that optimizes the machine learning model, the system comprising:

a processor;

a non-transitory computer-readable storage medium storing instructions that, when executed by the processor, cause the processor to perform a method comprising:

obtaining the machine learning model configured with a first one of a plurality of architectures, the machine learning model comprising a first set of parameters;

determining a second one of the plurality of architectures using a quantization of the first set of parameters; and

updating the machine learning model to obtain the machine learning model configured with the second architecture.

20 . A non-transitory computer-readable storage medium storing instructions, wherein the instructions, when executed by a processor, cause the processor to perform a method comprising:

obtaining a machine learning model configured with a first one of a plurality of architectures, the machine learning model comprising a first set of parameters;

determining a second architecture the plurality of architectures using a quantization of the first set of parameters; and

updating the machine learning model to obtain the machine learning model configured with the second architecture.

21 . A device comprising:

a processor;

a non-transitory computer-readable storage medium storing instructions that, when executed by the processor, cause the processor to perform a method comprising:

obtaining a set of data;

generating, using the set of data, an input to a trained machine learning model configured with an architecture selected from a plurality of architectures, wherein the architecture is selected from the plurality of architectures using a quantization of at least some parameters of the machine learning model; and

providing the input to the trained machine learning model to obtain an output.

23 . The device of claim 21 , wherein the processor has a first word size and the trained machine learning model is obtained by training a machine learning model using a processor with a second word size.

23 . The device of claim 22 , wherein the first word size is smaller than the second word size.

24 . The device of claim 22 , wherein the first word size is 8 bits.

25 . The device of claim 21 , wherein the processor comprises a photonics processing system.

26 . The device of claim 21 , wherein the trained machine learning model comprises a neural network.

27 . The device of claim 25 , wherein the neural network comprises a convolutional neural network, a recurrent neural network, and/or a transformer neural network.