Analog hardware realization of neural networks
Systems and methods are provided for analog hardware realization of neural networks. The method incudes obtaining a neural network topology and weights of a trained neural network. The method also includes transforming the neural network topology to an equivalent analog network of analog components. The method also includes computing a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection between analog components of the equivalent analog network. The method also includes generating a schematic model for implementing the equivalent analog network based on the weight matrix, including selecting component values for the analog components.
1 . A method for hardware realization of neural networks, comprising:
obtaining a neural network topology and weights of a trained neural network;
transforming the neural network topology to an equivalent analog network of analog components corresponding to the neural network, including:
selecting a plurality of individual electronic analog components, wherein the plurality of individual analog components include components corresponding to analog neurons and components corresponding to connections between pairs of analog neurons;
computing a first weight matrix for the equivalent analog network based on the weights of the trained neural network, wherein each element of the weight matrix represents a respective connection between a respective pair of analog neurons;
generating a first schematic model for implementing the equivalent analog network as electronic circuitry based on the weight matrix, wherein:
generating the first schematic model includes generating a first resistance matrix from the weight matrix, each element of the first resistance matrix (i) representing a respective fixed resistance valuem, (ii) corresponding to a respective weight of the weight matrix, and (iii) defining a respective single predetermined signal pathway between a respective pair of analog neurons;
the first schematic model includes a first portion corresponding to topology of analog neurons and a second portion corresponding to connections between the analog neurons; and
generating the first schematic model includes:
selecting component values for each analog neuron; and
assigning fixed values to the connections based on the weight matrix;
obtaining new weights for an updated neural network that corresponds to the trained neural network after it has been re-trained;
computing a new weight matrix for an equivalent analog network corresponding to the updated neural network based on the new weights:
generating a new resistance matrix for the new weight matrix; and
generating a new schematic model for implementing the equivalent analog network of analog components corresponding to the updated neural network based on the new resistance matrix, so that:
a first set of one or more lithographic masks generated for fabricating the neural network includes one or more first lithographic masks corresponding to the first portion of the first schematic model and a second lithographic mask corresponding to the second portion of the first schematic model; and
a second set of one or more lithographic masks generated for fabricating the updated neural network includes the one or more first lithographic masks corresponding to the first portion of the first schematic model and a third lithographic mask corresponding to the second portion of the new schematic model, wherein:
the topology of analog neurons for the updated neural network is the same as the topology of analog neurons for the neural network prior to updating:
the new schematic model includes a third portion corresponding to topology of analog neurons and a fourth portion corresponding to connections between the analog neurons for the updated neural network;
the third portion of the new schematic model is the same as the first portion of the first schematic model; and
the fourth portion of the new schematic model is different from the second portion of the first schematic model.
2 . The method of claim 1 , further comprising:
obtaining new weights, for an updated neural network, wherein:
the new weights are different from the weights of the trained neural network; and
the updated neural network corresponds to the trained neural network after it has been re-trained;
computing a new weight matrix, for the equivalent analog network based on the new weights, wherein the new weight matrix is different from the weight matrix;
generating a new resistance matrix, for the new weight matrix, wherein the new resistance matrix is different from the resistance matrix; and
updating the first schematic model to implement the updated neural network with new weights, including:
assigning new fixed values to the new connections based on the new weight matrix; and
replacing the connections between the analog neurons in the second portion of the first schematic model with the new connections between the analog neurons.
3 . The method of claim 1 , wherein the neural network topology includes one or more layers of neurons, each layer of neurons computing respective outputs based on a respective mathematical function, and
transforming the neural network topology to the equivalent analog network of analog components comprises:
for each layer of the one or more layers of neurons:
identifying one or more function blocks, based on the respective mathematical function for the respective layer, wherein each function block has a respective schematic implementation with block outputs that conform to outputs of a respective mathematical function; and
generating a respective multilayer network of analog neurons based on arranging the one or more function blocks, wherein each analog neuron implements a respective function of the one or more function blocks, and each analog neuron of a first layer of the multilayer network is connected to one or more analog neurons of a second layer of the multilayer network.
4 . The method of claim 1 , wherein the neural network topology includes one or more layers of neurons, each layer of neurons computing respective outputs based on a respective mathematical function, and transforming the neural network topology to the equivalent analog network of analog components comprises:
decomposing a first layer of the neural network topology to a plurality of sub-layers, including decomposing a mathematical function corresponding to the first layer to obtain one or more intermediate mathematical functions, wherein each sub-layer implements an intermediate mathematical function; and
for each sub-layer of the first layer of the neural network topology:
selecting one or more sub-function blocks, based on a respective intermediate mathematical function for the respective sub-layer; and
generating a respective multilayer analog sub-network of analog neurons based on arranging the one or more sub-function blocks, wherein each analog neuron implements a respective function of the one or more sub-function blocks, and each analog neuron of a first layer of the multilayer analog sub-network is connected to one or more analog neurons of a second layer of the multilayer analog sub-network.
5 . The method of claim 4 , wherein the mathematical function corresponding to the first layer includes one or more weights, and decomposing the mathematical function includes adjusting the one or more weights such that combining the one or more intermediate functions results in the mathematical function.
6 . The method of claim 1 , further comprising:
generating an equivalent digital network of digital components for one or more output layers of the neural network topology; and
connecting output of one or more layers of the equivalent analog network to the equivalent digital network of digital components.
7 . The method of claim 1 , wherein the analog components include a plurality of operational amplifiers and a plurality of resistors, each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons.
8 . The method of claim 7 , wherein selecting component values of the analog components includes performing a gradient descent method to identify possible resistance values for the plurality of resistors.
9 . The method of claim 1 , wherein the neural network topology includes one or more gated recurrent unit (GRU) or long short-term memory (LSTM) neurons, and transforming the neural network topology includes generating one or more signal delay blocks for each recurrent connection of the one or more GRU or LSTM neurons.
10 . The method of claim 9 , wherein the one or more signal delay blocks are activated at a frequency that matches a predetermined input signal frequency for the neural network topology.
11 . The method of claim 1 , wherein the neural network topology includes one or more layers of neurons that perform unlimited activation functions, and transforming the neural network topology includes applying one or more transformations selected from the group consisting of:
replacing the unlimited activation functions with limited activation; and
adjusting connections or weights of the equivalent analog network such that, for predetermined one or more inputs, a difference in output between the trained neural network and the equivalent analog network is minimized.
12 . The method of claim 1 , further comprising:
generating lithographic masks for fabricating a circuit implementing the equivalent analog network of analog components corresponding to the neural network based on the first resistance matrix; and
generating lithographic masks for fabricating a circuit implementing the equivalent analog network of analog components corresponding to the updated neural network based on the new resistance matrix.
13 . The method of claim 1 , wherein the trained neural network is trained using software simulations to generate the weights.
14 . The method of claim 1 , further comprising fabricating an electronic circuit based on the first set of one or more lithographic masks and the second lithographic masks using a lithographic process.
15 . The method of claim 1 , further comprising fabricating an electronic circuit at least partially based on (i) reusing the first set of one or more lithographic masks for a plurality of analog neurons of the equivalent analog network, and (ii) using the new lithographic mask for one or more connections between the plurality of analog neurons, wherein the new weights (i) correspond to the one or more connections and (ii) have new corresponding fixed resistance values.
16 . The method of claim 1 , further comprising transforming the first schematic model into a physical manufacturable circuit layout representing an integrated electronic circuit's arrangement on a semiconductor substrate, wherein the circuit layout includes the selected individual electronic analog components and fixed component values in accordance with the schematic model.
17 . The method of claim 1 , wherein the second portion can be updated, independently of the first portion, to have new fixed values.
18 . The method of claim 1 , wherein the second lithographic mask can be replaced, independently of the first set of one or more lithographic masks, by a new lithographic mask that includes new fixed values.
19 . A system for hardware realization of neural networks, comprising:
one or more processors;
memory;
wherein the memory stores one or more programs configured for execution by the one or more processors, and the one or more programs comprising instructions for:
obtaining a neural network topology and weights of a trained neural network;
transforming the neural network topology to an equivalent analog network of analog components corresponding to the neural network, including:
selecting a plurality of individual electronic analog components, wherein the plurality of individual analog components include components corresponding to analog neurons and components corresponding to connections between pairs of analog neurons;
computing a first weight matrix for the equivalent analog network based on the weights of the trained neural network, wherein each element of the weight matrix represents a respective connection between a respective pair of analog neurons;
generating a first schematic model for implementing the equivalent analog network as electronic circuitry based on the weight matrix, wherein:
generating the first schematic model includes generating a first resistance matrix from the weight matrix, each element of the first resistance matrix (i) representing a respective fixed resistance value, (ii) corresponding to a respective weight of the weight matrix, and (iii) defining a respective single predetermined signal pathway between a respective pair of analog neurons:
the first schematic model includes a first portion corresponding to topology of analog neurons and a second portion corresponding to connections between the analog neurons; and
generating the first schematic model includes:
selecting component values for each analog neuron; and
assigning fixed values to the connections based on the weight matrix;
obtaining new weights for an updated neural network that corresponds to the trained neural network after it has been re-trained;
computing a new weight matrix for an equivalent analog network corresponding to the updated neural network based on the new weights;
generating a new resistance matrix for the new weight matrix; and
generating a new schematic model for implementing the equivalent analog network of analog components corresponding to the updated neural network based on the new resistance matrix, so that:
a first set of one or more lithographic masks generated for fabricating the neural network includes one or more first lithographic masks corresponding to the first portion of the first schematic model and a second lithographic mask corresponding to the second portion of the first schematic model; and
a second set of one or more lithographic masks generated for fabricating the updated neural network includes the one or more first lithographic masks corresponding to the first portion of the first schematic model and a third lithographic mask corresponding to the second portion of the new schematic model, wherein:
the topology of analog neurons for the updated neural network is the same as the topology of analog neurons for the neural network prior to updating:
the new schematic model includes a third portion corresponding to topology of analog neurons and a fourth portion corresponding to connections between the analog neurons for the updated neural network;
the third portion of the new schematic model is the same as the first portion of the first schematic model; and
the fourth portion of the new schematic model is different from the second portion of the first schematic model.
20 . The system of claim 19 , wherein generating the schematic model includes generating a resistance matrix from the weight matrix, each element of the resistance matrix (i) representing a respective fixed resistance value (ii) corresponding to a respective weight of the weight matrix, and (iii) defining a respective single predetermined signal pathway between a respective pair of analog neurons.
21 . A non-transitory computer readable storage medium storing one or more programs configured for execution by a computer system having one or more processors, the one or more programs comprising instructions for:
obtaining a neural network topology and weights of a trained neural network;
transforming the neural network topology to an equivalent analog network of analog components corresponding to the neural network, including:
selecting a plurality of individual electronic analog components, wherein the plurality of individual analog components include components corresponding to analog neurons and components corresponding to connections between pairs of analog neurons;
computing a first weight matrix for the equivalent analog network based on the weights of the trained neural network, wherein each element of the weight matrix represents a respective connection between a respective pair of analog neurons;
generating a first schematic model for implementing the equivalent analog network as electronic circuitry based on the weight matrix, wherein:
generating the first schematic model includes generating a first resistance matrix from the weight matrix, each element of the first resistance matrix (i) representing a respective fixed resistance value, (ii) corresponding to a respective weight of the weight matrix, and (iii) defining a respective single predetermined signal pathway between a respective pair of analog neurons;
the first schematic model includes a first portion corresponding to topology of analog neurons and a second portion corresponding to connections between the analog neurons; and
generating the first schematic model includes:
selecting component values for each analog neuron; and
assigning fixed values to the connections based on the weight matrix;
obtaining new weights for an updated neural network that corresponds to the trained neural network after it has been re-trained;
computing a new weight matrix for an equivalent analog network corresponding to the updated neural network based on the new weights:
generating a new resistance matrix for the new weight matrix; and
generating a new schematic model for implementing the equivalent analog network of analog components corresponding to the updated neural network based on the new resistance matrix, so that:
a first set of one or more lithographic masks generated for fabricating the neural network includes one or more first lithographic masks corresponding to the first portion of the first schematic model and a second lithographic mask corresponding to the second portion of the first schematic model; and
a second set of one or more lithographic masks generated for fabricating the updated neural network includes the one or more first lithographic masks corresponding to the first portion of the first schematic model and a third lithographic mask corresponding to the second portion of the new schematic model, wherein:
the topology of analog neurons for the updated neural network is the same as the topology of analog neurons for the neural network prior to updating;
the new schematic model includes a third portion corresponding to topology of analog neurons and a fourth portion corresponding to connections between the analog neurons for the updated neural network;
the third portion of the new schematic model is the same as the first portion of the first schematic model; and
the fourth portion of the new schematic model is different from the second portion of the first schematic model.