RESIDUAL INITIALIZATION FOR TRAINING NODE NETWORKS
This disclosure describes systems, methods, and computer-readable media for initializing and training differentiable logic gate networks for machine-learning inference. In various embodiments, parameters of learnable logic gate operators are initialized using a residual initialization that biases the initial probability distribution of discrete gate choices toward a feedforward operation, such as a wire or an inverter, thereby preserving information flow and mitigating vanishing gradients in deep networks. During training, the gate parameters are updated so that gates can remain as feedforward connections or transition to other logic operations as needed. The residual initialization can be applied to convolutional logic gate-tree architectures and pooling operations, and trained networks can be discretized and synthesized into efficient logic circuits in hardware or software.
1 . A computer-implemented method for training a computational network, comprising:
instantiating an untrained node network, wherein at least some nodes of the untrained node network are associated with a respective set of trainable parameters usable to select a node function for the node from a set of candidate node functions, including candidate node functions that depend on at least two node inputs;
performing residual initialization of the untrained node network by initializing the trainable parameters to select, for each respective node of at least some of the nodes, a candidate node function that functionally depends on only a single one of the node inputs;
training the node network via a plurality of training iterations, wherein training the node network includes updating at least some of the trainable parameters of at least some of the nodes; and
generating, after completion of the plurality of training iterations, a trained node network by selecting a fixed node function based on final trainable parameters for at least some of the nodes and outputting a fixed representation of the trained node network.
2 . A computer-implemented method for training a computational network, comprising:
instantiating a node network comprising a plurality of nodes and connections, the plurality of nodes including a set of multi-input nodes each configured to receive at least two inputs and produce a node output;
for each multi-input node in the set, providing a plurality of candidate discrete operations selectable for use in a discretized form of the node network, each candidate discrete operation being configured to map the at least two inputs of the multi-input node to an output value;
representing, during training, each multi-input node in a trainable form parameterized by trainable parameters that are adjustable by an optimization algorithm, the trainable form being configured to compute the node output from the at least two inputs in accordance with current values of the trainable parameters;
performing a residual initialization by initializing the trainable parameters for each of a plurality of the multi-input nodes such that, prior to iterative training, the corresponding trainable form causes the node output to be determined primarily by a residual operation that propagates information from a single input of the multi-input node to the node output;
training the node network by iteratively adjusting the trainable parameters using the optimization algorithm based on a training objective;
wherein, during the training, at least some of the residually initialized multi-input nodes transition such that the corresponding trainable form causes the node output to be determined primarily by a different candidate discrete operation that functionally depends on each of at least two inputs of the multi-input node; and
after the training, generating the discretized form of the node network by applying a discretization rule to the trainable parameters of at least some nodes to select corresponding candidate discrete operations for the at least some nodes.
3 . The method of claim 2 , wherein the plurality of the multi-input nodes comprises a majority of the set of multi-input nodes.
4 . The method of claim 2 , wherein the plurality of the multi-input nodes comprises residualizable nodes, each residualizable node being configurable, in the discretized form of the node network, to implement (i) a residual operation that causes the node output to be functionally dependent on only one input of the residualizable node, and (ii) a non-residual operation that causes the node output to be functionally dependent on each of at least two inputs of the residualizable node.
5 . The method of claim 4 , wherein the plurality of the multi-input nodes comprises a majority of the residualizable nodes,
wherein at least some of the multi-input nodes comprise learnable differentiable logic gates,
wherein, for each of the learnable differentiable logic gates, the plurality of candidate discrete operations includes at least one of: AND, OR, NAND, NOR, XOR, XNOR, a constant operator, a pass-through operator, or an inverter operator,
wherein the residual operation comprises at least one of: (i) a pass-through operation that outputs a selected input of the multi-input node without combining the selected input with another input, or (ii) an inverter operation that outputs an inversion of the selected input, and
wherein the optimization algorithm is a gradient-based optimization algorithm using one of: (i) gradients and (ii) gradient-approximations.
6 . The method of claim 2 , wherein the node network comprises a plurality of nodes arranged in a plurality of layers, including at least a first layer and a second layer, and wherein a forward-propagation of inputs through the node network comprises:
for each node of the first layer, computing a real-valued non-binary differentiable output that is a real-valued non-binarizing non-linear differentiable function of:
(i) input activations to the node, and
(ii) current trainable parameters of the respective node; and
for each node of the second layer, computing a real-valued non-binary differentiable output that is a real-valued non-binarizing non-linear differentiable function of the:
(i) input activations to the node, at least some of which are the real-valued non-binary differentiable outputs of the first layer, and
(ii) current differentiable parameters of the respective node.
7 . The method of claim 4 , wherein the plurality of the multi-input nodes comprises a majority of the residualizable nodes.
8 . The method of claim 2 , wherein at least some of the multi-input nodes comprise learnable differentiable logic gates.
9 . The method of claim 8 , wherein, for each of the learnable differentiable logic gates, the plurality of candidate discrete operations includes at least one of: AND, OR, NAND, NOR, XOR, XNOR, a constant operator, a pass-through operator, or an inverter operator.
10 . The method of claim 8 , wherein representing each multi-input node in the trainable form comprises mapping the trainable parameters to a categorical distribution over the plurality of candidate discrete operations using a softmax operation.
11 . The method of claim 10 , wherein the trainable form computes the node output as a weighted combination of outputs of at least some of the candidate discrete operations in accordance with the categorical distribution.
12 . The method of claim 2 , wherein the residual operation comprises at least one of: (i) a pass-through operation that outputs a selected input of the multi-input node without combining the selected input with another input, or (ii) an inverter operation that outputs an inversion of the selected input.
13 . The method of claim 2 , wherein the discretization rule selects, for a given multi-input node, one of: (i) a candidate discrete operation corresponding to a largest value among the trainable parameters for the given multi-input node, or (ii) a largest probability derived from the trainable parameters for the given multi-input node.
14 . The method of claim 2 , wherein the optimization algorithm is a gradient-based optimization algorithm using one of: (i) gradients and (ii) gradient-approximations.
15 . The method of claim 2 , wherein generating the discretized form of the node network comprises outputting a fixed representation comprising at least one of: a gate-level netlist, a circuit description, a hardware-description-language representation, configuration data for programming a programmable logic device, or a lookup-table representation defining per-node operations.
16 . The method of claim 2 , wherein generating the discretized form includes generating a gate-level netlist configured for synthesis into an application-specific integrated circuit (ASIC).
17 . The method of claim 2 , wherein generating the discretized form includes generating configuration data for programming a field-programmable gate array (FPGA) to implement the discretized form of the node network.
18 . A computing system for generating a trained node network for performing an inference task, comprising:
at least one processor; and
a memory coupled to the at least one processor and storing instructions that, when executed by the at least one processor, cause the computing system to:
instantiate, in the memory, an untrained node network comprising a plurality of nodes arranged in a network topology, each node configured to receive a plurality of node inputs, including at least a first node input and a second node input, and to produce a node output, wherein each of at least some of the nodes is associated with a respective set of trainable parameters usable to select, for a discretized form of the node network, a node function for the node from a set of candidate node functions;
define a subset of the plurality of nodes;
perform residual initialization of the untrained node network by initializing, for each node of the subset of nodes, the trainable parameters such that application of a discretization rule to the trainable parameters selects, for the node, a candidate node function that functionally depends on only a single one of the node inputs;
iteratively train the node network via a plurality of training iterations, where at least some of the training iterations include updating at least some of the trainable parameters; and
generate, after completion of the plurality of training iterations, a trained node network by applying the discretization rule to the trainable parameters to select, for at least some of the nodes, respective node functions and outputting a fixed representation of the trained node network.
19 . An application-specific integrated circuit (ASIC) with logic circuitry that is an implementation of a fixed logic gate network, manufactured using a process comprising:
instantiating a node network comprising a plurality of nodes and connections, the plurality of nodes including a set of multi-input nodes each configured to receive at least two inputs and produce a node output;
for each multi-input node in the set, providing a plurality of candidate discrete operations selectable for use in a discretized form of the node network, each candidate discrete operation being configured to map the at least two inputs of the multi-input node to an output value;
representing, during training, each multi-input node in a trainable form parameterized by trainable parameters that are adjustable by an optimization algorithm, the trainable form being configured to compute the node output from the at least two inputs in accordance with current values of the trainable parameters;
performing a residual initialization by initializing the trainable parameters for each of a plurality of the multi-input nodes such that, prior to iterative training, the corresponding trainable form causes the node output to be determined primarily by a residual operation that propagates information from a single input of the multi-input node to the node output;
training the node network by iteratively adjusting the trainable parameters using the optimization algorithm based on a training objective;
wherein, during the training, at least some of the residually initialized multi-input nodes transition such that the corresponding trainable form causes the node output to be determined primarily by a different candidate discrete operation that functionally depends on each of at least two inputs of the multi-input node; and
after the training, generating the discretized form of the node network as a fixed logic gate network by applying a discretization rule to the trainable parameters of at least some nodes to select corresponding candidate discrete operations for the at least some nodes.
20 . The ASIC of claim 19 , wherein the process further comprises one or more of:
synthesizing the fixed logic gate network;
technology-mapping the fixed logic gate network; and
place-and-routing operations for circuit components.
21 . The ASIC of claim 19 , wherein the process further comprises fabricating circuit components corresponding to the fixed logic gate network on a silicon wafer using one or more of a photolithographic patterning process, an ion implantation process, an etching process, and a deposition process.