Training apparatus with integrated circuits for training logic gate networks
The disclosure describes integrated circuits and training apparatuses for efficiently training learnable logic networks. One or more hardware-implemented learnable logic engines execute forward and backward propagation through differentiable relaxations of Boolean logic gates arranged in configurable clusters. Each engine processes multiple inputs and outputs with shared parameter sets, supports variable numbers of gate inputs, and may reuse locally stored parameters across batched samples to reduce memory bandwidth. Engines and associated cores employ mixed-precision arithmetic, including low-precision activations and higher-precision gradients with on-chip gradient accumulation. Configurable interconnects route activations between engines, and topology bits select among logic operations and wiring options. Systems including a host processor orchestrate the use of the learnable logic engines to design fixed logic gate networks for efficient inference.
1 . An integrated circuit for training learnable logic networks, comprising:
memory configured to store activation values and parameter values of a learnable logic network; and
one or more hardware-implemented learnable logic engines, at least one of the one or more learnable logic engines being configured to:
process a plurality N of input activations, where N is an integer greater than or equal to two, and
to generate M output activations, where M is an integer greater than or equal to one, through S learnable logic gate operations, where S is an integer greater than or equal to one,
wherein at least one of the one or more learnable logic engines is configured to process multiple sets of inputs using shared parameter values.
2 . The integrated circuit of claim 1 , further comprising configurable interconnect circuitry between the one or more learnable logic engines, the configurable interconnect circuitry being configured to route activations and gradients between one or more outputs and one or more inputs of one or more learnable logic engines.
3 . The integrated circuit of claim 1 , wherein one or more of the learnable logic engines comprises internal reconfigurable interconnect circuitry with one or more learnable logic operators, the internal reconfigurable interconnect circuitry being configured to internally route activations and gradients within the learnable logic engine.
4 . The integrated circuit of claim 3 , wherein at least one of the one or more learnable logic engines is configured to process multiple sets of inputs using shared parameter values.
5 . The integrated circuit of claim 1 , wherein at least one of the one or more learnable logic engines maintains different numerical precisions for forward-propagation input activations and for gradients used during backpropagation, and wherein the forward-propagation input activations are represented using at most twelve bits of precision.
6 . The integrated circuit of claim 1 , wherein, for at least one of the learnable logic engines, a precision of input activations is less than a precision of input activation gradients.
7 . The integrated circuit of claim 1 , wherein some of the hardware-implemented learnable logic engines are for forward propagation, and wherein some of the hardware-implemented learnable logic engines are for backward propagation.
8 . The integrated circuit of claim 1 , wherein at least one of the one or more learnable logic engines supports configurable connectivity between the learnable logic gate operations, the configurable connectivity being specified by configuration bits that control interconnections among inputs and outputs of the learnable logic gate operations within the learnable logic engine.
9 . The integrated circuit of claim 1 , further comprising gradient accumulation buffers co-located with or within one or more learnable logic engines, wherein each gradient accumulation buffer is configured to accumulate parameter gradients.
10 . The integrated circuit of claim 1 , wherein one or more learnable logic gate operations realize a parameterized relaxation of a choice of a logic operator from a set of at least two logic operators, including Boolean operations and wiring options, and
wherein the Boolean operations and wiring options comprise at least one of: AND operators, OR operators, NAND operators, NOR operators, XOR operators, XNOR operators, inverter operators, and direct-connection operators.
11 . The integrated circuit of claim 1 , wherein at least one of the learnable logic engines hardware-implements at least one of the learnable logic gate operators using polynomial computations.
12 . The integrated circuit of claim 1 , wherein at least one of the learnable logic engines hardware-implements at least one of the learnable logic gate operators using probabilistic relaxations.
13 . The integrated circuit of claim 1 , wherein at least one of the learnable logic engines hardware-implements at least one of the learnable logic gate operators using fuzzy-logic operations.
14 . The integrated circuit of claim 1 , wherein at least one of the one or more learnable logic engines has at least three inputs and performs exactly one learnable logic gate operation.
15 . The integrated circuit of claim 1 , wherein at least one of the learnable logic engines supports a vectorized dimension over input activations such that, when loading a parameterization, the parameterization is reused for a plurality of input sets.
16 . The integrated circuit of claim 1 , further comprising:
a vectorized data type for input activation gradients of the learnable logic engine; and
a hardware-implemented atomic addition instruction for the vectorized data type of the input activation gradients.
17 . The integrated circuit of claim 1 , wherein at least one of the one or more learnable logic engines comprises circuitry for computing w 0 ·a·b+w 1 ·a+w 2 ·b+w 3 , wherein a, b are functions also dependent on input activations and w 0 , w 1 , w 2 , w 3 are functions also dependent on the parameters.
18 . The integrated circuit of claim 1 , wherein the memory is coupled to the one or more hardware-implemented learnable logic engines,
wherein one or more learnable logic gate operations realize a parameterized relaxation of a choice of a logic operator from a set of at least two logic operators, including Boolean operations and wiring options,
wherein the Boolean operations and wiring options comprise at least one of: AND operators, OR operators, NAND operators, NOR operators, XOR operators, XNOR operators, inverter operators, and direct-connection operators,
wherein the learnable logic engine performs at least two learnable logic gate operations, and after forward propagating through the learnable logic engine, some intermediate activations are not stored in the memory,
wherein, during backpropagation through the learnable logic engine, the intermediate activations not stored in the memory are recomputed rather than retrieved from the memory, and
wherein at least one of the learnable logic engines supports a vectorized dimension over input activations such that, when loading a parameterization, the parameterization is reused for a plurality of input sets.
19 . The integrated circuit of claim 1 , wherein at least one of the one or more learnable logic engines is configured to:
process a plurality N of input activations and a plurality M of output activation gradients, where N is an integer greater than or equal to two and M is an integer greater than or equal to one, and
to generate a plurality N of input activation gradients, through S learnable logic gate operations, where S is an integer greater than or equal to one.
20 . The integrated circuit of claim 1 , further comprising:
a hardware-implemented vectorized instruction configured for forward propagation through a learnable logic gate operator,
wherein the learnable logic operator has:
at least two vectorized activation inputs,
at least one vectorized output, and
a parameterization is provided through three or more parameters, and
wherein the vectorized instruction processes the vectorized activation inputs, each vectorized activation input comprising two or more activation values, each activation value represented by no more than sixteen bits of precision.
21 . The integrated circuit of claim 20 , wherein the hardware-implemented vectorized instruction has three inputs, comprising: two input activation sets a, b with each 4 vectorized activations (a 0 , a 1 , a 2 , a 3 , b 0 , b 1 , b 2 , b 3 ) of each 8-bit precision, each of the two sets represented via one vectorized datatype with 32 bits, and one parameterization vector with four parameters, w 0 , w 1 , w 2 , w 3 represented via one vectorized datatype with at most 32; and wherein the hardware-implemented vectorized instruction provides one output activation set c with 4 vectorized activations (c 0 , c 1 , c 2 , c 3 ) of each 8-bit precision, represented via one vectorized datatype with 32 bits, and wherein the hardware-implemented vectorized instruction computes c 0 as a function of w 0 ·a 0 ·b 0 +w 1 ·a 0 +w 2 ·b 0 +w 3 , computes c 1 as a function of w 0 ·a 1 ·b 1 +w 1 ·a 1 +w 2 b 1 +w 3 , computes c 2 as a function of w 0 ·a 2 ·b 2 +w 1 ·a 2 +w 2 b 2 +w 3 , and computes c 3 as a function of w 0 ·a 3 ·b 3 +w 1 ·a 3 +w 2 b 3 +w 3 .
22 . The integrated circuit of claim 1 , wherein at least one of the one or more learnable logic engines utilizes a parameterization that non-linearly depends on a parameterization used by an optimization algorithm.
23 . The integrated circuit of claim 1 , wherein at least one of the one or more learnable logic engines has at least four inputs and performs at least two learnable logic gate operations.
24 . The system claim 1 , wherein at least one of the learnable logic engines hardware-implements a learnable logic gate operator that realizes a differentially parameterized selection of a logic gate operator from among a set of Boolean operations and wiring options, the learnable logic gate operator being implemented using at least one of polynomial computations, probabilistic relaxations, and fuzzy-logic operations.
25 . An integrated circuit for training learnable logic networks, comprising:
memory configured to store activation values and parameter values of a learnable logic network; and
one or more hardware-implemented learnable logic engines, at least one of the one or more learnable logic engines being configured to:
process a plurality N of input activations, where N is an integer greater than or equal to two, and
to generate M output activations, where M is an integer greater than or equal to one, through S learnable logic gate operations, where S is an integer greater than or equal to one,
wherein the learnable logic engine performs at least two learnable logic gate operations, and after forward propagating through the learnable logic engine, some intermediate activations are not stored in the memory, and
wherein, during backpropagation through the learnable logic engine, the intermediate activations not stored in the memory are recomputed rather than retrieved from the memory.
26 . The integrated circuit of claim 25 , further comprising configurable interconnect circuitry between the one or more learnable logic engines, the configurable interconnect circuitry being configured to route activations and gradients between one or more outputs and one or more inputs of one or more learnable logic engines.
27 . The integrated circuit of claim 25 , wherein one or more of the learnable logic engines comprises internal reconfigurable interconnect circuitry with one or more learnable logic operators, the internal reconfigurable interconnect circuitry being configured to internally route activations and gradients within the learnable logic engine.
28 . The integrated circuit of claim 25 , wherein at least one of the one or more learnable logic engines maintains different numerical precisions for forward-propagation input activations and for gradients used during backpropagation, and wherein the forward-propagation input activations are represented using at most twelve bits of precision.
29 . The integrated circuit of claim 25 , wherein, for at least one of the learnable logic engines, a precision of input activations is less than a precision of input activation gradients.
30 . The integrated circuit of claim 25 , wherein some of the hardware-implemented learnable logic engines are for forward propagation, and wherein some of the hardware-implemented learnable logic engines are for backward propagation.
31 . The integrated circuit of claim 25 , wherein at least one of the one or more learnable logic engines supports configurable connectivity between the learnable logic gate operations, the configurable connectivity being specified by configuration bits that control interconnections among inputs and outputs of the learnable logic gate operations within the learnable logic engine.
32 . The integrated circuit of claim 25 , further comprising gradient accumulation buffers co-located with or within one or more learnable logic engines, wherein each gradient accumulation buffer is configured to accumulate parameter gradients.
33 . The integrated circuit of claim 25 , wherein one or more learnable logic gate operations realize a parameterized relaxation of a choice of a logic operator from a set of at least two logic operators, including Boolean operations and wiring options, and
wherein the Boolean operations and wiring options comprise at least one of: AND operators, OR operators, NAND operators, NOR operators, XOR operators, XNOR operators, inverter operators, and direct-connection operators.
34 . The integrated circuit of claim 25 , wherein at least one of the learnable logic engines hardware-implements at least one of the learnable logic gate operators using polynomial computations.
35 . The integrated circuit of claim 25 , wherein at least one of the learnable logic engines hardware-implements at least one of the learnable logic gate operators using probabilistic relaxations.
36 . The integrated circuit of claim 25 , wherein at least one of the learnable logic engines hardware-implements at least one of the learnable logic gate operators using fuzzy-logic operations.
37 . The integrated circuit of claim 25 , wherein at least one of the one or more learnable logic engines utilizes a parameterization that non-linearly depends on a parameterization used by an optimization algorithm.
38 . The integrated circuit of claim 25 , wherein at least one of the one or more learnable logic engines has at least four inputs and performs at least two learnable logic gate operations.
39 . The integrated circuit of claim 25 , wherein at least one of the one or more learnable logic engines has at least three inputs and performs exactly one learnable logic gate operation.
40 . The integrated circuit of claim 25 , wherein at least one of the learnable logic engines supports a vectorized dimension over input activations such that, when loading a parameterization, the parameterization is reused for a plurality of input sets.
41 . The integrated circuit of claim 25 , further comprising:
a vectorized data type for input activation gradients of the learnable logic engine; and
a hardware-implemented atomic addition instruction for the vectorized data type of the input activation gradients.
42 . The integrated circuit of claim 25 , wherein at least one of the one or more learnable logic engines comprises circuitry for computing w 0 ·a·b+w 1 ·a+w 2 ·b+w 3 , wherein a, b are functions also dependent on input activations and w 0 , w 1 , w 2 , w 3 are functions also dependent on the parameters.
43 . The integrated circuit of claim 25 , wherein at least one of the one or more learnable logic engines is configured to:
process a plurality N of input activations and a plurality M of output activation gradients, where N is an integer greater than or equal to two and M is an integer greater than or equal to one, and
to generate a plurality N of input activation gradients, through S learnable logic gate operations, where S is an integer greater than or equal to one.
44 . The system claim 25 , wherein at least one of the learnable logic engines hardware-implements a learnable logic gate operator that realizes a differentiably parameterized selection of a logic gate operator from among a set of Boolean operations and wiring options, the learnable logic gate operator being implemented using at least one of polynomial computations, probabilistic relaxations, and fuzzy-logic operations.
45 . An integrated circuit for training learnable logic networks, comprising:
memory configured to store activation values and parameter values of a learnable logic network; and
one or more hardware-implemented learnable logic engines, at least one of the one or more learnable logic engines being configured to:
process a plurality N of input activations, where N is an integer greater than or equal to two, and
to generate M output activations, where M is an integer greater than or equal to one, through S learnable logic gate operations, where S is an integer greater than or equal to one,
wherein at least one of the one or more learnable logic engines utilizes a parameterization that non-linearly depends on a parameterization used by an optimization algorithm.
46 . The integrated circuit of claim 45 , wherein one or more learnable logic gate operations realize a parameterized relaxation of a choice of a logic operator from a set of at least two logic operators, including Boolean operations and wiring options, and
wherein the Boolean operations and wiring options comprise at least one of: AND operators, OR operators, NAND operators, NOR operators, XOR operators, XNOR operators, inverter operators, and direct-connection operators.
47 . The integrated circuit of claim 45 , wherein at least one of the one or more learnable logic engines has at least three inputs and performs exactly one learnable logic gate operation.
48 . An integrated circuit for training learnable logic networks, comprising:
memory configured to store activation values and parameter values of a learnable logic network; and
one or more hardware-implemented learnable logic engines, at least one of the one or more learnable logic engines being configured to:
process a plurality N of input activations, where N is an integer greater than or equal to two, and
to generate M output activations, where M is an integer greater than or equal to one, through S learnable logic gate operations, where S is an integer greater than or equal to one,
wherein at least one of the one or more learnable logic engines has at least four inputs and performs at least two learnable logic gate operations.
49 . The integrated circuit of claim 48 , wherein one or more learnable logic gate operations realize a parameterized relaxation of a choice of a logic operator from a set of at least two logic operators, including Boolean operations and wiring options, and
wherein the Boolean operations and wiring options comprise at least one of: AND operators, OR operators, NAND operators, NOR operators, XOR operators, XNOR operators, inverter operators, and direct-connection operators.