Granted Patent B1

US 12,579,416 · App. 17/543,471 · Granted Mar 17, 2026

Inventors: Kenneth Duong (San Jose, CA); Jung Ko (San Jose, CA); Steven L. Teig (Menlo Park, CA); Won Rhee (Los Altos, CA)

Assignee: Amazon Technologies, Inc.

G06N3/063G06F17/16G06N3/048

Abstract

Claims (38)

1 . A neural network inference circuit for executing a neural network that comprises a plurality of computation nodes, each respective computation node of a set of the computation nodes comprising (i) a respective linear function that comprises a respective dot product of input values for the computation node and weight values for the computation node and (ii) a respective non-linear activation function, the neural network inference circuit comprising:

a set of dot product circuits to compute the dot product for a computation node; and

a post-processing circuit to compute (i) a result of the linear function for the computation node based on the dot product for the computation node and (ii) an output for the computation node by applying a piecewise linear function to the result of the linear function for the computation node to apply the non-linear activation function for the computation node.

2 . The neural network inference circuit of claim 1 , wherein the neural network comprises at least a particular layer using exponential functions, wherein the piecewise linear function is applied to input values of the particular layer to compute the exponential functions.

3 . The neural network inference circuit of claim 2 , wherein the particular layer computes a softmax function for a set of input values.

4 . The neural network inference circuit of claim 1 further comprising:

a plurality of sets of dot product circuits, each respective set of dot product circuits of the plurality of sets of dot product circuits to compute the dot product for a respective computation node of a particular layer; and

a plurality of post-processing circuits, each respective post-processing circuit of the plurality of post-processing circuits corresponding to a respective set of dot product circuits and to compute (i) a result of the linear function for the respective computation node of the particular layer and (ii) an output for the respective computation node by applying a respective piecewise linear function to the result of the linear function for the respective computation node to apply the non-linear activation function for the respective computation node.

5 . The neural network inference circuit of claim 4 , wherein different piecewise linear functions are applied for different computation nodes of the particular layer.

6 . The neural network inference circuit of claim 4 , wherein all of the post-processing circuits in the plurality of post-processing circuits apply a same piecewise linear function for the different computation nodes of the particular layer.

7 . The neural network inference circuit of claim 4 , wherein each post-processing circuit computes outputs for a different plurality of computation nodes of the particular layer based on a plurality of different dot products computed by the corresponding set of dot product circuits.

8 . The neural network inference circuit of claim 1 , wherein the post-processing circuit comprises:

a shift circuit to add a first configured value to the computed dot product for the computation node;

a scale circuit to multiply an output of the shift circuit by a second configured value; and

a piecewise linear computation circuit to apply the piecewise linear function to an output of the scale circuit and compute the output for the computation node.

9 . The neural network inference circuit of claim 8 , wherein the first configured value is based on at least one of a bias value for the computation node and a batch normalization shift value for the computation node.

10 . The neural network inference circuit of claim 8 , wherein the second configured value is based on at least one of a weight scale value for the computation node and a batch normalization scale value for the computation node.

11 . The neural network inference circuit of claim 1 , wherein the post-processing circuit comprises a configurable piecewise linear function circuit that receives an input value and applies a configured piecewise linear function to the input value.

12 . The neural network inference circuit of claim 11 , wherein:

the configured piecewise linear function comprises a plurality of linear segments; and

the configurable piecewise linear function circuit receives, as configuration data, a plurality of segment endpoint values of the plurality of linear segments.

13 . The neural network inference circuit of claim 12 , wherein the configurable piecewise linear function circuit comprises:

an interval decoder circuit that determines, based on the received input value, to which particular linear segment of the plurality of linear segments the input value belongs; and

an input adjustment circuit that determines a difference between the received input value and a start of the particular linear segment.

14 . The neural network inference circuit of claim 13 , wherein the configurable piecewise linear function circuit further comprises:

a first multiplexer that selects a segment starting endpoint value from the plurality of segment endpoint values based on an output of the interval decoder circuit;

a second multiplexer that selects a segment ending endpoint value from the plurality of segment endpoint values based on the output of the interval decoder circuit; and

a subtractor circuit that subtracts the segment starting endpoint value from the segment ending endpoint value.

15 . The neural network inference circuit of claim 14 , wherein the configurable piecewise linear function circuit further comprises:

a multiplier circuit that multiplies an output of the subtractor circuit by an output of the input adjustment circuit;

a bitshift circuit that divides an output of the multiplier circuit by a length of the particular segment; and

an adder circuit that adds an output of the bitshift circuit to the segment starting endpoint value to compute an output of the configured piecewise linear function.

16 . The neural network inference circuit of claim 12 , wherein each of linear segments has a same length.

17 . The neural network inference circuit of claim 12 , wherein a number of linear segments of the piecewise linear function is a power of two value.

18 . The neural network inference circuit of claim 1 , wherein:

the set of dot product circuits computes dot products for a plurality of computation nodes of a plurality of different layers of the neural network; and

the post-processing circuit computes outputs for the plurality of computation nodes.

19 . The neural network inference circuit of claim 1 further comprising a math circuit to apply math functions to input values for computation nodes that do not include weight values, wherein the post-processing circuit applies a piecewise linear function to results of the math functions applied by the math circuit.

Assignments (3)

Continuity (1)

References Cited (216)

Cited By (1)

US 12,682,236