Granted Patent B1

US 12,682,232 · App. 17/935,419 · Granted Jul 14, 2026

Inventors: Paul Gilbert Meyer (Jericho, VT); Sundeep Amirineni (Cedar Park, TX); Ron Diamant (San Jose, CA)

Assignee: Amazon Technologies, Inc.

G06N3/08G06F18/10

Abstract

Claims (54)

1 . A computer-implemented method comprising:

receiving an input tensor representing training data to train a neural network;

performing a normalization operation on the input tensor, the normalization operation including:

streaming data elements of a vector of the input tensor from a memory into a vector compute channel having a pipeline formed by coupling a set of compute stages in series;

computing a first running mean and a first running variance associated with data elements of the vector having odd sequence indices, and a second running mean and a second running variance associated with data elements of the vector having even sequence indices from a single pass of streaming the data elements into the vector compute channel;

aggregating the first running mean and the second running mean to compute a mean associated with the data elements of the vector, and aggregating the first running variance and the second running variance to compute a variance associated with the data elements of the vector; and

modifying the input tensor to generate a normalized input tensor based on the computed mean and the computed variance;

writing the normalized input tensor back to the memory; and

training the neural network using the normalized input tensor.

2 . The computer-implemented method of claim 1 , wherein the vector compute channel includes eight compute stages coupled in series in a pipeline.

3 . The computer-implemented method of claim 2 , wherein for each of the data elements streamed into the vector compute channel:

performing, at a first compute stage of the vector compute channel, multiplication of the data element with a reciprocal of a sequence index of the data element;

performing, at a second compute stage of the vector compute channel, subtraction of the reciprocal of the sequence index of the data element from a floating-point value of one;

performing, at a third compute stage of the vector compute channel, multiplication of an output of the second compute stage with a previous mean value computed for a previous data element of the vector;

performing, at a fourth compute stage of the vector compute channel, addition of an output of the third compute stage to an output of the second compute stage to compute a mean value for the data element;

performing, at a fifth compute stage of the vector compute channel, subtraction of the mean value from the data element;

performing, at a sixth compute stage of the vector compute channel, subtraction of the previous mean value from the data element;

performing, at a seventh compute stage of the vector compute channel, multiplication of an output of the fifth compute stage with an output of the sixth compute stage; and

performing, at an eighth compute stage of the vector compute channel, addition of a previous variance value computed for the previous data element to an output of the seventh compute stage to compute a variance value for the data element.

4 . The computer-implemented method of claim 1 , wherein the normalization operation includes streaming reciprocals of sequence indices from a parameters table into the vector compute channel in parallel with the data elements of the vector.

5 . A computer-implemented method comprising:

serially inputting data elements of a vector into a compute channel;

generating, in the compute channel, a first running mean and a first running variance associated with data elements of the vector having odd sequence indices, and a second running mean and a second running variance associated with data elements of the vector having even sequence indices; and

subsequent to serially inputting data elements into the compute channel:

aggregating the first running mean and the second running mean to generate a mean associated with the data elements of the vector; and

aggregating the first running variance and the second running variance to generate a variance associated with the data elements of the vector.

6 . The computer-implemented method of claim 5 , further comprising serially inputting reciprocals of sequence indices into the compute channel in parallel with the data elements of the vector.

7 . The computer-implemented method of claim 6 , wherein the reciprocals of sequence indices are inputted from a parameter table that is preloaded with the reciprocals.

8 . The computer-implemented method of claim 5 , wherein the compute channel includes a compute stage that toggles between updating the first running mean and updating the second running mean at each clock cycle.

9 . The computer-implemented method of claim 8 , wherein the compute stage includes:

a first feedback register to store the first running mean; and

a second feedback register to store the second running mean.

10 . The computer-implemented method of claim 5 , wherein the compute channel includes a compute stage that toggles between updating the first running variance and updating the second running variance at each clock cycle.

11 . The computer-implemented method of claim 10 , wherein the compute stage includes:

a first feedback register to store the first running variance; and

a second feedback register to store the second running variance.

12 . The computer-implemented method of claim 5 , further comprising outputting a first count of a number of the data elements of the vector having odd sequence indices, and outputting a second count of a number of the data elements of the vector having even sequence indices.

13 . An integrated circuit device comprising:

a plurality of compute stages coupled in series to form a compute channel,

wherein the compute channel is operable to:

serially receive data elements of a vector; and

generate a first running variance for data elements of the vector having odd sequence indices, and a second running variance for data elements of the vector having even sequence indices from a single pass of the data elements through the compute channel.

14 . The integrated circuit device of claim 13 , wherein one of the compute stages of the compute channel includes a first feedback register to store the first running variance, and a second feedback register to store the second running variance.

15 . The integrated circuit device of claim 13 , wherein the compute channel is operable to generate a first running mean for data elements of the vector having odd sequence indices, and a second running mean for the data elements of the vector having even sequence indices.

16 . The integrated circuit device of claim 15 , wherein one of the compute stages of the compute channel includes a first feedback register to store the first running mean; and a second feedback register to store the second running mean.

17 . The integrated circuit device of claim 13 , wherein the compute channel is operable to serially receive reciprocals of sequence indices in parallel with the data elements.

18 . The integrated circuit device of claim 17 , further comprising a parameter table that is preloaded with the reciprocals of sequence indices.

19 . The integrated circuit device of claim 13 , wherein the compute channel includes eight compute stages.

20 . A non-transitory computer readable medium having stored therein instructions that, when executed by one or more processors, cause the one or more processors to execute a compiler, the compiler performing operations comprising:

receiving a description of a neural network model;

determining that the neural network model includes a tensor normalization operation;

generating a batch statistics machine instruction to compute a first mean and a first variance associated with data elements of the tensor having odd sequence indices, and a second mean and a second variance associated with the data elements of the tensor having even sequence indices;

generating an aggregate machine instruction to compute a mean of the data elements by aggregating the first mean and the second mean, and a variance of the data elements by aggregating the first variance and the second variance; and

generating a tensor scale machine instruction to normalize the tensor based on the computed mean and the computed variance.

Assignments (1)

References Cited (100)