IP Library › Granted Patent US 11,468,302

Granted Patent B2

US 11,468,302 · App. 16/273,616 · Granted Oct 11, 2022

Efficient convolutional engine

Inventor: Eugene M. Feinberg (San Jose, CA)

Assignee: Recogni Inc.

G06N3/063 G06F5/01 G06F17/15 G06N3/04 G06N5/046

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 11,468,302
App. No.: 16/273,616
Granted: Oct 11, 2022
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

A hardware architecture for implementing a convolutional neural network.

Claims (332)

1. An apparatus, comprising:

a two-dimensional synchronous shift register comprising a three by four array of data storage elements:

[

]

wherein:

data storage element d 1,1 stores data value d 2,1 ,

data storage element d 2,1 stores data value d 3,1 ,

data storage element d 1,2 stores data value d 2,2 ,

data storage element d 2,2 stores data value d 3,2 ,

data storage element d 1,3 stores data value d 2,3 ,

data storage element d 2,3 stores data value d 3,3 ,

data storage element d 1,4 stores data value d 2,4 , and

data storage element d 2,4 stores data value d 3,4 ,

a first convolver unit comprising a first plurality of multipliers, m 1,1 1 , m 1,2 1 , m 1,3 1 , m 2,1 1 , m 2,2 1 , m 2,3 1 , m 3,1 1 , m 3,2 1 , and m 3,3 1 , wherein:

multiplier m 1,1 1 is electrically coupled in a permanent manner to data storage element d 1,1 ,

multiplier m 1,2 1 is electrically coupled in a permanent manner to data storage element d 1,2 ,

multiplier m 1,3 1 is electrically coupled in a permanent manner to data storage element d 1,3 ,

multiplier m 2,1 1 is electrically coupled in a permanent manner to data storage element d 2,1 ,

multiplier m 2,2 1 is electrically coupled in a permanent manner to data storage element d 2,2 ,

multiplier m 2,3 1 is electrically coupled in a permanent manner to data storage element d 2,3 ,

multiplier m 3,1 1 is electrically coupled in a permanent manner to data storage element d 3,1 ,

multiplier m 3,2 1 is electrically coupled in a permanent manner to data storage element d 3,2 , and

multiplier m 3,3 1 is electrically coupled in a permanent manner to data storage element d 3,3 ; and

a second convolver unit comprising a second plurality of multipliers, m 1,1 2 , m 1,2 2 , m 1,3 2 , m 2,1 2 , m 2,2 2 , m 2,3 2 , m 3,1 2 , m 3,2 2 , and m 3,3 2 , wherein:

multiplier m 1,1 2 is electrically coupled in a permanent manner to data storage element d 1,2 ,

multiplier m 1,2 2 is electrically coupled in a permanent manner to data storage element d 1,3 ,

multiplier m 1,3 2 is electrically coupled in a permanent manner to data storage element d 1,4 ,

multiplier m 2,1 2 is electrically coupled in a permanent manner to data storage element d 2,2 ,

multiplier m 2,2 2 is electrically coupled in a permanent manner to data storage element d 2,3 ,

multiplier m 2,3 2 is electrically coupled in a permanent manner to data storage element d 2,4 ,

multiplier m 3,1 2 is electrically coupled in a permanent manner to data storage element d 3,2 ,

multiplier m 3,2 2 is electrically coupled in a permanent manner to data storage element d 3,3 , and

multiplier m 3,3 2 is electrically coupled in a permanent manner to data storage element d 3,4 ,

wherein, at a start of a first clock cycle:

data storage element d 1,1 is configured to store data value x 1,1 ,

data storage element d 1,2 is configured to store data value x 1,2 ,

data storage element d 1,3 is configured to store data value x 1,3 ,

data storage element d 1,4 is configured to store data value x 1,4 ,

data storage element d 2,1 is configured to store data value x 2,1 ,

data storage element d 2,2 is configured to store data value x 2,2 ,

data storage element d 2,3 is configured to store data value x 2,3 ,

data storage element d 2,4 is configured to store data value x 2,4 ,

data storage element d 3,1 is configured to store data value x 3,1 ,

data storage element d 3,2 is configured to store data value x 3,2 ,

data storage element d 3,3 is configured to store data value x 3,3 , and

data storage element d 3,4 is configured to store data value x 3,4 ,

wherein, during the first clock cycle:

multiplier m 1,1 1 is configured to multiply data value x 1,1 with weight w 1 so as to generate a product w 1 x 1,1 ,

multiplier m 1,2 1 is configured to multiply data value x 1,2 with weight w 2 so as to generate a product w 2 x 1,2 ,

multiplier m 1,3 1 is configured to multiply data value x 1,3 with weight w 3 so as to generate a product w 3 x 1,3 ,

multiplier m 2,1 1 is configured to multiply data value x 2,1 with weight w 4 so as to generate a product w 4 x 2,1 ,

multiplier m 2,2 1 is configured to multiply data value x 2,2 with weight w 5 so as to generate a product w 5 x 2,2 ,

multiplier m 2,3 1 is configured to multiply data value x 2,3 with weight w 6 so as to generate a product w 6 x 2,3 ,

multiplier m 3,1 1 is configured to multiply data value x 3,1 with weight w 7 so as to generate a product w 7 x 3,1 ,

multiplier m 3,2 1 is configured to multiply data value x 3,2 with weight w 8 so as to generate a product w 8 x 3,2 ,

multiplier m 3,3 1 is configured to multiply data value x 3,3 with weight w 9 so as to generate a product w 9 x 3,3 ;

multiplier m 1,1 2 is configured to multiply data value x 1,2 with weight w 1 so as to generate a product w 1 x 1,2 ,

multiplier m 1,2 2 is configured to multiply data value x 1,3 with weight w 2 so as to generate-a product w 2 x 1,3 ,

multiplier m 1,3 2 is configured to multiply data value x 1,4 with weight w 3 so as to generate a product w 3 x 1,4 ,

multiplier m 2,1 2 is configured to multiply data value x 2,2 with weight w 4 so as to generate a product w 4 x 2,2 ,

multiplier m 2,2 2 is configured to multiply data value x 2,3 with weight w 5 so as to generate a product w 5 x 2,3 ,

multiplier m 2,3 2 is configured to multiply data value x 2,4 with weight w 6 so as to generate a product w 6 x 2,4 ,

multiplier m 3,1 2 is configured to multiply data value x 3,2 with weight w 7 so as to generate a product w 7 x 3,2 ,

multiplier m 3,2 2 is configured to multiply data value x 3,3 with weight w 8 so as to generate a product w 8 x 3,3 , and

multiplier m 3,3 2 is configured to multiply data value x 3,4 with weight w 9 so as to generate a product w 9 x 3,4 ,

wherein, prior to a second clock cycle that follows the first clock cycle:

the first convolver unit is configured to generate a first sum of terms including at least the product w 1 x 1,1 , the product w 2 x 1,2 , the product w 3 x 1,3 , the product w 4 x 2,1 , the product w 5 x 2,2 , the product w 6 x 2,3 , the product w 7 x 3,1 , the product w 8 x 3,2 , and the product w 9 x 3,3 , and

the second convolver unit is configured to generate a second sum of terms including at least the product w 1 x 1,2 , the product w 2 x 1,3 , the product w 3 x 1,4 , the product w 4 x 2,2 , the product w 5 x 2,3 , the product w 6 x 2,4 , the product w 7 x 3,2 , the product w 8 x 3,3 , and the product w 9 x 3,4 , and

wherein, at a beginning of the second clock cycle that follows the first clock cycle:

a first accumulator is configured to store the first sum of terms, and

a second accumulator is configured to store the second sum of terms.

2. The apparatus of claim 1 , wherein the first sum of terms additionally includes b 1 , wherein b 1 is a bias value.

3. The apparatus of claim 1 , wherein the second sum of terms additionally includes b 1 , wherein b 1 is a bias value.

4. The apparatus of claim 1 , further comprising:

a third convolver unit comprising a third plurality of multipliers, m 1,1 3 , m 1,2 3 , m 1,3 3 , m 2,1 3 , m 2,2 3 , m 2,3 3 , m 3,1 3 , m 3,2 3 , and m 3,3 3 , wherein:

multiplier m 1,1 3 is electrically coupled in a permanent manner to data storage element d 1,1 ,

multiplier m 1,2 3 is electrically coupled in a permanent manner to data storage element d 1,2 ,

multiplier m 1,3 3 is electrically coupled in a permanent manner to data storage element d 1,3 ,

multiplier m 2,1 3 is electrically coupled in a permanent manner to data storage element d 2,1 ,

multiplier m 2,2 3 is electrically coupled in a permanent manner to data storage element d 2,2 ,

multiplier m 2,3 3 is electrically coupled in a permanent manner to data storage element d 2,3 ,

multiplier m 3,1 3 is electrically coupled in a permanent manner to data storage element d 3,1 ,

multiplier m 3,2 3 is electrically coupled in a permanent manner to data storage element d 3,2 , and

multiplier m 3,3 3 is electrically coupled in a permanent manner to data storage element d 3,3 ,

a fourth convolver unit comprising a fourth plurality of multipliers, m 1,1 4 , m 1,2 4 , m 1,3 4 , m 2,1 4 , m 2,2 4 , m 2,3 4 , m 3,1 4 , m 3,2 4 , and m 3,3 4 , wherein:

multiplier m 1,1 4 is electrically coupled in a permanent manner to data storage element d 1,2 ,

multiplier m 1,2 4 is electrically coupled in a permanent manner to data storage element d 1,3 ,

multiplier m 1,3 4 is electrically coupled in a permanent manner to data storage element d 1,4 ,

multiplier m 2,1 4 is electrically coupled in a permanent manner to data storage element d 2,2 ,

multiplier m 2,2 4 is electrically coupled in a permanent manner to data storage element d 2,3 ,

multiplier m 2,3 4 is electrically coupled in a permanent manner to data storage element d 2,4 ,

multiplier m 3,1 4 is electrically coupled in a permanent manner to data storage element d 3,2 ,

multiplier m 3,2 4 is electrically coupled in a permanent manner to data storage element d 3,3 , and

multiplier m 3,3 4 is electrically coupled in a permanent manner to data storage element d 3,4 .

5. An apparatus, comprising:

a two-dimensional synchronous shift register comprising a four by four array of data storage elements,

[

]

wherein:

data storage element d 1,1 is electrically coupled to data storage element d 2,1 ,

data storage element d 2,1 is electrically coupled to data storage element d 3,1 ,

data storage element d 2,1 is electrically coupled to data storage element d 4,1 ,

data storage element d 1,2 is electrically coupled to data storage element d 2,2 ,

data storage element d 2,2 is electrically coupled to data storage element d 3,2 ,

data storage element d 3,2 is electrically coupled to data storage element d 4,2 ,

data storage element d 1,3 is electrically coupled to data storage element d 2,3 ,

data storage element d 2,3 is electrically coupled to data storage element d 3,3 ,

data storage element d 3,3 is electrically coupled to data storage element d 4,3 ,

data storage element d 1,4 is electrically coupled to data storage element d 2,4 ,

data storage element d 2,4 is electrically coupled to data storage element d 3,4 , and

data storage element d 3,4 is electrically coupled to data storage element d 4,4 ,

a first convolver unit comprising a first plurality of multipliers,