IP Library Granted Patent US 11,080,056
Granted Patent B2
US 11,080,056 · App. 16/670,800 · Granted Aug 3, 2021

Deep vision processor

Inventors: Wajahat Qadeer (Cupertino, CA); Rehan Hameed (Palo Alto, CA)
Assignee: Deep Vision, Inc.
G06F9/3013G06F9/3001G06F9/30036G06F9/30112G06F9/30134G06F17/16G06N3/0454G06N3/063G06N3/08G06T15/005G06T1/20
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,080,056
App. No.
16/670,800
Granted
Aug 3, 2021
Kind
B2
Abstract

Disclosed herein is a processor for deep learning. In one embodiment, the processor comprises: a load and store unit configured to load and store image pixel data and stencil data; a register unit, implementing a banked register file, configured to: load and store a subset of the image pixel data from the load and store unit, and concurrently provide access to image pixel values stored in a register file entry of the banked register file, wherein the subset of the image pixel data comprises the image pixel values stored in the register file entry; and a plurality of arithmetic logic units configured to concurrently perform one or more operations on the image pixel values stored in the register file entry and corresponding stencil data of the stencil data.

Claims (68)

1. A processor comprising:

a banked register file:

comprising a set of register banks;

configured to store image pixel data within the set of register banks; and

representing a two-dimensional register abstraction;

an interconnect unit in communication with the banked register file, the interconnect unit configured to:

concurrently access multiple subsets of the image pixel data from the banked register file, each subset of the image pixel data defining a smaller stencil pattern in a set of smaller stencil patterns, the set of smaller stencil patterns combining to define a larger two-dimensional stencil pattern within the two-dimensional register abstraction; and

concurrently access corresponding stencil data defining the larger two-dimensional stencil pattern; and

a set of arithmetic logic units in communication with the interconnect unit configured to concurrently execute a set of operations on each subset of the image pixel data and the corresponding stencil data, the set of operations representing a larger stencil operation characterized by the larger two-dimensional stencil pattern.

2. The processor of claim 1 , wherein the set of arithmetic logic units is configured to concurrently execute a set of stencil operations characterized by a set of smaller stencil sizes based on the multiple subsets of the image pixel data and the corresponding stencil data, the set of stencil operations representing the larger stencil operation on the image pixel data characterized by a larger stencil size.

3. The processor of claim 1 , wherein the set of arithmetic logic units is configured to concurrently execute a set of convolution operations characterized by a set of smaller kernel sizes based on the multiple subsets of the image pixel data and the corresponding stencil data, the set of convolution operations representing a larger convolution operation on the image pixel data characterized by a larger kernel size.

4. The processor of claim 3 , wherein the set of arithmetic logic units is configured to concurrently execute the set of convolution operations characterized by the set of smaller kernel sizes based on the multiple subsets of the image pixel data and the corresponding stencil data:

the set of convolution operations representing the larger convolution operation on the image pixel data characterized by the larger kernel size; and

the larger kernel size selected from a group of kernel sizes comprising:

a five-by-five kernel size;

a seven-by-seven kernel size;

an eight-by-eight kernel size;

a one-by-five kernel size;

a one-by-seven kernel size;

a one-by-eight kernel size;

a five-by one kernel size;

a seven-by-one kernel size; and

an eight-by-one kernel size.

5. The processor of claim 3 , wherein the set of arithmetic logic units is configured to concurrently execute the set of convolution operations characterized by the set of smaller kernel sizes based on the multiple subsets of the image pixel data and the corresponding stencil data:

the set of convolution operations representing the larger convolution operation on the image pixel data characterized by the larger kernel size; and

the set of smaller kernel sizes selected from a group of kernel sizes comprising:

a three-by-three kernel size;

a four-by-four kernel size;

a one-by-three kernel size;

a three-by-one kernel size;

a one-by-four kernel size; and

a four-by-one kernel size.

6. The processor of claim 1 , wherein each register bank in the set of register banks represents a one-dimensional row in the two-dimensional register abstraction.

7. The processor of claim 1 , wherein each register bank in the set of register banks comprises a set of vector registers.

8. The processor of claim 7 , wherein the set of register banks comprises four register banks.

9. The processor of claim 1 :

further comprising a load and store unit configured to load and store the image pixel data and the stencil data; and

wherein the banked register file is configured to access the load and store unit to store the image pixel data.

10. The processor of Claim 1 :

wherein the set of operations comprises a set of one-dimensional stencil operations; and

wherein the larger stencil operation comprises a two-dimensional stencil operation.

11. A processor comprising:

a shift register unit:

configured to store image pixel data; and

representing a two-dimensional register abstraction; and

a mapping unit in communication with the shift register unit:

configured to concurrently access multiple subsets of the image pixel data from the shift register unit, each subset of the image pixel data defining a smaller stencil pattern in a set of smaller stencil patterns, the set of smaller stencil patterns combining to define a larger two-dimensional stencil pattern within the two-dimensional register abstraction;

configured to concurrently access corresponding stencil data for each subset of the image pixel data in the multiple subsets of the image pixel data; and

comprising a set of arithmetic logic units configured to concurrently execute a set of operations on each subset of the image pixel data and the corresponding stencil data, the set of operations representing a larger stencil operation characterized by the larger two-dimensional stencil pattern.

12. The processor of claim 11 :

wherein the mapping unit comprises the set of arithmetic logic units further configured to generate a set of outputs resulting from the set of operations; and

further comprising a reduction unit, configured to combine the set of outputs to generate a reduced output of the larger stencil operation.

13. The processor of claim 12 , configured to execute a convolution operation by:

executing a set of multiplication operations via the mapping unit to generate the set of outputs; and

executing a summation operation to combine the set of outputs via the reduction unit.

14. The processor of claim 12 , configured to execute a convolution operation by executing a set of subtraction operations via the set of arithmetic logic units to generate the set of outputs.

15. The processor of Claim 12 , configured to execute a convolution operation by:

executing a set of absolute difference operations via the set of arithmetic logic units to generate the set of outputs; and

executing a summation operation to combine the set of outputs via the reduction unit.

16. The processor of claim 12 , configured to execute a convolution operation by executing a set of averaging operations via the set of arithmetic logic units to generate the set of outputs.

17. The processor of claim 12 , further comprising a set of accumulator registers configured to accumulate multiple reduced outputs generated via the reduction unit to execute the larger stencil operation.

18. The processor of claim 12 , further comprising an output register implemented as a shift register and configured to access the set of outputs from the mapping unit.

19. The processor of claim 11 , further comprising a single-input-multiple-data unit configured to execute general purpose computations.

20. A register unit of a processor core:

comprising a set of register banks;

configured to store image pixel data in the set of register banks;

representing a two-dimensional register abstraction; and

configured to provide access to multiple subsets of the image pixel data in the set of register banks, each subset of the image pixel data defining a smaller stencil pattern in a set of smaller stencil patterns, the set of smaller stencil patterns combining to define a larger two-dimensional stencil pattern within the two-dimensional register abstraction.

Assignments (2)
MERGER Recorded Jun 11, 2026
From: KINARA, INC.
To: NXP USA, INC.
Reel/Frame 074926/0187 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 5, 2021
From: QADEER, WAJAHAT; HAMEED, REHAN
To: DEEP VISION, INC.
Reel/Frame 055822/0509 →
Continuity (3)
Continuation 16026480 · Jul 3, 2018
Provisional Application 62528796 · Jul 5, 2017
Related Publication 20200409699A1 · Dec 31, 2020