IP Library Granted Patent US 11,436,014
Granted Patent B2
US 11,436,014 · App. 17/356,372 · Granted Sep 6, 2022

Deep vision processor

Inventors: Wajahat Qadeer (Los Altos, CA); Rehan Hameed (Los Altos, CA)
Assignee: Deep Vision, Inc.
G06F9/3013G06F9/3001G06F9/30036G06F9/30112G06F9/30134G06F17/16G06N3/0454G06N3/063G06N3/08G06T15/005G06T1/20
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,436,014
App. No.
17/356,372
Granted
Sep 6, 2022
Kind
B2
Abstract

Disclosed herein is a processor for deep learning. In one embodiment, the processor comprises: a load and store unit configured to load and store image pixel data and stencil data; a register unit, implementing a banked register file, configured to: load and store a subset of the image pixel data from the load and store unit, and concurrently provide access to image pixel values stored in a register file entry of the banked register file, wherein the subset of the image pixel data comprises the image pixel values stored in the register file entry; and a plurality of arithmetic logic units configured to concurrently perform one or more operations on the image pixel values stored in the register file entry and corresponding stencil data of the stencil data.

Claims (75)

1. A processor comprising:

a banked register file:

representing a two-dimensional register abstraction;

comprising a set of register banks; and

configured to store image pixel data within the set of register files;

an interconnect unit in communication with the banked register file, the interconnect unit configured to concurrently access:

multiple subsets of the image pixel data from the banked register file; and

corresponding stencil data for the multiple subsets of image pixel data; and

a set of arithmetic logic units in communication with the interconnect unit configured to concurrently execute a set of smaller stencil operations on each subset of the image pixel data and the corresponding stencil data, the set of smaller stencil operations representing a larger stencil operation characterized by greater than two dimensions.

2. The processor of claim 1 , wherein the set of arithmetic logic units is configured to concurrently execute a set of smaller convolution operations on each subset of the image pixel data and the corresponding stencil data, the set of smaller convolution operations:

characterized by a set of smaller kernel sizes; and

representing a larger convolution operation on the image pixel data characterized by a larger kernel size.

3. The processor of claim 2 , wherein the set of arithmetic logic units is configured to concurrently execute the set of smaller convolution operations on each subset of the image pixel data and the corresponding stencil data, the set of smaller convolution operations:

representing the larger convolution operation on the image pixel data characterized by the larger kernel size; and

characterized by the set of smaller kernel sizes selected from a group of kernel sizes comprising:

a three-by-three kernel size;

a four-by-four kernel size;

a one-by-three kernel size;

a three-by-one kernel size;

a one-by-four kernel size; and

a four-by-one kernel size.

4. The processor of claim 2 , wherein the set of arithmetic logic units is configured to concurrently execute the set of smaller convolution operations on each subset of the image pixel data and the corresponding stencil data, the set of smaller convolution operations representing the larger convolution operation on the image pixel data characterized by the larger kernel size selected from a group of kernel sizes comprising:

a five-by-five kernel size;

a seven-by-seven kernel size;

an eight-by-eight kernel size;

a one-by-five kernel size;

a one-by-seven kernel size;

a one-by-eight kernel size;

a five-by one kernel size;

a seven-by-one kernel size; and

an eight-by-one kernel size.

5. The processor of claim 1 , wherein each register bank in the set of register banks represents a one-dimensional row in the two-dimensional register abstraction.

6. The processor of claim 1 , wherein each register bank in the set of register banks comprises a set of vector registers.

7. The processor of claim 6 , wherein the set of register banks comprises four register banks.

8. The processor of claim 1 :

wherein the set of smaller operations comprises a set of one-dimensional stencil operations; and

wherein the larger stencil operation comprises a two-dimensional stencil operation.

9. A processor comprising:

a shift register unit:

configured to store image pixel data; and

representing a two-dimensional register abstraction;

a mapping unit in communication with the shift register unit:

configured to concurrently access multiple subsets of the image pixel data from the shift register unit based on a two-dimensional stencil pattern within the two-dimensional register abstraction;

configured to concurrently access corresponding stencil data for each subset of the image pixel data in the multiple subsets of the image pixel data; and

comprising a set of arithmetic logic units configured to concurrently execute a set of operations on each subset of the image pixel data and the corresponding stencil data, the set of operations representing a stencil operation characterized by the two-dimensional stencil pattern.

10. The processor of claim 9 , wherein the mapping unit is configured to concurrently access the multiple subsets of the image pixel data, each subset of the image pixel data defining a smaller stencil pattern in a set of smaller stencil patterns, the set of smaller stencil patterns combining to define the two-dimensional stencil pattern.

11. The processor of claim 9 :

wherein the set of arithmetic logic units is further configured to generate a set of outputs resulting from the set of operations; and

further comprising a reduction unit, configured to combine the set of outputs to generate a reduced output of the stencil operation.

12. The processor of claim 11 :

wherein the set of arithmetic logic units is configured to concurrently execute a set of multiplication operations to generate the set of outputs; and

wherein the reduction unit is configured to execute a summation operation on the set of outputs to generate the reduced output.

13. The processor of claim 11 , wherein the set of arithmetic logic units are configured to concurrently execute a set of subtraction operations to generate the set of outputs.

14. The processor of claim 11 :

wherein the set of arithmetic logic units are configured to concurrently execute a set of absolute difference operations to generate the set of outputs; and

wherein the reduction unit is configured to execute a summation operation on the set of outputs to generate the reduced output.

15. The processor of claim 11 , wherein the set of arithmetic logic units are configured to concurrently execute a set of averaging operations to generate the set of outputs.

16. The processor of claim 11 , further comprising a set of accumulator registers configured to accumulate a set of reduced outputs comprising the reduced output to execute the stencil operation.

17. The processor of claim 11 , further comprising an output register implemented as a shift register and configured to access the set of outputs from the mapping unit.

18. The processor of claim 9 , further comprising a single-input-multiple-data unit configured to execute general purpose computations.

19. A register unit of a processor core:

comprising a set of register banks;

configured to store image pixel data within the set of register banks;

representing a two-dimensional register abstraction; and

configured to provide access to multiple subsets of the image pixel data in the set of register banks, the multiple subsets of the image pixel data defining a two-dimensional stencil pattern within the two-dimensional register abstraction.

20. A processor comprising:

a banked register file:

comprising a set of register banks; and

configured to store image pixel data within the set of register files;

an interconnect unit in communication with the banked register file, the interconnect unit configured to concurrently access:

multiple subsets of the image pixel data from the banked register file; and

corresponding stencil data for the multiple subsets of image pixel data; and

a set of arithmetic logic units in communication with the interconnect unit configured to concurrently execute a set of smaller stencil operations on each subset of the image pixel data and the corresponding stencil data, the set of smaller stencil operations representing a larger stencil operation characterized by greater than two-dimensions;

wherein the set of smaller operations comprises a set of one-dimensional stencil operations; and

wherein the larger stencil operation comprises a two-dimensional stencil operation.

Assignments (2)
MERGER Recorded Jun 11, 2026
From: KINARA, INC.
To: NXP USA, INC.
Reel/Frame 074926/0187 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 23, 2021
From: QADEER, WAJAHAT; HAMEED, REHAN
To: DEEP VISION, INC.
Reel/Frame 056644/0925 →
Continuity (4)
Continuation 16670800 · Oct 31, 2019
Continuation 16026480 · Jul 3, 2018
Provisional Application 62528796 · Jul 5, 2017
Related Publication 20210326133A1 · Oct 21, 2021
Cited By (1)
US 12,619,515