IP Library Granted Patent US 10,922,833
Granted Patent B2
US 10,922,833 · App. 15/980,322 · Granted Feb 16, 2021

Image processing

Inventors: Daren Croxford (Swaffham Prior, GB); David Hennah Mansell (Norwich, GB)
Assignees: Apical Ltd.; Arm Limited
G06T7/60G06T1/20G06T1/60
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,922,833
App. No.
15/980,322
Granted
Feb 16, 2021
Kind
B2
Abstract

A method of processing image data representative of an image using a multi-stage system comprising a first neural network (NN) for identifying a first image characteristic and a second NN for identifying a second image characteristic. The method comprises processing the image data using t a first at least one layer of the first NN to generate feature data representative of at least one feature of the image and processing the feature data using a second at least one layer of the first NN to generate first image characteristic data indicative of whether the image includes the first image characteristic. The feature data is transferred from the first NN to the second NN. The feature data is processed using the second NN to generate second image characteristic data representative of whether the image includes the second image characteristic.

Claims (49)

1. A method of processing image data representative of an image using a multi-stage system comprising a first neural network for identifying a first image characteristic and a second neural network for identifying a second image characteristic different from the first image characteristic, the method comprising:

processing the image data using the first neural network, the processing the image data using the first neural network comprising:

i) processing the image data using a first at least one layer of the first neural network to generate feature data representative of at least one feature of the image, wherein the feature data represents a feature map; and

ii) processing the feature data using a second at least one layer of the first neural network to generate first image characteristic data indicative of whether the image includes the first image characteristic;

transferring the feature data from the first neural network to the second neural network; and

processing the feature data using the second neural network, without processing the image data using the second neural network, to generate second image characteristic data indicative of whether the image includes the second image characteristic.

2. The method according to claim 1 , wherein:

the first image characteristic data is indicative of whether the image includes an object of a predetermined class; and

the second image characteristic data is indicative of whether the image includes a particular instance of the object of the predetermined class.

3. The method according to claim 1 , wherein the second neural network comprises:

a third at least one layer which corresponds, respectively, to the first at least one layer of the first neural network; and

a fourth at least one layer, wherein the method comprises bypassing the third at least one layer of the second neural network by transferring the feature data from the first neural network to the fourth at least one layer of the second neural network.

4. The method according to claim 3 , wherein the second neural network is a second convolutional neural network (CNN), each of the third at least one layer is a convolutional layer, respectively, and the fourth at least one layer comprises a fully connected layer.

5. The method according to claim 1 , wherein the first neural network is a convolutional neural network (CNN), each of the first at least one layer is a convolutional layer, respectively, and the second at least one layer comprises a fully connected layer.

6. The method according to claim 1 , wherein the second neural network has a greater complexity than the first neural network.

7. The method according to claim 6 , wherein the second neural network comprises more layers than the first neural network.

8. The method according to claim 1 , wherein transferring the feature data from the first neural network to the second neural network comprises:

writing the feature data to storage after processing the image data using the first at least one layer; and

retrieving the feature data from the storage before processing the feature data using the second neural network.

9. The method according to claim 1 , comprising processing the feature data using the second neural network in response to the first neural network determining that the image includes the first image characteristic.

10. The method according to claim 1 , comprising:

writing the feature data to storage; and

subsequently, determining whether the image includes the first image characteristic.

11. The method according to claim 10 , comprising:

determining that the image does not include the first image characteristic; and

in response, discarding the feature data from the storage.

12. The method according to claim 1 , comprising:

receiving input image data representative of an input image comprising the image; and

processing the input image data to segment the input image into a plurality of blocks, a block of the plurality of blocks corresponding to the image.

13. The method according to claim 12 , comprising writing the feature data to storage without writing further feature data to the storage, the further feature data corresponding to a further block of the plurality of blocks which the first neural network has determined does not include the first image characteristic.

14. The method according to claim 1 , wherein the first at least one layer is an initial at least one convolutional layer of the first neural network.

15. An image processing system comprising:

at least one processor;

storage accessible by the at least one processor, the storage configured to store image data representative of an image; and

at least one neural network accelerator configured to implement a first neural network for identifying a first image characteristic and a second neural network for identifying a second image characteristic different from the first image characteristic, wherein the at least one neural network accelerator is configured to:

process the image data using a first at least one layer of the first neural network to generate feature data representative of at least one feature of the image, wherein the feature data represents a feature map;

process the feature data using a second at least one layer of the first neural network to generate first image characteristic data indicative of whether the image includes the first image characteristic;

receive the feature data from the first neural network; and

process the feature data using the second neural network, without processing the image data using the second neural network, to generate second image characteristic data representative of whether the image includes the second image characteristic.

16. The image processing system according to claim 15 , wherein the at least one neural network accelerator is configured to:

write the feature data to the storage; and

read the feature data from the storage.

17. The image processing system according to claim 16 , wherein the storage is a main memory.

18. The image processing system according to claim 15 , wherein the at least one neural network accelerator is a neural network accelerator configured to implement both the first neural network and the second neural network, the neural network accelerator comprising local storage and being configured to:

write the feature data to the local storage; and

read the feature data from the local storage.

19. The image processing system according to claim 15 , wherein the at least one neural network accelerator comprises:

a first neural network accelerator configured to implement the first neural network; and

a second neural network accelerator, different from the first neural network accelerator, configured to implement the second neural network.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 23, 2022
From: APICAL LIMITED
To: ARM LIMITED
Reel/Frame 061300/0471 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 15, 2018
From: CROXFORD, DAREN
To: APICAL LTD
Reel/Frame 045811/0195 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 15, 2018
From: MANSELL, DAVID HENNAH
To: ARM LIMITED
Reel/Frame 045811/0210 →
Continuity (1)
Related Publication 20190355142A1 · Nov 21, 2019
Cited By (2)
US 12,579,784 US 12,666,142