IP Library › Granted Patent US 10,728,489
Granted Patent B2
US 10,728,489 · App. 16/109,708 · Granted Jul 28, 2020

Low power framework for controlling image sensor mode in a mobile image capture device

Inventors: Aaron Michael Donsbach (Seattle, WA); Benjamin Vanik (Seattle, WA); Jon Gabriel Clapper (Seattle, WA); Alison Lentz (Seattle, WA); Joshua Denali Lovejoy (Seattle, WA); Robert Douglas Fritz, III (Seattle, WA); Krzysztof Duleba (Seattle, WA); Li Zhang (Seattle, WA); Juston Payne (San Mateo, CA); Emily Anne Fortuna (Seattle, WA); Iwona Bialynicka-Birula (Redmond, WA); Blaise Aguera-Arcas (Seattle, WA); Daniel Ramage (Seattle, WA); Benjamin James McMahan (Sunnyvale, CA); Oliver Fritz Lange (Seattle, WA); Jess Holbrook (Seattle, WA)
Assignee: Google LLC
H04N5/77G06K9/00221G06K9/00664G06K9/4628G06K9/6201G06K9/6274G06K9/66G06N3/0454G06N3/08H04N5/23219H04N5/23222H04N5/23241H04N5/23245H04N9/8042H04N9/8205H04N19/132H04N19/136H04N19/423H04N19/426H04N19/436H04N19/46H04N19/85G06N3/0445G06N5/003
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,728,489
App. No.
16/109,708
Granted
Jul 28, 2020
Kind
B2
Abstract

The present disclosure provides an image capture, curation, and editing system that includes a resource-efficient mobile image capture device that continuously captures images. In particular, the present disclosure provides low power frameworks for controlling image sensor mode in a mobile image capture device. On example low power frame work includes a scene analyzer that analyzes a scene depicted by a first image and, based at least in part on such analysis, causes an image sensor control signal to be provided to an image sensor to adjust at least one of the frame rate and the resolution of the image sensor.

Claims (34)

1. A computer-implemented method, comprising:

obtaining, by one or more server computing devices, data descriptive of a pre-trained artificial neural network, the pre-trained artificial neural network having been previously trained based at least in part on a set of first training images;

receiving, by the one or more server computing devices, a set of user training images provided by a user, the set of user training images including at least one image not included in the set of first training images, wherein at least one of the set of user training images has been edited by the user resulting in photographic re-composition;

re-training, by the one or more server computing devices, the pre-trained artificial neural network based at least in part on the set of user training images to form a re-trained artificial neural network; and

transmitting, by the one or more server computing devices, the re-trained artificial neural network to a user computing device associated with the user for implementation at the user computing device.

2. The computer-implemented method of claim 1 , wherein at least one of the set of user training images has been hand-labeled by the user.

3. The computer-implemented method of claim 1 , wherein the user computing device comprises an image capture device associated with the user.

4. The computer-implemented method of claim 3 , wherein the set of user training images comprise images previously captured by the image capture device.

5. The computer-implemented method of claim 1 , wherein the pre-trained artificial neural network and the re-trained artificial neural network comprise convolutional artificial neural networks.

6. The computer-implemented method of claim 1 , wherein the pre-trained artificial neural network and the re-trained artificial neural network comprise image classification artificial neural networks.

7. The computer-implemented method of claim 1 , wherein the pre-trained artificial neural network and the re-trained artificial neural network comprise face detection artificial neural networks.

8. The computer-implemented method of claim 1 , wherein the pre-trained artificial neural network and the re-trained artificial neural network comprise image content artificial neural networks.

9. A training computing system for training personalized artificial neural networks based on user-submitted training data, the training computing system comprising one or more server computing devices configured to perform operations, the operations comprising:

obtaining data descriptive of a pre-trained artificial neural network, the pre-trained artificial neural network having been previously trained based at least in part on a set of first training images;

receiving a set of user training images selected by a user, the set of user training images including at least one image not included in the set of first training images, wherein at least one of the set of user training images has been edited by the user resulting in photographic re-composition;

re-training the pre-trained artificial neural network based at least in part on the set of user training images to form a re-trained artificial neural network; and

transmitting the re-trained artificial neural network to a user computing device associated with the user for implementation at the user computing device.

10. The training computing system of claim 9 , wherein at least one of the set of user training images has been hand-labeled by the user.

11. The training computing system of claim 9 , wherein the user computing device comprises an image capture device associated with the user.

12. The training computing system of claim 11 , wherein the set of user training images comprise images previously captured by the image capture device.

13. The training computing system of claim 9 , wherein the pre-trained artificial neural network and the re-trained artificial neural network comprise convolutional artificial neural networks.

14. The training computing system of claim 9 , wherein the pre-trained artificial neural network and the re-trained artificial neural network comprise image classification artificial neural networks.

15. The training computing system of claim 9 , wherein the pre-trained artificial neural network and the re-trained artificial neural network comprise face detection artificial neural networks.

16. The training computing system of claim 9 , wherein the pre-trained artificial neural network and the re-trained artificial neural network comprise image content artificial neural networks.

17. A user computing system comprising:

an image capture system comprising an artificial neural network, the image capture system configured to capture images based at least in part on an output of the artificial neural network; and

one or more computing devices configured to:

obtain a set of images captured by the image capture device;

receive user input that edits one or more of the set of images captured by the image capture device to form a set of edited images having photographic re-composition based on the user input; and

after receiving the user input, provide the set of edited images to a training computing system; and

wherein the image capture system is configured to:

receive and store a re-trained version of the artificial neural network that has been re-trained by the training computing system based on the set of edited images provided to the training computing system by the one or more computing devices; and

capture images based at least in part on an output of the re-trained version of the artificial neural network.

18. The user computing system of claim 17 , wherein the image capture system comprises a mobile image capture device.

Assignments (3)
CORRECTIVE ASSIGNMENT TO CORRECT THE SPELLING OF THE FIFTH INVENTOR'S NAME PREVIOUSLY RECORDED ON REEL 047493 FRAME 0453. ASSIGNOR(S) HEREBY CONFIRMS THE CORRECTIVE ASSIGNMENT. Recorded Nov 15, 2018
From: DONSBACH, AARON MICHAEL; VANIK, BENJAMIN; CLAPPER, JON GABRIEL; LENTZ, ALISON; LOVEJOY, JOSH DENALI; FRITZ, ROBERT DOUGLAS, III; DULEBA, KRZYSZTOF; ZHANG, LI; PAYNE, JUSTON; FORTUNA, EMILY ANNE; BIALYNICKA-BIRULA, IWONA; AGUERA-ARCAS, BLAISE; RAMAGE, DANIEL; MCMAHAN, BENJAMIN JAMES; LANGE, OLIVER FRITZ; HOLBROOK, JESS
To: GOOGLE LLC
Reel/Frame 047577/0306 →
CORRECTIVE ASSIGNMENT TO CORRECT THE SPELLING OF THE FOURTEENTH ASSIGNOR'S NAME PREVIOUSLY RECORDED ON REEL 046669 FRAME 0401. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Recorded Aug 28, 2018
From: DONSBACH, AARON MICHAEL; VANIK, BENJAMIN; CLAPPER, JON GABRIEL; LENTZ, ALISON; LOVEJOY, JOHN DENALI; FRITZ, ROBERT DOUGLAS, III; DULEBA, KRZYSZTOF; ZHANG, LI; PAYNE, JUSTON; FORTUNA, EMILY ANNE; BIALYNICKA-BIRULA, IWONA; AGUERA-ARCAS, BLAISE; RAMAGE, DANIEL; MCMAHAN, BENJAMIN JAMES; LANGE, OLIVER FRITZ; HOLBROOK, JESS
To: GOOGLE LLC
Reel/Frame 047493/0453 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 22, 2018
From: DONSBACH, AARON MICHAEL; VANIK, BENJAMIN; CLAPPER, JON GABRIEL; LENTZ, ALISON; LOVEJOY, JOSH DENALI; FRITZ, ROBERT DOUGLAS, III; DULEBA, KRZYSZTOF; ZHANG, LI; PAYNE, JUSTON; FORTUNA, EMILY ANNE; BIALYNICKA-BIRULA, IWONA; AGUERA-ARCAS, BLAISE; RAMAGE, DANIEL; MCMAHAN, BANJAMIN JAMES; LANGE, OLIVER FRITZ; HOLBROOK, JESS
To: GOOGLE LLC
Reel/Frame 046669/0401 →
Continuity (2)
Continuation 14984869 · Dec 30, 2015
Related Publication 20180367752A1 · Dec 20, 2018
Cited By (1)
US 12,586,217