IP Library › Granted Patent US 11,037,026
Granted Patent B2
US 11,037,026 · App. 16/749,626 · Granted Jun 15, 2021

Fully parallel, low complexity approach to solving computer vision problems

Inventors: Sean Ryan Fanello (San Francisco, CA); Julien Pascal Christophe Valentin (San Francisco, CA); Adarsh Prakash Murthy Kowdle (San Francisco, CA); Christoph Rhemann (San Francisco, CA); Vladimir Tankovich (Renton, WA); Philip L. Davidson (Boston, MA); Shahram Izadi (Tiburon, CA)
Assignee: Google LLC
G06K9/6256G06K9/6202G06K9/6276G06K9/6298
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,037,026
App. No.
16/749,626
Granted
Jun 15, 2021
Kind
B2
Abstract

Values of pixels in an image are mapped to a binary space using a first function that preserves characteristics of values of the pixels. Labels are iteratively assigned to the pixels in the image in parallel based on a second function. The label assigned to each pixel is determined based on values of a set of nearest-neighbor pixels. The first function is trained to map values of pixels in a set of training images to the binary space and the second function is trained to assign labels to the pixels in the set of training images. Considering only the nearest neighbors in the inference scheme results in a computational complexity that is independent of the size of the solution space and produces sufficient approximations of the true distribution when the solution for each pixel is most likely found in a small subset of the set of potential solutions.

Claims (43)

1. A method comprising:

mapping values of pixels in an image to a binary space using a binary function; and

iteratively assigning labels to the pixels in the image in parallel based on a unary function, wherein the label assigned to each pixel is determined based on values of a set of nearest-neighbor pixels.

2. The method of claim 1 , further comprising:

training the binary function to map values of pixels in a set of training images to the binary space; and

training the binary function to assign labels to the pixels in the set of training images.

3. The method of claim 2 , wherein training the binary and unary functions comprises reconstructing estimated values of pixels in the set of training images from the values of the pixels in the binary space and iteratively updating the binary and unary functions based on a comparison of actual values of the pixels in the set of training images and the estimated values.

4. The method of claim 3 , wherein training the binary and unary functions comprises learning a set of hyperplanes and a task function that minimize a loss function subject to an anti-sparsity constraint.

5. The method of claim 4 , wherein training the binary and unary functions comprises training first and second regularizer functions using the set of training images, wherein the first and second regularizer functions encourage corresponding structures for the set of hyperplanes and the task function.

6. The method of claim 5 , wherein learning the set of hyperplanes comprises iteratively updating values of the set of hyperplanes based on the first regularizer, wherein the first regularizer is chosen to induce sparse solutions in the set of hyperplanes.

7. The method of claim 6 , wherein learning the task function comprises iteratively updating values of the task function using a gradient descent technique.

8. The method of claim 1 , wherein iteratively assigning the labels to the pixels in parallel comprises estimating distributions of labels of the pixels as independent marginal functions.

9. The method of claim 8 , wherein estimating the distributions of the labels of the pixels as independent marginal functions comprises estimating the distributions of the labels of the pixels as Dirac δ functions.

10. The method of claim 9 , wherein iteratively assigning the labels to the pixels in parallel comprises assigning, during an iteration, the labels of the pixels to a maximal value of the marginal functions of the nearest neighbor pixels.

11. The method of claim 8 , wherein iteratively assigning the labels to the pixels in parallel comprises iteratively assigning the labels to the pixels in parallel using a coordinate ascent procedure until convergence.

12. An apparatus comprising:

a processor configured to map values of pixels in an image to a binary space using a binary function and iteratively assign labels to the pixels in the image in parallel based on a unary function, wherein the label assigned to each pixel is determined based on values of a set of nearest-neighbor pixels; and

a memory to store the labels of the pixels.

13. The apparatus of claim 12 , wherein the binary function is trained to map values of pixels in a set of training images to the binary space and the unary function is trained to assign labels to the pixels in the set of training images.

14. The apparatus of claim 13 , wherein training the binary and unary functions comprises reconstructing estimated values of pixels in the set of training images from the values of the pixels in the binary space and iteratively updating the binary and unary functions based on a comparison of actual values of the pixels in the set of training images and the estimated values.

15. The apparatus of claim 14 , wherein training the binary and unary functions comprises learning a set of hyperplanes and a task function that minimize a loss function subject to an anti-sparsity constraint.

16. The apparatus of claim 15 , wherein training the binary and unary functions comprises training first and second regularizer functions using the set of training images, wherein the first and second regularizer functions encourage corresponding structures for the set of hyperplanes and the task function.

17. The apparatus of claim 16 , wherein learning the set of hyperplanes comprises iteratively updating values of the set of hyperplanes based on the first regularizer, wherein the first regularizer is chosen to induce sparse solutions in the set of hyperplanes.

18. The apparatus of claim 17 , wherein learning the task function comprises iteratively updating values of the task function using a gradient descent technique.

19. The apparatus of claim 12 , wherein the processor is configured to estimate distributions of labels of the pixels as independent marginal functions.

20. The apparatus of claim 19 , wherein the processor is configured to estimate the distributions of the labels of the pixels as Dirac δ functions.

21. The apparatus of claim 20 , wherein the processor is configured to assign, during an iteration, the labels of the pixels to a maximal value of the marginal functions of the nearest neighbor pixels.

22. The apparatus of claim 19 , wherein the processor is configured to iteratively assign the labels to the pixels in parallel using a coordinate ascent procedure until convergence.

23. A non-transitory computer readable medium embodying a set of executable instructions, the set of executable instructions to manipulate at least one processor to:

map values of pixels in an image to a binary space using a binary function; and

iteratively assign labels to the pixels in the image in parallel based on a unary function, wherein the label assigned to each pixel is determined based on values of a set of nearest-neighbor pixels.

24. The non-transitory computer readable medium of claim 23 , wherein the set of executable instructions is to manipulate the at least one processor to:

train the binary function to map values of pixels in a set of training images to the binary space; and

train the binary function to assign labels to the pixels in the set of training images.

25. The non-transitory computer readable medium of claim 24 , wherein the set of executable instructions is to manipulate the at least one processor to reconstruct estimated values of pixels in the set of training images from the values of the pixels in the binary space and iteratively updating the binary and unary functions based on a comparison of actual values of the pixels in the set of training images and the estimated values.

26. The non-transitory computer readable medium of claim 25 , wherein the set of executable instructions is to manipulate the at least one processor to learn a set of hyperplanes and a task function that minimize a loss function subject to an anti-sparsity constraint.

27. The non-transitory computer readable medium of claim 26 , wherein the set of executable instructions is to manipulate the at least one processor to train first and second regularizer functions using the set of training images, wherein the first and second regularizer functions encourage corresponding structures for the set of hyperplanes and the task function.

28. The non-transitory computer readable medium of claim 27 , wherein the set of executable instructions is to manipulate the at least one processor to iteratively update values of the set of hyperplanes based on the first regularizer, wherein the first regularizer is chosen to induce sparse solutions in the set of hyperplanes.

29. The non-transitory computer readable medium of claim 28 , wherein the set of executable instructions is to manipulate the at least one processor to iteratively update values of the task function using a gradient descent technique.

30. The non-transitory computer readable medium of claim 23 , wherein the set of executable instructions is to manipulate the at least one processor to estimate distributions of labels of the pixels as independent marginal functions.

31. The non-transitory computer readable medium of claim 30 , wherein the set of executable instructions is to manipulate the at least one processor to estimate the distributions of the labels of the pixels as Dirac δ functions.

32. The non-transitory computer readable medium of claim 31 , wherein the set of executable instructions is to manipulate the at least one processor to assign, during an iteration, the labels of the pixels to a maximal value of the marginal functions of the nearest neighbor pixels.

33. The non-transitory computer readable medium of claim 30 , wherein the set of executable instructions is to manipulate the at least one processor to iteratively assign the labels to the pixels in parallel using a coordinate ascent procedure until convergence.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 23, 2020
From: FANELLO, SEAN RYAN; VALENTIN, JULIEN PASCAL CHRISTOPHE; KOWDLE, ADARSH PRAKASH MURTHY; RHEMANN, CHRISTOPH; TANKOVICH, VLADIMIR; DAVIDSON, PHILIP L.; IZADI, SHAHRAM
To: PERCEPTIVEIO, INC.
Reel/Frame 051593/0089 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 23, 2020
From: PERCEPTIVEIO, INC.
To: GOOGLE LLC
Reel/Frame 051593/0107 →
Continuity (3)
Continuation 15925141 · Mar 19, 2018
Provisional Application 62473280 · Mar 17, 2017
Related Publication 20200160109A1 · May 21, 2020