Granted Patent B2

US 10,650,573 · App. 16/457,780 · Granted May 12, 2020

Inventors: James Andrew Youngquist (Seattle, WA); David Julio Colmenares (Seattle, WA); Adam Gabriel Jones (Seattle, WA)

Assignee: PROPRIO, INC.

G06T15/20G06T7/20G06T7/55G06T7/70G06T15/06G06T15/40G06T2210/21

Abstract

Claims (110)

1. A method for synthesizing an image corresponding to a virtual

perspective of a scene, the method comprising:

obtaining a virtual pose of a virtual perspective, a plurality of poses of physical imagers of a camera array, and a geometry of the scene;

identifying, for a target virtual pixel location of the image, a world point in the scene comprising an intersection point in the scene of a virtual ray extending to the target virtual pixel location;

detecting, based on the world point, the plurality of poses of the physical imagers, and the geometry of the scene, candidate physical imager pixels corresponding to the world point;

determining a plurality of error surfaces corresponding to different depth errors in the geometry of the scene;

determining error surface points at intersections of the error surfaces by rays between the world point and the physical imager pixels;

projecting rays from the error surface points to the virtual imager to determine virtual error pixel locations for each of the candidate physical imager pixels;

determining virtual pixel error sensitivities representing a rate of change of errors between the virtual error pixel locations and the target pixel location in response to a change in depth error of the geometry of the scene;

determining pixel weights associated with each of the candidate physical imager pixels as a function of the virtual pixel error sensitivities; and

determining a value for the target virtual pixel based on a weighted combination of pixel values of the candidate physical imager pixels in accordance with the pixel weights.

2. The method of claim 1 , wherein determining the virtual pixel error sensitivities comprises:

determining world error points at intersections of the virtual ray and the error surfaces;

determining imager error pixels by projecting rays from the world error points to the physical imagers;

selecting one of the plurality of error surfaces corresponding to imager error pixels having maximum agreement to obtain a refined scene geometry; and

evaluating the virtual pixel error sensitivities using the refined scene geometry.

3. The method of claim 1 , where determining the geometry of the scene comprises:

estimating an initial surface geometry of the scene;

for a plurality of error levels relative to the initial surface geometry, determining sets of respective error pixel values from the plurality of physical imagers corresponding to common world points on respective error surfaces based on the plurality of error levels;

determining respective correspondence measures between the sets of respective error pixel values; and

selecting an error level corresponding to a maximum one of the respective correspondence measures; and

refining the initial surface geometry based on the selected error level.

4. The method of claim 1 , wherein determining the pixel weights comprises:

detecting an occlusion associated with a given candidate pixel of the plurality of candidate pixels; and

setting a pixel weight associated with the given candidate pixel to zero in response to detecting the occlusion.

5. The method of claim 4 , wherein detecting the occlusion comprises:

detecting a pixel correspondence anomaly associated with the given candidate pixel relative to other candidate pixels of the plurality of candidate pixels; and

detecting the occlusion in response to detecting the pixel correspondence anomaly.

6. The method of claim 4 , wherein detecting the occlusion comprises:

performing an object tracking of an object in a vicinity of the scene; and

detecting the occlusion in response to a location of the object intersecting a ray between a physical imager and the scene.

7. The method of claim 1 , wherein determining the pixel weights comprises:

for a given candidate pixel of the plurality of candidate pixels, determining if a function of a sensitivity for the given candidate pixel exceeds a threshold;

responsive to the function of the sensitivity not exceeding the threshold, assigning to function of the sensitivity as a pixel weight for the given candidate pixel.

8. The method of claim 1 , wherein determining the pixel weights comprises:

for a given candidate pixel of the plurality of candidate pixels, determining if a function of a sensitivity for the given candidate pixel exceeds a threshold;

responsive to the function of the sensitivity exceeding the threshold, assigning a constant value as a pixel weight for the given candidate pixel.

9. The method of claim 1 , wherein determining the value for the virtual pixel based on

the weighted combination of the plurality of candidate pixels using the pixel weights

comprises:

obtaining a quality threshold for the virtual pixel;

identifying a minimum subset of the candidate pixels such that when corresponding pixel weights are added in order of decreasing weight meet or exceed the quality threshold; and

determining the value for the virtual pixel as a weighted combination of the minimum subset of the candidate pixels.

10. The method of claim 9 , wherein the quality threshold for the virtual pixel is

determined based on a pixel location of the virtual pixel.

11. A non-transitory computer-readable storage medium storing instructions for

synthesizing an image corresponding to a virtual perspective of a scene, the instructions

when executed by one or more processors causing the one or more processors to perform

steps including:

obtaining a virtual pose of a virtual perspective, a plurality of poses of physical imagers of a camera array, and a geometry of the scene;

identifying, for a target virtual pixel location of the image, a world point in the scene comprising an intersection point in the scene of a virtual ray extending to the target virtual pixel location;

detecting, based on the world point, the plurality of poses of the physical imagers, and the geometry of the scene, candidate physical imager pixels corresponding to the world point;

determining a plurality of error surfaces corresponding to different depth errors in the geometry of the scene;

determining error surface points at intersections of the error surfaces by rays between the world point and the physical imager pixels;

projecting rays from the error surface points to the virtual imager to determine virtual error pixel locations for each of the candidate physical imager pixels;

determining virtual pixel error sensitivities representing a rate of change of errors between the virtual error pixel locations and the target pixel location in response to a change in depth error of the geometry of the scene;

determining pixel weights associated with each of the candidate physical imager pixels as a function of the virtual pixel error sensitivities; and

determining a value for the target virtual pixel based on a weighted combination of pixel values of the candidate physical imager pixels in accordance with the pixel weights.

12. The non-transitory computer-readable storage medium of claim 11 , wherein determining the virtual pixel error sensitivities comprises:

determining world error points at intersections of the virtual ray and the error surfaces;

determining imager error pixels by projecting rays from the world error points to the physical imagers;

selecting one of the plurality of error surfaces corresponding to imager error pixels having maximum agreement to obtain a refined scene geometry; and

evaluating the virtual pixel error sensitivities using the refined scene geometry.

13. The non-transitory computer-readable storage medium of claim 11 , where

determining the geometry of the scene comprises:

estimating an initial surface geometry of the scene;

for a plurality of error levels relative to the initial surface geometry, determining sets of respective error pixel values from the plurality of physical imagers corresponding to common world points on respective error surfaces based on the plurality of error levels;

determining respective correspondence measures between the sets of respective error pixel values; and

selecting an error level corresponding to a maximum one of the respective correspondence measures; and

refining the initial surface geometry based on the selected error level.

14. The non-transitory computer-readable storage medium of claim 11 , wherein determining the pixel weights comprises:

detecting an occlusion associated with a given candidate pixel of the plurality of candidate pixels; and

setting a pixel weight associated with the given candidate pixel to zero in response to detecting the occlusion.

15. The non-transitory computer-readable storage medium of claim 11 , wherein determining the pixel weights comprises:

for a given candidate pixel of the plurality of candidate pixels, determining if a function of a sensitivity for the given candidate pixel exceeds a threshold;

responsive to the function of the sensitivity not exceeding the threshold, assigning to function of the sensitivity as a pixel weight for the given candidate pixel.

16. The non-transitory computer-readable storage medium of claim 11 , wherein determining the pixel weights comprises:

for a given candidate pixel of the plurality of candidate pixels, determining if a function of a sensitivity for the given candidate pixel exceeds a threshold;

responsive to the function of the sensitivity exceeding the threshold, assigning a constant value as a pixel weight for the given candidate pixel.

17. The non-transitory computer-readable storage medium of claim 11 , wherein

determining the value for the virtual pixel based on the weighted combination of the

plurality of candidate pixels using the pixel weights comprises:

obtaining a quality threshold for the virtual pixel;

identifying a minimum subset of the candidate pixels such that when corresponding pixel weights are added in order of decreasing weight meet or exceed the quality threshold; and

determining the value for the virtual pixel as a weighted combination of the minimum subset of the candidate pixels.

18. An imaging system comprising:

a camera array comprising a plurality of physical imagers;

one or more processors; and

a non-transitory computer-readable storage medium storing instructions for synthesizing an image corresponding to a virtual perspective of a scene, the instructions when executed by one or more processors causing the one or more processors to perform steps including:

obtaining a virtual pose of a virtual perspective, a plurality of poses of the physical imagers of the camera array, and a geometry of the scene;

identifying, for a target virtual pixel location of the image, a world point in the scene comprising an intersection point in the scene of a virtual ray extending to the target virtual pixel location;

detecting, based on the world point, the plurality of poses of the physical imagers, and the geometry of the scene, candidate physical imager pixels corresponding to the world point;

determining a plurality of error surfaces corresponding to different depth errors in the geometry of the scene;

determining error surface points at intersections of the error surfaces by rays between the world point and the physical imager pixels;

projecting rays from the error surface points to the virtual imager to determine virtual error pixel locations for each of the candidate physical imager pixels;

determining virtual pixel error sensitivities representing a rate of change of errors between the virtual error pixel locations and the target pixel location in response to a change in depth error of the geometry of the scene;

determining pixel weights associated with each of the candidate physical imager pixels as a function of the virtual pixel error sensitivities; and

determining a value for the target virtual pixel based on a weighted combination of pixel values of the candidate physical imager pixels in accordance with the pixel weights.

19. The imaging system of claim 18 , wherein determining the virtual pixel error sensitivities comprises:

determining world error points at intersections of the virtual ray and the error surfaces;

determining imager error pixels by projecting rays from the world error points to the physical imagers;

selecting one of the plurality of error surfaces corresponding to imager error pixels having maximum agreement to obtain a refined scene geometry; and

evaluating the virtual pixel error sensitivities using the refined scene geometry.

20. The imaging system of claim 18 , where determining the geometry of the scene

comprises:

estimating an initial surface geometry of the scene;

for a plurality of error levels relative to the initial surface geometry, determining sets of respective error pixel values from the plurality of physical imagers corresponding to common world points on respective error surfaces based on the plurality of error levels;

determining respective correspondence measures between the sets of respective error pixel values; and

selecting an error level corresponding to a maximum one of the respective correspondence measures; and

refining the initial surface geometry based on the selected error level.

Assignments (2)

Continuity (2)

Cited By (9)

US 12,243,162 US 12,261,988 US 12,299,907 US 12,357,397 US 12,383,350 US 12,413,693 US 12,450,777 US 12,664,668 US 12,684,103