Synthesizing an image from a virtual perspective using pixels from a physical imager array weighted based on depth error sensitivity
A method assigns weights to physical imager pixels in order to generate photorealistic images for virtual perspectives in real-time. The imagers are arranged in three-dimensional space such that they sparsely sample the light field within a scene of interest. This scene is defined by the overlapping fields of view of all the imagers or for subsets of imagers. The weights assigned to imager pixels are calculated based on the relative poses of the virtual perspective and physical imagers, properties of the scene geometry, and error associated with the measurement of geometry. This method is particularly useful for accurately rendering numerous synthesized perspectives within a digitized scene in real-time in order to create immersive, three-dimensional experiences for applications such as performing surgery, infrastructure inspection, or remote collaboration.
1. A method for synthesizing an image corresponding to a virtual perspective of a scene, the method comprising:
obtaining a plurality of images of the scene from respective physical imagers;
determining an initial predicted surface geometry of the scene;
determining, for a virtual pixel of the image corresponding to the virtual perspective of the scene based on the initial predicted surface geometry, a plurality of candidate pixels from different ones of the physical imagers that are predicted to correspond to a same world point in the scene as the virtual pixel;
detecting for each of the plurality of candidate pixels, respective sensitivities to an error in the initial predicted surface geometry;
determining pixel weights for each of the plurality of candidate pixels based on the respective sensitivities; and
determining a value for the virtual pixel based on a weighted combination of the plurality of candidate pixels using the pixel weights.
2. The method of claim 1 , wherein detecting the respective sensitivities comprises:
determining a plurality of error surfaces corresponding to different depth errors in the geometry of the scene;
for a given candidate pixel, determining a depth error sensitivity function mapping the different depth errors to respective changes in a virtual pixel location that corresponds to the given candidate pixel; and
determining a sensitivity of the given candidate pixel based on a derivative of the depth error sensitivity function.
3. The method of claim 1 , where determining the initial predicted surface geometry of the scene comprises:
estimating an initial surface geometry of the scene;
for a plurality of error levels relative to the initial surface geometry, determining sets of respective error pixel values from the plurality of physical imagers corresponding to common world points on respective error surfaces based on the plurality of error levels;
determining respective correspondence measures between the sets of respective error pixel values;
selecting an error level corresponding to a maximum one of the respective correspondence measures; and
refining the initial surface geometry based on the selected error level.
4. The method of claim 1 , wherein determining the pixel weights comprises:
detecting an occlusion associated with a given candidate pixel of the plurality of candidate pixels; and
setting a pixel weight associated with the given candidate pixel to zero in response to detecting the occlusion.
5. The method of claim 4 , wherein detecting the occlusion comprises:
detecting a pixel correspondence anomaly associated with the given candidate pixel relative to other candidate pixels of the plurality of candidate pixels; and
detecting the occlusion in response to detecting the pixel correspondence anomaly.
6. The method of claim 4 , wherein detecting the occlusion comprises:
performing an object tracking of an object in a vicinity of the scene; and
detecting the occlusion in response to a location of the object intersecting a ray between a physical imager and the scene.
7. The method of claim 1 , wherein determining the pixel weights comprises:
for a given candidate pixel of the plurality of candidate pixels, determining if a function of a sensitivity for the given candidate pixel exceeds a threshold; and
responsive to the function of the sensitivity not exceeding the threshold, assigning to function of the sensitivity as a pixel weight for the given candidate pixel.
8. The method of claim 1 , wherein determining the pixel weights comprises:
for a given candidate pixel of the plurality of candidate pixels, determining if a function of a sensitivity for the given candidate pixel exceeds a threshold; and
responsive to the function of the sensitivity exceeding the threshold, assigning a constant value as a pixel weight for the given candidate pixel.
9. The method of claim 1 , wherein determining the value for the virtual pixel based on the weighted combination of the plurality of candidate pixels using the pixel weights comprises:
obtaining a quality threshold for the virtual pixel;
identifying a minimum subset of the candidate pixels such that when corresponding pixel weights are added in order of decreasing weight meet or exceed the quality threshold; and
determining the value for the virtual pixel as a weighted combination of the minimum subset of the candidate pixels.
10. The method of claim 9 , wherein the quality threshold for the virtual pixel is determined based on a pixel location of the virtual pixel.
11. A non-transitory computer-readable storage medium storing instructions for synthesizing an image corresponding to a virtual perspective of a scene, the instructions when executed by a processor causing the processor to perform steps including:
obtaining a plurality of images of the scene from respective physical imagers;
determining an initial predicted surface geometry of the scene;
determining, for a virtual pixel of the image corresponding to the virtual perspective of the scene based on the initial predicted surface geometry, a plurality of candidate pixels from different ones of the physical imagers that are predicted to correspond to a same world point in the scene as the virtual pixel;
detecting for each of the plurality of candidate pixels, respective sensitivities to an error in the initial predicted surface geometry;
determining pixel weights for each of the plurality of candidate pixels based on the respective sensitivities; and
determining a value for the virtual pixel based on a weighted combination of the plurality of candidate pixels using the pixel weights.
12. The non-transitory computer-readable storage medium of claim 11 , wherein detecting the respective sensitivities comprises:
determining a plurality of error surfaces corresponding to different depth errors in the geometry of the scene;
for a given candidate pixel, determining a depth error sensitivity function mapping the different depth errors to respective changes in a virtual pixel location that corresponds to the given candidate pixel; and
determining a sensitivity of the given candidate pixel based on a derivative of the depth error sensitivity function.
13. The non-transitory computer-readable storage medium of claim 11 , where determining the initial predicted surface geometry of the scene comprises:
estimating an initial surface geometry of the scene;
for a plurality of error levels relative to the initial surface geometry, determining sets of respective error pixel values from the plurality of physical imagers corresponding to common world points on respective error surfaces based on the plurality of error levels;
determining respective correspondence measures between the sets of respective error pixel values;
selecting an error level corresponding to a maximum one of the respective correspondence measures; and
refining the initial surface geometry based on the selected error level.
14. The non-transitory computer-readable storage medium of claim 11 , wherein determining the pixel weights comprises:
detecting an occlusion associated with a given candidate pixel of the plurality of candidate pixels; and
setting a pixel weight associated with the given candidate pixel to zero in response to detecting the occlusion.
15. The non-transitory computer-readable storage medium of claim 14 , wherein detecting the occlusion comprises:
detecting a pixel correspondence anomaly associated with the given candidate pixel relative to other candidate pixels of the plurality of candidate pixels; and
detecting the occlusion in response to detecting the pixel correspondence anomaly.
16. The non-transitory computer-readable storage medium of claim 14 , wherein detecting the occlusion comprises:
performing an object tracking of an object in a vicinity of the scene; and
detecting the occlusion in response to a location of the object intersecting a ray between a physical imager and the scene.
17. The non-transitory computer-readable storage medium of claim 11 , wherein determining the pixel weights comprises:
for a given candidate pixel of the plurality of candidate pixels, determining if a function of a sensitivity for the given candidate pixel exceeds a threshold; and
responsive to the function of the sensitivity not exceeding the threshold, assigning to function of the sensitivity as a pixel weight for the given candidate pixel.
18. The non-transitory computer-readable storage medium of claim 11 , wherein determining the pixel weights comprises:
for a given candidate pixel of the plurality of candidate pixels, determining if a function of a sensitivity for the given candidate pixel exceeds a threshold; and
responsive to the function of the sensitivity exceeding the threshold, assigning a constant value as a pixel weight for the given candidate pixel.
19. An imaging system comprising:
a camera array comprising a plurality of physical imagers;
one or more processors; and
a non-transitory computer-readable storage medium storing instructions for synthesizing an image corresponding to a virtual perspective of a scene, the instructions when executed by one or more processors causing the one or more processors to perform steps including:
obtaining a plurality of images of the scene from respective physical imagers;
determining an initial predicted surface geometry of the scene;
determining, for a virtual pixel of the image corresponding to the virtual perspective of the scene based on the initial predicted surface geometry, a plurality of candidate pixels from different ones of the physical imagers that are predicted to correspond to a same world point in the scene as the virtual pixel;
detecting for each of the plurality of candidate pixels, respective sensitivities to an error in the initial predicted surface geometry;
determining pixel weights for each of the plurality of candidate pixels based on the respective sensitivities; and
determining a value for the virtual pixel based on a weighted combination of the plurality of candidate pixels using the pixel weights.
20. The imaging system of claim 19 , wherein detecting the respective sensitivities comprises:
determining a plurality of error surfaces corresponding to different depth errors in the geometry of the scene;
for a given candidate pixel, determining a depth error sensitivity function mapping the different depth errors to respective changes in a virtual pixel location that corresponds to the given candidate pixel; and
determining a sensitivity of the given candidate pixel based on a derivative of the depth error sensitivity function.