Expanded field of view using multiple cameras
Methods and apparatus for providing an expanded field of view (FoV) using multiple cameras of a device. A device may include front-facing cameras that capture views of an environment, as well as side cameras that capture wider-angle views. Images from the front and side cameras may be processed to provide a stereo view of the environment with an expanded FoV. Images from the front cameras are processed to provide stereo images, and depth information is generated from the stereo images. The depth information is expanded into a second layer, and a median depth is set for a third layer. The side camera images are undistorted, pixel information from the side camera images is then reprojected into the second and third layers, and the second and third layers are blurred.
1 . A system, comprising:
one or more primary cameras configured to capture one or more first images of an environment;
one or more secondary cameras configured to capture one or more second images of the environment from different viewpoints than the one or more primary cameras; and
one or more processors configured to:
determine depth information for the one or more first images;
extend the depth information outward from one or more edges of the one or more first images to generate an expanded region; and
reproject pixel data from the one or more second images into the expanded region to generate an expanded field of view (FoV) image of a scene in the environment.
2 . The system as recited in claim 1 , wherein, to extend the depth information outward from one or more edges of the one or more first images to generate an expanded region, the one or more processors are configured to:
extend the depth information outward from the one or more edges of the one or more first images for a first distance to generate a second layer, wherein the one or more first images are a first layer; and
extend a median depth determined from the depth information outward from one or more edges of the second layer for a second distance to generate a second layer.
3 . The system as recited in claim 2 , wherein the first and second distances are dynamically determined from content of the one or more first images.
4 . The system as recited in claim 2 , wherein, to extend the depth information outward from the one or more edges of the one or more first images for a first distance to generate a second layer, wherein the one or more first images are a first layer, the depth information is extended in an inpainting step in which depth information from edges of a depth map is propagated outwards row and column wise away from a center of the depth map.
5 . The system as recited in claim 1 , wherein the one or more processors are configured to undistort the one or more second images prior to said reproject.
6 . The system as recited in claim 1 , wherein the one or more processors are configured to blur the expanded region.
7 . The system as recited in claim 1 , wherein the one or more primary cameras include two front-facing cameras on a device that provide stereo images of the scene, and wherein the one or more secondary cameras include at least one camera on at least two sides of the device.
8 . The system as recited in claim 7 , wherein the device is a head-mounted device (HMD), and wherein the one or more processors are configured to provide the expanded FoV image to a display panel of the HMD for display to a user.
9 . The system as recited in claim 7 , wherein the cameras on the at least two sides of the device include wider FOV cameras than the two front-facing cameras.
10 . The system as recited in claim 7 , wherein the images captured by at least one camera on a first side of the device are used to extend the FoV of a first one of the two front-facing cameras, and wherein the images captured by at least one camera on a second side of the device are used to extend the FoV of a second one of the two front-facing cameras.
11 . The system as recited in claim 1 , wherein the depth information is sparse depth information that provides depth for edges in a scene captured by the one or more primary cameras.
12 . The system as recited in claim 1 , wherein the one or more secondary cameras include grayscale cameras, and wherein the one or more processors are further configured to extend color from images captured by the one or more primary colors into the expanded region.
13 . A method, comprising:
capturing, by one or more primary cameras, one or more first images of an environment;
capturing, by one or more secondary cameras, one or more second images of the environment from different viewpoints than the one or more primary cameras; and
performing, by one or more processors:
determining depth information for the one or more first images;
extending the depth information outward from one or more edges of the one or more first images to generate an expanded region; and
reprojecting pixel data from the one or more second images into the expanded region to generate an expanded field of view (FoV) image of a scene in the environment.
14 . The method as recited in claim 13 , wherein extending the depth information outward from one or more edges of the one or more first images to generate an expanded region comprises:
extending the depth information outward from the one or more edges of the one or more first images for a first distance; and
extending a median depth determined from the depth information outward from one or more edges of a second layer for a second distance.
15 . The method as recited in claim 14 , further comprising dynamically determining the first and second distances from content of the one or more first images.
16 . The method as recited in claim 13 , further comprising undistorting the one or more second images prior to said reprojecting.
17 . The method as recited in claim 13 , further comprising blurring the expanded region.
18 . The method as recited in claim 13 , wherein the one or more primary cameras include two front-facing cameras on a device that provide stereo images of the scene, and wherein the one or more secondary cameras include at least one camera on at least two sides of the device.
19 . The method as recited in claim 18 , wherein reprojecting pixel data from the one or more second images into the expanded region to generate an expanded field of view (FoV) image of a scene in the environment comprises:
reprojecting pixel data from the images captured by at least one camera on a first side of the device into an expanded region around an image captured by a first one of the two front-facing cameras; and
reprojecting pixel data from the images captured by at least one camera on a second side of the device into an expanded region around an image captured by a second one of the two front-facing cameras.
20 . A device, comprising:
two front-facing cameras configured to capture stereo images of a scene in an environment;
at least one camera on at least two sides of the device configured to capture additional images of the scene; and
one or more processors configured to render expanded field of view (FoV) stereo images of the scene, wherein, to render the expanded FoV stereo images, the one or more processors are configured to:
determine depth information for the stereo images captured by the front-facing cameras;
extend the depth information outward from the one or more edges of each of the stereo images for a first distance to generate a second layer, wherein the stereo images are a first layer;
extend a median depth determined from the depth information outward from one or more edges of the second layer for a second distance to generate a third layer;
reproject pixel data from the images captured by at least one camera on a first side of the device into the second and third layers around an image captured by a first one of the two front-facing cameras; and
reproject pixel data from the images captured by at least one camera on a second side of the device into the second and third layers around an image captured by a second one of the two front-facing cameras.