Determining traversable space from single images
A model predicts the geometry of both visible and occluded traversable surfaces from input images. The model may be trained from stereo video sequences, using camera poses, per-frame depth, and semantic segmentation to form training data, which is used to supervise an image to image network. In various embodiments, the model is applied to a single RGB image depicting a scene to produce information describing traversable space of the scene that includes occluded traversable. The information describing traversable space can include a segmentation mask of traversable space (both visible and occluded) and non-traversable space and a depth map indicating an estimated depth to traversable surfaces corresponding to each pixel determined to correspond to traversable space.
1 . A non-transitory computer-readable storage medium storing instructions that, when executed by a computing device, cause the computing device to perform operations comprising:
receiving an image comprising a plurality of pixels, the plurality of pixels comprising information representing an object in a scene and depth information for the scene;
inputting the image into a traversability model configured to identify traversable and non-traversable space in the scene, the traversability model configured to:
identify a traversable surface in the scene based on the depth information represented by the plurality of pixels in the image, the traversable surface comprising traversable space and non-traversable space,
identify pixels in the image representing the object as object pixels, the object pixels comprising a subset of object pixels representing a footprint of the object on the traversable surface, and
determine non-traversable space and traversable space on the traversable surface based on the depth information for the object pixels, the non-traversable space including space represented by the subset of object pixels representing the footprint of the object on the traversable surface;
determining a traversable path through the scene for a virtual agent based on the determined non-traversable space and traversable space; and
displaying the virtual agent as traversing along the traversable path.
2 . The non-transitory computer-readable storage medium of claim 1 , wherein the traversability model is further configured to:
determine, based on the object pixels, the object is occluding space in the scene; and
determine, based on the depth information for the object pixels, the traversable space includes the space determined to be occluded by the object in the image.
3 . The non-transitory computer-readable storage medium of claim 2 , wherein the determined traversable path includes traversable space occluded by the object.
4 . The non-transitory computer-readable storage medium of claim 1 , wherein identifying the traversable surface in the scene based on depth information further comprises:
generating a depth map for the scene based on the depth information for the plurality of pixels;
determining traversable space in the scene based on the depth map and the footprint of the object; and
generating the traversable surface based on the depth information for the determined traversable space in the scene.
5 . The non-transitory computer-readable storage medium of claim 1 , wherein identifying the traversable surface in the scene based on depth information further comprises:
identifying pixels in the image representing a static object, the static object comprising a surface; and
determining the traversable surface is the surface of the static object.
6 . The non-transitory computer-readable storage medium of claim 1 , wherein the object pixels comprise an additional subset of object pixels positioned above a plane of the traversable surface, and the traversability model is configured to:
generate the footprint of the object on the traversable surface by projecting the additional subset of object pixels above the plane of the traversable surface onto the plane of the traversable surface.
7 . The non-transitory computer-readable storage medium of claim 1 , wherein the traversability model is configured to identify a type of the object and the footprint of the object on the traversable surface is based on the type of the object.
8 . A system comprising:
a computing device; and
a non-transitory computer-readable storage medium storing computer instructions that, when executed by the computing device, cause the computing device to perform operations comprising:
receiving an image comprising a plurality of pixels, the plurality of pixels comprising information representing an object in a scene and depth information for the scene;
inputting the image into a traversability model configured to identify traversable and non-traversable space in the scene, the traversability model configured to:
identify a traversable surface in the scene based on the depth information represented by the plurality of pixels in the image, the traversable surface comprising traversable space and non-traversable space,
identify pixels in the image representing the object as object pixels, the object pixels comprising a subset of object pixels representing a footprint of the object on the traversable surface, and
determine non-traversable space and traversable space on the traversable surface based on the depth information for the object pixels, the non-traversable space including space represented by the subset of object pixels representing the footprint of the object on the traversable surface;
determining a traversable path through the scene for a virtual agent based on the determined non-traversable space and traversable space; and
displaying the virtual agent as traversing along the traversable path.
9 . The system of claim 8 , wherein the traversability model is further configured to:
determine, based on the object pixels, the object is occluding space in the scene; and
determine, based on the depth information for the object pixels, the traversable space includes the space determined to be occluded by the object in the image.
10 . The system of claim 9 , wherein the determined traversable path includes traversable space occluded by the object.
11 . The system of claim 8 , wherein identifying the traversable surface in the scene based on depth information further comprises:
generating a depth map for the scene based on the depth information for the plurality of pixels;
determining traversable space in the scene based on the depth map and the footprint of the object; and
generating the traversable surface based on the depth information for the determined traversable space in the scene.
12 . The system of claim 8 , wherein identifying the traversable surface in the scene based on depth information further comprises:
identifying pixels in the image representing a static object, the static object comprising a surface; and
determining the traversable surface is the surface of the static object.
13 . The system of claim 8 , wherein the object pixels comprise an additional subset of object pixels positioned above a plane of the traversable surface, and the traversability model is configured to:
generate the footprint of the object on the traversable surface by projecting the additional subset of object pixels above the plane of the traversable surface onto the plane of the traversable surface.
14 . The system of claim 8 , wherein the traversability model is configured to identify a type of the object and the footprint of the object on the traversable surface is based on the type of the object.
15 . A method comprising:
receiving an image comprising a plurality of pixels, the plurality of pixels comprising information representing an object in a scene and depth information for the scene;
inputting the image into a traversability model configured to identify traversable and non-traversable space in the scene, the traversability model configured to:
identify a traversable surface in the scene based on the depth information represented by the plurality of pixels in the image, the traversable surface comprising traversable space and non-traversable space,
identify pixels in the image representing the object as object pixels, the object pixels comprising a subset of object pixels representing a footprint of the object on the traversable surface, and
determine non-traversable space and traversable space on the traversable surface based on the depth information for the object pixels, the non-traversable space including space represented by the subset of object pixels representing the footprint of the object on the traversable surface;
determining a traversable path through the scene for a virtual agent based on the determined non-traversable space and traversable space; and
displaying the virtual agent as traversing along the traversable path.
16 . The method of claim 15 , wherein the traversability model is further configured to:
determine, based on the object pixels, the object is occluding space in the scene; and
determine, based on the depth information for the object pixels, the traversable space includes the space determined to be occluded by the object in the image.
17 . The method of claim 16 , wherein the determined traversable path includes traversable space occluded by the object.
18 . The method of claim 15 , wherein identifying the traversable surface in the scene based on depth information further comprises:
generating a depth map for the scene based on the depth information for the plurality of pixels;
determining traversable space in the scene based on the depth map and the footprint of the object; and
generating the traversable surface based on the depth information for the determined traversable space in the scene.
19 . The method of claim 15 , wherein identifying the traversable surface in the scene based on depth information further comprises:
identifying pixels in the image representing a static object, the static object comprising a surface; and
determining the traversable surface is the surface of the static object.
20 . The method of claim 15 , wherein the object pixels comprise an additional subset of object pixels positioned above a plane of the traversable surface, and the traversability model is configured to:
generate the footprint of the object on the traversable surface by projecting the additional subset of object pixels above the plane of the traversable surface onto the plane of the traversable surface.
21 . The method of claim 15 , wherein the traversability model is configured to identify a type of the object and the footprint of the object on the traversable surface is based on the type of the object.