Systems and methods for image-based free space detection
The present disclosure provides systems and methods for image-based free space detection. In one example embodiment, a computer-implemented method includes obtaining image data representing the environment proximate to the autonomous vehicle, the image data including a representation of the environment from a perspective associated with the autonomous vehicle. The method includes reprojecting the image data to generate a reprojected image. The method includes inputting the reprojected image to a machine-learned detector model. The method includes obtaining as an output of the machine-learned detector model, object data characterizing one or more objects in the environment. The method includes determining a free space in the environment based at least in part on the object data.
1. A computer-implemented method of detecting free space in a surrounding environment of an autonomous vehicle, the method comprising:
obtaining, by a computing system comprising one or more computing devices, image data from one or more cameras affixed to the autonomous vehicle, the image data comprising a representation of the surrounding environment from a perspective associated with each of the one or more cameras affixed to the autonomous vehicle;
generating, by the computing system, a cylindrical image depicting an approximately 360-degree perspective of the surrounding environment based, at least in part, on the image data from the one or more cameras;
generating, by the computing system, a reprojected image depicting a top-down perspective of the approximately 360-degree perspective of the surrounding environment based, at least in part, on the cylindrical image;
inputting, by the computing system, the reprojected image to a machine-learned detector model to obtain, as an output of the machine-learned detector model, object data characterizing one or more objects in the surrounding environment;
determining, by the computing system, a free space in the surrounding environment of the autonomous vehicle based at least in part on the object data.
2. The computer-implemented method of claim 1 , further comprising:
controlling, by the computing system, the autonomous vehicle to travel through the environment based at least in part on the determined free space.
3. The computer-implemented method of claim 1 , wherein the perspective associated with the one or more cameras affixed to the autonomous vehicle is from a location that is proximate to one or more locations along a motion path of the autonomous vehicle.
4. The computer-implemented method of claim 1 , wherein each pixel of the reprojected image is associated with a single fixed distance in the environment.
5. The computer-implemented method of claim 1 , wherein determining, by the computing system, a free space in the environment based at least in part on the object data comprises:
determining, by the computing system, one or more regions in the environment that the autonomous vehicle can travel without colliding with the one or more objects in the environment.
6. The computer-implemented method of claim 1 , wherein the image data includes a front-facing image, the front-facing image representing the environment from a perspective of a direction that the autonomous vehicle is facing.
7. The computer-implemented method of claim 1 , wherein the image data includes a plurality of images, the plurality of images representing the surrounding environment from a plurality of perspectives proximate to the autonomous vehicle, the plurality of perspectives including one or more overlapping perspectives.
8. The computer-implemented method of claim 1 , wherein the object data includes a predicted identification of one or more objects in the environment, a predicted classification of the one or more objects, and an estimated distance from the autonomous vehicle to the one or more objects.
9. The computer-implemented method of claim 8 , wherein the predicted classification of the one or more objects indicates whether each object is a moving object or a stationary object.
10. The computer-implemented method of claim 1 , further comprising:
determining, by the computing system, a loss function that evaluates a difference between the determined free space and a ground truth free space; and
training, by the computing system, the machine-learned detector model based at least in part on the loss function.
11. The computer-implemented method of claim 10 , wherein determining, by the computing system, a loss function that evaluates the difference between the determined free space in the environment and a ground truth free space comprises:
obtaining, by the computing system, a first representation of the ground truth free space, the first representation corresponding to a grid-view;
performing, by the computing system, non-uniform ray tracing on the first representation to generate a second representation of the ground truth free space, the second representation corresponding to a radial-view; and
evaluating, by the computing system, a difference between the determined free space and the second representation of the ground truth free space, the difference being associated with the loss function.
12. A free space detection system comprising:
one or more processors;
a machine-learned detector model, wherein the machine-learned detector model has been trained to receive a reprojected image and, in response to receipt of the reprojected image, output object data characterizing one or more objects in a surrounding environment of an autonomous vehicle; and
at least one tangible, non-transitory computer readable medium that stores instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising:
obtaining input images from one or more cameras affixed to the autonomous vehicle, wherein the input image represents the surrounding environment of the autonomous vehicle from a perspective associated with each of the one or more cameras affixed to the autonomous vehicle;
generating a cylindrical image depicting an approximately 360-degree perspective of the surrounding environment based, at least in part, on the input images from the one or more camera
generating the reprojected image depicting a top-down perspective of the approximately 360-degree perspective of the surrounding environment based, at least in part, on the cylindrical image;
inputting the reprojected image to the machine-learned detector model to receive, as an output of the machine-learned detector model, object data characterizing one or more objects in the surrounding environment of the autonomous vehicle; and
determining a free space in the surrounding environment of the autonomous vehicle based at least in part on the object data.
13. The free space detection system of claim 12 , further comprising:
generating a graphical representation of the free space in the surrounding environment of the autonomous vehicle based at least in part on the object data characterizing the one or more objects identified in the surrounding environment.
14. The free space detection system of claim 12 , wherein the object data characterizing the one or more objects in the surrounding environment comprises a distance from a given reference point to each object detected in the environment.
15. The free space detection system of claim 12 , wherein the machine-learned detector model comprises a neural network.
16. A computer-implemented method of detecting free space in a surrounding environment of an autonomous vehicle, the method comprising:
obtaining, by a computing system comprising one or more computing devices, image data representing the surrounding environment of the autonomous vehicle, the image data including a representation of the surrounding environment from a perspective associated with the autonomous vehicle;
generating, by the computing system, a reprojected image based on the image data;
inputting, by the computing system, the reprojected image to a machine-learned detector model;
obtaining, by the computing system, as an output of the machine-learned detector model, object data characterizing one or more objects in the environment;
determining, by the computing system, a free space in the surrounding environment based at least in part on the object data;
determining, by the computing system, a loss function that evaluates a difference between the determined free space and a ground truth free space; and
training, by the computing system, the machine-learned detector model based at least in part on the loss function.
17. The computer-implemented method of claim 16 , wherein determining, by the computing system, a loss function that evaluates the difference between the determined free space in the environment and a ground truth free space comprises:
obtaining, by the computing system, a first representation of the ground truth free space, the first representation corresponding to a grid-view;
performing, by the computing system, non-uniform ray tracing on the first representation to generate a second representation of the ground truth free space, the second representation corresponding to a radial-view; and
evaluating, by the computing system, a difference between the determined free space and the second representation of the ground truth free space, the difference being associated with the loss function.
18. The computer-implemented method of claim 17 , wherein performing, by the computing system, non-uniform ray tracing on the first representation to generate a second representation of the ground truth free space comprises:
identifying, by the computing system, a region of interest in the first representation of the ground truth free space;
performing, by the computing system, ray tracing to one or more objects in the first representation that are outside the region of interest with a first ray density; and
performing, by the computing system, ray tracing to one or more objects in the first representation that are inside the region of interest with a second ray density that is higher than the first ray density.