Retrofit light detection and ranging (LIDAR)-based vehicle system to operate with vision-based sensor data
Systems and methods for retrofitting a light detection and ranging (LIDAR)-based vehicle computing system to operate with vision-based sensor data are provided. For example, a method implemented by a vehicle may include receiving, from one or more sensors of a first sensing modality at the vehicle, first sensor data associated with a surrounding environment of the vehicle; and retrofitting a vehicle controller of the vehicle that is based on a second sensing modality different from the first sensing modality to operate on the first sensor data, where the retrofitting includes generating second sensor data from the first sensor data based on the second sensing modality; and determining, by the vehicle controller, an action for the vehicle based at least in part on the generated second sensor data.
1. A method, the method comprising:
receiving, from one or more sensors of a first sensing modality disposed on a vehicle, first sensor data including an image of a scene in associated with a surrounding environment of the vehicle, wherein the one or more sensors of the first sensing modality are vision-based sensors; and
retrofitting a vehicle controller of the vehicle that is based on a second sensing modality different from the first sensing modality to operate on the first sensor data, wherein the second sensing modality is light detection and ranging (LIDAR), wherein the vehicle controller is based on a particular LIDAR sensor, wherein the retrofitting comprises:
generating second sensor data from the first sensor data based on the second sensing modality, wherein the generating of the second sensor data comprises generating, based on the image, a point cloud representative of at least a portion of the scene, wherein the generating of the point cloud representative of the at least the portion of the scene in the image is based on:
a characteristic of the particular LIDAR sensor, and
a limitation of the particular LIDAR sensor;
determining, by the vehicle controller, an action for the vehicle based at least in part on the generated second sensor data; and
implementing, by the vehicle, the action determined based on the generated second sensor data,
wherein at least one of the one or more sensors of the first sensing modality have been retrofit into the vehicle to replace a sensor of the second sensing modality previously disposed on the vehicle.
2. The method of claim 1 , wherein the generating of the second sensor data comprises:
determining, from the image, at least one object in the portion of the scene; and
generating a first point cloud representative of the at least one determined object.
3. The method of claim 1 , wherein the limitation of the particular LIDAR sensor is associated with at least one of a scan range of the particular LIDAR sensor, a reflectivity of the particular LIDAR sensor, or a behavior of the particular LIDAR sensor under a weather condition.
4. The method of claim 1 , wherein the generating of the point cloud representative of the at least the portion of the scene in the image is further based on a heuristic algorithm that emulates the characteristic of the particular LIDAR sensor.
5. The method of claim 1 , wherein the generating of the second sensor data comprises:
processing the first sensor data using a machine learning model to generate the second sensor data.
6. The method of claim 5 , wherein the machine learning model is a generator model trained jointly with a discriminator model in a generative adversarial network (GAN) model.
7. The method of claim 1 , further comprising:
receiving, from the one or more sensors of the first sensing modality at the vehicle, third sensor data;
receiving, from one or more sensors of the second sensing modality at the vehicle, fourth sensor data;
generating fifth sensor data of the second sensing modality based on the third sensor data of the first sensing modality and the fourth sensor data of the second sensing modality; and
determining, by the vehicle controller, another action for the vehicle based on the generated fifth sensor data.
8. A vehicle comprising:
one or more vision sensors to capture image data;
a sensor data converter to generate light detection and ranging (LIDAR) data based on the image data, wherein the sensor data converter generates the LIDAR data by:
generating a point cloud representative of at least one object captured by the image data, wherein the generating of the point cloud representative of the at least the portion of the scene in the image is based on:
a characteristic of a particular LIDAR sensor, and
a limitation of the particular LIDAR sensor;
a vehicle controller to determine an action for the vehicle based at least in part on the generated LIDAR data, wherein the vehicle controller operates based on LIDAR sensing; and
wherein the vehicle controller is further configured to implementing the action determined based on the generated LIDAR data,
wherein at least one of the one or more vision sensors have been retrofit into the vehicle to replace a LIDAR sensor previously disposed on the vehicle.
9. The vehicle of claim 8 , wherein the sensor data converter generates the LIDAR data by:
generating a point cloud representative of at least one object captured by the image data.
10. The vehicle of claim 8 , wherein the sensor data converter generates the LIDAR data by processing the image data using a machine learning model to generate the LIDAR data.
11. The vehicle of claim 10 , wherein the machine learning model is a generator model trained jointly with a discriminator model in a generative adversarial network (GAN) model.