Data fusion system for a vehicle equipped with unsynchronized perception sensors
A sensor data fusion system for a vehicle with multiple sensors includes a first-sensor, a second-sensor, and a controller-circuit. The first-sensor is configured to output a first-frame of data and a subsequent-frame of data indicative of objects present in a first-field-of-view. The first-frame is characterized by a first-time-stamp, the subsequent-frame of data characterized by a subsequent-time-stamp different from the first-time-stamp. The second-sensor is configured to output a second-frame of data indicative of objects present in a second-field-of-view that overlaps the first-field-of-view. The second-frame is characterized by a second-time-stamp temporally located between the first-time-stamp and the subsequent-time-stamp. The controller-circuit is configured to synthesize an interpolated-frame from the first-frame and the subsequent-frame. The interpolated-frame is characterized by an interpolated-time-stamp that corresponds to the second-time-stamp. The controller-circuit fuses the interpolated-frame with the second-frame to provide a fused-frame of data characterized by the interpolated-time-stamp, and operates the host-vehicle in accordance with the fused-frame.
1. A sensor data fusion system for a host-vehicle with multiple sensors, said system comprising:
a camera mounted on the host-vehicle, said camera configured to output a first-frame of camera data and a subsequent-frame of camera data indicative of objects present in a first-field-of-view, said first-frame of camera data characterized by a first-time-stamp, said subsequent-frame of camera data characterized by a subsequent-time-stamp different from the first-time-stamp;
a LiDAR mounted on the host-vehicle, said LiDAR configured to output a second-frame of point-cloud data indicative of objects present in a second-field-of-view that overlaps the first-field-of-view, said second-frame of point-cloud data characterized by a second-time-stamp temporally located between the first-time-stamp and the subsequent-time-stamp; and
a controller-circuit in communication with the camera and the LiDAR, said controller-circuit configured to:
synthesize an interpolated-frame from the first-frame and the subsequent-frame using motion flow analysis responsive to determining that both the first-time-stamp and the subsequent-time-stamp differ from the second-time-stamp by greater than a time-threshold, said interpolated-frame characterized by an interpolated-time-stamp that corresponds to the second-time-stamp, such that the interpolated-frame is temporally synchronized with the point-cloud data of the second-frame;
fuse the interpolated-frame with the second-frame to provide a fused-frame of data characterized by the interpolated-time-stamp;
determine a three-dimensional model of an environment surrounding the host-vehicle based on the fused-frame; and
operate the host-vehicle in accordance with the three-dimensional model of the environment.
2. The system of claim 1 , wherein the objects present in the first-field-of-view are in a travel path of the host-vehicle.
3. The system of claim 1 , wherein the objects present in the second-field-of-view are in a travel path of the host-vehicle.
4. The system of claim 1 , wherein the controller-circuit is further configured to:
synthesize the interpolated-frame from the first-frame responsive to determining that the first-time-stamp and the subsequent-time-stamp differ from the second-time-stamp by less than five milliseconds, wherein the interpolated-frame is the same as the first-frame.
5. A controller-circuit for a sensor data fusion system for a host-vehicle with multiple sensors, said controller-circuit comprising:
a first-input configured to communicate with a camera mounted on the host-vehicle, said camera configured to output a first-frame of camera data and a subsequent-frame of camera data indicative of objects present in a first-field-of-view, said first-frame of camera data characterized by a first-time-stamp, said subsequent-frame of camera data characterized by a subsequent-time-stamp different from the first-time-stamp;
a second-input configured to communicate with a LiDAR mounted on the host-vehicle, said LiDAR configured to output a second-frame of point-cloud data indicative of objects present in a second-field-of-view that overlaps the first-field-of-view, said second-frame of point-cloud data characterized by a second-time-stamp temporally located between the first-time-stamp and the subsequent-time-stamp; and
a processor in communication with the camera and the LiDAR, said processor configured to:
synthesize an interpolated-frame from the first-frame and the subsequent-frame using motion flow analysis responsive to determining that the both the first-time-stamp and the subsequent-time-stamp differ from the second-time-stamp by greater than a time-threshold, said interpolated-frame characterized by an interpolated-time-stamp that corresponds to the second-time-stamp, such that the interpolated-frame is temporally synchronized with the point-cloud of the second-frame;
fuse the interpolated-frame with the second-frame to provide a fused-frame of data characterized by the interpolated-time-stamp,
determine a three-dimensional model of an environment surrounding the host-vehicle based on the fused-frame; and
operate the host-vehicle in accordance with the three-dimensional model of the environment.
6. The controller-circuit of claim 5 , wherein the objects present in the first-field-of-view are in a travel path of the host-vehicle.
7. The controller-circuit of claim 5 , wherein the objects present in the second-field-of-view are in a travel path of the host-vehicle.
8. The controller-circuit of claim 5 , wherein the processor is further configured to:
synthesize the interpolated-frame from the first-frame responsive to determining that the first-time-stamp and the subsequent-time-stamp differ from the second-time-stamp by less than five milliseconds, wherein the interpolated-frame is the same as the first-frame.
9. A method of operating a sensor data fusion system for a host-vehicle with multiple sensors, said method comprising:
receiving a first-frame of camera data and a subsequent-frame of camera data from a camera mounted on the host-vehicle, said first-frame of camera data and said subsequent-frame of camera data indicative of objects present in a first-field-of-view, said first-frame of camera data characterized by a first-time-stamp, said subsequent-frame of camera data characterized by a subsequent-time-stamp different from the first-time-stamp;
receiving a second-frame of LiDAR data from a LiDAR mounted on the host-vehicle, said second-frame of LiDAR data indicative of objects present in a second-field-of-view that overlaps the first-field-of-view, said second-frame of LiDAR data characterized by a second-time-stamp temporally located between the first-time-stamp and the subsequent-time-stamp;
synthesizing an interpolated-frame from the first-frame and the subsequent-frame using motion flow analysis responsive to determining that both the first-time-stamp and the subsequent-time-stamp differ from the second-time-stamp by greater than a time-threshold, said interpolated-frame characterized by an interpolated-time-stamp that corresponds to the second-time-stamp, such that the interpolated-frame is temporally synchronized with the LiDAR data of the second-frame;
fusing the interpolated-frame with the second-frame to provide a fused-frame of data characterized by the interpolated-time-stamp;
determining a three-dimensional model of an environment based on the fused-frame; and
operating the host-vehicle in accordance with the three-dimensional model of the environment.
10. The method of claim 9 , wherein the objects present in the first-field-of-view are in a travel path of the host-vehicle.
11. The method of claim 9 , wherein the objects present in the second-field-of-view are in a travel path of the host-vehicle.
12. The method of claim 9 , further comprising:
synthesizing the interpolated-frame from the first-frame responsive to determining that the first-time-stamp and the subsequent-time-stamp differ from the second-time-stamp by less than five milliseconds, wherein the interpolated-frame is the same as the first-frame.
13. A sensor data fusion system for a host-vehicle with multiple sensors, said system comprising:
a LiDAR mounted on the host-vehicle, said LiDAR configured to output a first-frame of point-cloud data and a subsequent-frame of point cloud data indicative of objects present in a first-field-of-view, said first-frame of point cloud data characterized by a first-time-stamp, said subsequent-frame of point-cloud data characterized by a subsequent-time-stamp different from the first-time-stamp;
a camera mounted on the host-vehicle, said cameraconfigured to output a second-frame of camera data indicative of objects present in a second-field-of-view that overlaps the first-field-of-view, said second-frame of camera data characterized by a second-time-stamp temporally located between the first-time-stamp and the subsequent-time-stamp; and
a controller-circuit in communication with the LiDAR and the camera, said controller-circuit configured to:
synthesize an interpolated-frame from the first-frame and the subsequent-frame using motion flow analysis responsive to determining that both the first-time-stamp and the subsequent-time-stamp differ from the second-time-stamp by greater than a time-threshold, said interpolated-frame characterized by an interpolated-time-stamp that corresponds to the second-time-stamp, such that the interpolated-frame is temporally synchronized with the camera data of the second-frame;
fuse the interpolated-frame with the second-frame to provide a fused-frame of data characterized by the interpolated-time-stamp;
determine a three-dimensional model of an environment surrounding the host-vehicle based on the fused-frame; and operate the host-vehicle in accordance with the three-dimensional model of the environment.
14. The system of claim 13 , wherein the objects present in the first-field-of-view are in a travel path of the host-vehicle.
15. The system of claim 13 , wherein the objects present in the second-field-of-view are in a travel path of the host-vehicle.
16. The system of claim 13 , wherein the controller-circuit is further configured to:
synthesize the interpolated-frame from the first-frame responsive to determining that the first-time-stamp and the subsequent-time-stamp differ from the second-time-stamp by less than five milliseconds, wherein the interpolated-frame is the same as the first-frame.