Data fusion system for a vehicle equipped with unsynchronized perception sensors

1. A sensor data fusion system for a host-vehicle with multiple sensors, said system comprising:

a camera mounted on the host-vehicle, said camera configured to output a first-frame of camera data and a subsequent-frame of camera data indicative of objects present in a first-field-of-view, said first-frame of camera data characterized by a first-time-stamp, said subsequent-frame of camera data characterized by a subsequent-time-stamp different from the first-time-stamp;

a radar mounted on the host-vehicle, said radar configured to output a second-frame of radar map data indicative of objects present in a second-field-of-view that overlaps the first-field-of-view, said second-frame of radar map data characterized by a second-time-stamp temporally located between the first-time-stamp and the subsequent-time-stamp; and

a controller-circuit in communication with the camera and the radar, said controller-circuit configured to:

synthesize an interpolated-frame from the first-frame and the subsequent-frame using motion flow analysis responsive to determining that both the first-time-stamp and the subsequent-time-stamp differ from the second-time-stamp by greater than a time-threshold, said interpolated-frame characterized by an interpolated-time-stamp that corresponds to the second-time-stamp, such that the interpolated-frame is temporally synchronized with the radar map data of the second-frame;

fuse the interpolated-frame with the second-frame to provide a fused-frame of data characterized by the interpolated-time-stamp;

determine a three-dimensional model of an environment surrounding the host-vehicle based on the fused-frame; and

operate the host-vehicle in accordance with the three-dimensional model of the environment.

2. The system of claim 1 , wherein the objects present in the first-field-of-view are in a travel path of the host-vehicle.

3. The system of claim 1 , wherein the objects present in the second-field-of-view are in a travel path of the host-vehicle.

4. The system of claim 1 , wherein the controller-circuit is further configured to:

synthesize the interpolated-frame from the first-frame responsive to determining that the first-time-stamp and the subsequent-time-stamp differ from the second-time-stamp by less than five milliseconds, wherein the interpolated-frame is the same as the first-frame.

5. A controller-circuit for a sensor data fusion system for a host-vehicle with multiple sensors, said controller-circuit comprising:

a first-input configured to communicate with a camera mounted on the host-vehicle, said camera configured to output a first-frame of camera data and a subsequent-frame of camera data indicative of objects present in a first-field-of-view, said first-frame of camera data characterized by a first-time-stamp, said subsequent-frame of camera data characterized by a subsequent-time-stamp different from the first-time-stamp;

a second-input configured to communicate with a radar mounted on the host-vehicle, said radar configured to output a second-frame of radar map data indicative of objects present in a second-field-of-view that overlaps the first-field-of-view, said second-frame of radar map data characterized by a second-time-stamp temporally located between the first-time-stamp and the subsequent-time-stamp; and

a processor in communication with the camera and the radar, said processor configured to:

synthesize an interpolated-frame from the first-frame and the subsequent-frame using motion flow analysis responsive to determining that the both the first-time-stamp and the subsequent-time-stamp differ from the second-time-stamp by greater than a time-threshold, said interpolated-frame characterized by an interpolated-time-stamp that corresponds to the second-time-stamp, such that the interpolated-frame is temporally synchronized with the radar map data of the second-frame;

fuse the interpolated-frame with the second-frame to provide a fused-frame of data characterized by the interpolated-time-stamp,

determine a three-dimensional model of an environment surrounding the host-vehicle based on the fused-frame; and

operate the host-vehicle in accordance with the three-dimensional model of the environment.

6. The controller-circuit of claim 5 , wherein the objects present in the first-field-of-view are in a travel path of the host-vehicle.

7. The controller-circuit of claim 5 , wherein the objects present in the second-field-of-view are in a travel path of the host-vehicle.

8. The controller-circuit of claim 5 , wherein the processor is further configured to:

synthesize the interpolated-frame from the first-frame responsive to determining that the first-time-stamp and the subsequent-time-stamp differ from the second-time-stamp by less than five milliseconds, wherein the interpolated-frame is the same as the first-frame.

9. A method of operating a sensor data fusion system for a host-vehicle with multiple sensors, said method comprising:

receiving a first-frame of camera data and a subsequent-frame of camera data from a camera mounted on the host-vehicle, said first-frame of camera data and said subsequent-frame of camera data indicative of objects present in a first-field-of-view, said first-frame of camera data characterized by a first-time-stamp, said subsequent-frame of camera data characterized by a subsequent-time-stamp different from the first-time-stamp;

receiving a second-frame of radar map data from a radar mounted on the host-vehicle, said second-frame of radar map data indicative of objects present in a second-field-of-view that overlaps the first-field-of-view, said second-frame of radar map data characterized by a second-time-stamp temporally located between the first-time-stamp and the subsequent-time-stamp;

synthesizing an interpolated-frame from the first-frame and the subsequent-frame using motion flow analysis responsive to determining that both the first-time-stamp and the subsequent-time-stamp differ from the second-time-stamp by greater than a time-threshold, said interpolated-frame characterized by an interpolated-time-stamp that corresponds to the second-time-stamp, such that the interpolated-frame is temporally synchronized with the radar map data of the second-frame;

fusing the interpolated-frame with the second-frame to provide a fused-frame of data characterized by the interpolated-time-stamp;

determining a three-dimensional model of an environment based on the fused-frame; and

operating the host-vehicle in accordance with the three-dimensional model of the environment.

10. The method of claim 9 , wherein the objects present in the first-field-of-view are in a travel path of the host-vehicle.

11. The method of claim 9 , wherein the objects present in the second-field-of-view are in a travel path of the host-vehicle.

12. The method of claim 9 , further comprising:

synthesizing the interpolated-frame from the first-frame responsive to determining that the first-time-stamp and the subsequent-time-stamp differ from the second-time-stamp by less than five milliseconds, wherein the interpolated-frame is the same as the first-frame.

13. A sensor data fusion system for a host-vehicle with multiple sensors, said system comprising:

a radar mounted on the host-vehicle, said radar configured to output a first-frame of radar map data and a subsequent-frame of radar map data indicative of objects present in a first-field-of-view, said first-frame of radar map data characterized by a first-time-stamp, said subsequent-frame of radar map data characterized by a subsequent-time-stamp different from the first-time-stamp;

a camera mounted on the host-vehicle, said camera configured to output a second-frame of camera data indicative of objects present in a second-field-of-view that overlaps the first-field-of-view, said second-frame of camera data characterized by a second-time-stamp temporally located between the first-time-stamp and the subsequent-time-stamp; and

a controller-circuit in communication with the radar and the camera, said controller-circuit configured to:

synthesize an interpolated-frame from the first-frame and the subsequent-frame using motion flow analysis responsive to determining that both the first-time-stamp and the subsequent-time-stamp differ from the second-time-stamp by greater than a time-threshold, said interpolated-frame characterized by an interpolated-time-stamp that corresponds to the second-time-stamp, such that the interpolated-frame is temporally synchronized with the camera data of the second-frame;

fuse the interpolated-frame with the second-frame to provide a fused-frame of data characterized by the interpolated-time-stamp;

determine a three-dimensional model of an environment surrounding the host-vehicle based on the fused-frame; and

operate the host-vehicle in accordance with the three-dimensional model of the environment.

14. The system of claim 13 , wherein the objects present in the first-field-of-view are in a travel path of the host-vehicle.

15. The system of claim 13 , wherein the objects present in the second-field-of-view are in a travel path of the host-vehicle.

16. The system of claim 13 , wherein the controller-circuit is further configured to:

synthesize the interpolated-frame from the first-frame responsive to determining that the first-time-stamp and the subsequent-time-stamp differ from the second-time-stamp by less than five milliseconds, wherein the interpolated-frame is the same as the first-frame.

17. A controller-circuit for a sensor data fusion system for a host-vehicle with multiple sensors, said controller-circuit comprising:

a first-input configured to communicate with a radar mounted on the host-vehicle, said radar configured to output a first-frame of radar map data and a subsequent-frame of radar map data indicative of objects present in a first-field-of-view, said first-frame of radar map data characterized by a first-time-stamp, said subsequent-frame of radar map data characterized by a subsequent-time-stamp different from the first-time-stamp;

a second-input configured to communicate with a camera mounted on the host-vehicle, said camera configured to output a second-frame of camera data indicative of objects present in a second-field-of-view that overlaps the first-field-of-view, said second-frame of camera data characterized by a second-time-stamp temporally located between the first-time-stamp and the subsequent-time-stamp; and

a processor in communication with the camera and the radar, said processor configured to:

synthesize an interpolated-frame from the first-frame and the subsequent-frame using motion flow analysis responsive to determining that the both the first-time-stamp and the subsequent-time-stamp differ from the second-time-stamp by greater than a time-threshold, said interpolated-frame characterized by an interpolated-time-stamp that corresponds to the second-time-stamp, such that the interpolated-frame is temporally synchronized with the camera data of the second-frame;

fuse the interpolated-frame with the second-frame to provide a fused-frame of data characterized by the interpolated-time-stamp,

determine a three-dimensional model of an environment surrounding the host-vehicle based on the fused-frame; and

operate the host-vehicle in accordance with the three-dimensional model of the environment.

18. The controller-circuit of claim 17 , wherein the objects present in the first-field-of-view are in a travel path of the host-vehicle.

19. A method of operating a sensor data fusion system for a host-vehicle with multiple sensors, said method comprising:

receiving a first-frame of radar map data and a subsequent-frame of radar map data from a radar mounted on the host-vehicle, said first-frame of radar map data and said subsequent-frame of radar map data indicative of objects present in a first-field-of-view, said first-frame of radar map data characterized by a first-time-stamp, said subsequent-frame of radar map data characterized by a subsequent-time-stamp different from the first-time-stamp;

receiving a second-frame of camera data from a camera mounted on the host-vehicle, said second-frame of camera data indicative of objects present in a second-field-of-view that overlaps the first-field-of-view, said second-frame of camera data characterized by a second-time-stamp temporally located between the first-time-stamp and the subsequent-time-stamp;

synthesizing an interpolated-frame from the first-frame and the subsequent-frame using motion flow analysis responsive to determining that both the first-time-stamp and the subsequent-time-stamp differ from the second-time-stamp by greater than a time-threshold, said interpolated-frame characterized by an interpolated-time-stamp that corresponds to the second-time-stamp, such that the interpolated-frame is temporally synchronized with the camera data of the second-frame;

fusing the interpolated-frame with the second-frame to provide a fused-frame of data characterized by the interpolated-time-stamp;

determining a three-dimensional model of an environment based on the fused-frame; and

operating the host-vehicle in accordance with the three-dimensional model of the environment.

20. The method of claim 19 , further comprising:

synthesizing the interpolated-frame from the first-frame responsive to determining that the first-time-stamp and the subsequent-time-stamp differ from the second-time-stamp by less than five milliseconds, wherein the interpolated-frame is the same as the first-frame.