IP Library › Granted Patent US 11,592,559
Granted Patent B2
US 11,592,559 · App. 16/798,702 · Granted Feb 28, 2023

Vehicle sensor fusion

Inventors: David Michael Herman (Oak Park, MI); Aaron Lesky (Ypsilanti, MI); Ronald Beras (Warren, MI); Ashwin Arunmozhi (Canton, MI)
Assignee: FORD GLOBAL TECHNOLOGIES, LLC
G01S17/26G01S7/4808G01S7/4972G01S17/931G06N3/0454
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,592,559
App. No.
16/798,702
Filed
Feb 24, 2020
Granted
Feb 28, 2023
Kind
B2
Examiner
BUI, BRYAN
Art Unit
2857
USPC
702/189
Abstract

A computer, including a processor and a memory, the memory including instructions to be executed by the processor to obtain velocity lidar point cloud data acquired with a frequency modulated continuous wave (FMCW) lidar sensor, wherein the velocity lidar point cloud data includes a speed with which a data point is moving with respect to the FMCW lidar sensor, filter the velocity lidar point cloud data to select static velocity data points, wherein the static velocity data points are velocity data points each correspond to a point on a roadway around a vehicle. The instructions can include further instructions to determine FMCW lidar sensor accelerations in six degrees of freedom based on the static velocity lidar data points and determine FMCW lidar sensor rotations and translations in six degrees of freedom based on the FMCW lidar sensor accelerations in six degrees of freedom. The instructions can include further instructions to determine vehicle rotations and translations in six degrees of freedom based on inertial measurement unit (IMU) data, determine FMCW lidar sensor mis-alignment based on comparing the FMCW lidar sensor rotations and translations with the vehicle rotations and translations and align the FMCW lidar sensor based on the FMCW lidar sensor mis-alignment. The instructions can include further instructions to operate a vehicle based on the aligned FMCW lidar sensor.

Claims (37)

1. A computer, comprising a processor; and

a memory, the memory including instructions executable by the processor to:

obtain velocity lidar point cloud data acquired with a frequency modulated continuous wave (FMCW) lidar sensor, wherein the velocity lidar point cloud data includes a speed with which a data point is moving with respect to the FMCW lidar sensor;

filter the velocity lidar point cloud data to select static velocity data points, wherein the static velocity data points each correspond to a point on a roadway around a vehicle;

determine FMCW lidar sensor accelerations in six degrees of freedom based on the static velocity data points;

determine FMCW lidar sensor rotations and translations in six degrees of freedom based on the FMCW lidar sensor accelerations in six degrees of freedom;

determine vehicle rotations and translations in six degrees of freedom based on inertial measurement unit (IMU) data;

determine FMCW lidar sensor mis-alignment based on differences between the FMCW lidar sensor rotations and translations and the vehicle rotations and translations;

align the FMCW lidar sensor based on the FMCW lidar sensor mis-alignment; and

operate the vehicle based on the aligned FMCW lidar sensor.

2. The computer of claim 1 , the instructions including further instructions to acquire velocity lidar point cloud data based on a doppler shift.

3. The computer of claim 1 , the instructions including further instructions to acquire the velocity lidar point cloud data based on acquiring two data points from a location in an environment around the FMCW lidar sensor at different times to measure acceleration.

4. The computer of claim 1 , the instructions including further instructions to filter the velocity lidar point cloud data to select the static velocity data points based on providing the velocity lidar point cloud data to a first neural network.

5. The computer of claim 4 , the instructions including further instructions to select the static velocity data points by filtering out dynamic data points using a second neural network where the dynamic data points are data points acquired from objects that are moving with respect to a background.

6. The computer of claim 5 , the instructions including further instructions to filter out dynamic data points by performing spatial filtering of data points on or near a moving object with the second neural network.

7. The computer of claim 1 , the instructions including further instructions to filter the static velocity data points based on processing the static velocity data points with a Savitzky-Golay filter to smooth the static velocity data, wherein the Savitzky-Golay filter convolves the velocity data points with a polynomial function.

8. The computer of claim 1 , the instructions including further instructions to determine the FMCW lidar sensor rotations and translations in six degrees of freedom, and the vehicle rotations and translations based on transfer functions determined based on finite element analysis simulations of FMCW lidar sensor and simulated vehicle rotations and translations under varying load inputs with varying vibration modes.

9. The computer of claim 8 , the instructions including further instructions to implement the transfer functions with a second neural network.

10. The computer of claim 1 , wherein the transfer functions account for differences in varying stiffness of vehicle structures upon which the FMCW lidar sensor and the IMU are mounted.

11. The computer of claim 1 , the instructions including further instructions to operate the vehicle by determining a vehicle path based on point cloud data obtained from the aligned FMCW lidar sensor.

12. The computer of claim 11 , the instructions including further instructions to operate the vehicle by controlling vehicle powertrain, vehicle steering and vehicle brakes to cause the vehicle to travel along the vehicle path.

13. A method, comprising:

obtaining velocity lidar point cloud data acquired with a frequency modulated continuous wave (FMCW) lidar sensor wherein the velocity lidar point cloud data includes a speed with which a data point is moving with respect to the FMCW lidar sensor;

filtering the velocity lidar point cloud data to select static velocity data points, wherein the static velocity data points each correspond to a point on a roadway around a vehicle;

determining FMCW lidar sensor accelerations in six degrees of freedom based on the static velocity data points;

determining FMCW lidar sensor rotations and translations in six degrees of freedom based on the FMCW lidar sensor accelerations in six degrees of freedom;

determining vehicle rotations and translations in six degrees of freedom based on inertial measurement unit (IMU) data;

determining FMCW lidar sensor mis-alignment based on differences between the FMCW lidar sensor rotations and translations and the vehicle rotations and translations;

aligning the FMCW lidar sensor based on the FMCW lidar sensor mis-alignment; and

operating the vehicle based on the aligned FMCW lidar sensor.

14. The method of claim 13 , further comprising acquiring velocity lidar point cloud data based on a doppler shift.

15. The method of claim 13 , further comprising acquiring the velocity lidar point cloud data based on acquiring two data points from a location in an environment around the FMCW lidar sensor at different times to measure acceleration.

16. The method of claim 13 , further comprising filtering the velocity lidar point cloud data to select the static velocity data points based on providing the velocity lidar point cloud data to a first neural network.

17. The method of claim 16 , further comprising selecting the static velocity data points by filtering out dynamic data points using a second neural network where the dynamic data points are data points acquired from objects that are moving with respect to a background.

18. The method of claim 17 , further comprising filtering out dynamic data points by performing spatial filtering of data points on or near a moving object with the second neural network.

19. The method of claim 13 , further comprising filtering the static velocity data points based on processing the static velocity data points with a Savitzky-Golay filter to smooth the static velocity data, wherein the Savitzky-Golay filter convolves the velocity data points with a polynomial function.

20. The method of claim 13 , further comprising determining the FMCW lidar sensor rotations and translations in six degrees of freedom, and the vehicle rotations and translations based on transfer functions determined based on finite element analysis simulations of FMCW lidar sensor and simulated vehicle rotations and translations under varying load inputs with varying vibration modes.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 24, 2020
From: HERMAN, DAVID MICHAEL; LESKY, AARON; BERAS, RONALD; ARUNMOZHI, ASHWIN
To: FORD GLOBAL TECHNOLOGIES, LLC
Reel/Frame 051900/0520 →
Continuity (1)
Related Publication 20210263154A1 · Aug 26, 2021
Cited By (2)
US 1,149,730 US 12,650,501