IP Library Granted Patent US 9,568,611
Granted Patent B2
US 9,568,611 · App. 14/830,873 · Granted Feb 14, 2017

Detecting objects obstructing a driver's view of a road

Inventors: Eric Cosatto (Red Bank, NJ); Hans Peter Graf (Lincroft, NJ)
Assignee: NEC CORPORATION
G01S19/51B60K35/00G01C21/3629G01C21/3697G01S11/12G01S13/931G01S17/89G01S17/936G01S19/14G01S19/48G01S19/49B60K2350/1084B60K2350/962B60K2350/965G01S13/867
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,568,611
App. No.
14/830,873
Granted
Feb 14, 2017
Kind
B2
Abstract

A system and method for a motorized land vehicle that detects objects obstructing a driver's view of an active road, includes an inertial measurement unit-enabled global position system (GPS/IMU) subsystem for obtaining global position system (GPS) position and heading data of a land vehicle operated by the driver as the vehicle travels along a road, a street map subsystem for obtaining street map data of the GPS position of the vehicle using the GPS position and heading data as the vehicle travels along the road, and a three-dimensional (3D) object detector subsystem for detecting objects ahead of the vehicle and determining a 3D position and 3D size data of each of the detected objects ahead of the vehicle. The street map subsystem merges the street map data, the GPS position and heading data of the vehicle and the 3D position data and 3D size data of the detected objects, to create real-time two-dimensional (2D) top-view map representation of a traffic scene ahead of the vehicle. The street map subsystems finds active roads ahead of the vehicle in the traffic scene, and finds each active road segment of the active roads ahead of the vehicle that is obstructed by one of the detected objects. A driver alert subsystem notifies a driver of the vehicle of each of the active road segments that is obstructed by one of the detected objects.

Claims (36)

1. A system for a motorized land vehicle that detects objects obstructing a driver's view of an active road, the system comprising:

an inertial measurement unit-enabled global position system (GPS/IMU) subsystem for obtaining global position system (GPS) position and heading data of a land vehicle operated by the driver as the vehicle travels along a road;

a street map subsystem for obtaining street map data of the GPS position of the vehicle using the GPS position and heading data as the vehicle travels along the road;

a three-dimensional (3D) object detector subsystem for detecting objects ahead of the vehicle and determining a 3D position data of each of the detected objects ahead of the vehicle; and

a driver alert subsystem;

wherein the street map subsystem merges the street map data, the GPS position and heading data of the vehicle and the 3D position data of the detected objects, to create real-time two-dimensional (2D) top-view representation of a traffic scene ahead of the vehicle, finds active roads ahead of the vehicle in the traffic scene, and finds each active road segment of the active roads ahead of the vehicle that is obstructed by one of the detected objects;

wherein the driver alert subsystem notifies the driver of the vehicle of each of the active road segments that is obstructed by one of the detected objects;

wherein the 3D object detector subsystem further determines a 3D size of each of the detected objects ahead of the vehicle; and

wherein the 3D object detector subsystem includes at least one video camera and a neural network, the at least one video camera providing the 3D position data of each of the detected objects from motion data in a video stream of the traffic scene obtained therewith and the neural network providing the type and scale of the object from the video stream, wherein the 3D size of each of the objects is determined from the type and scale of the object combined with approximate standard sizes appropriate of the recognized type of object.

2. A system for a motorized land vehicle that detects objects obstructing a driver's view of an active road, the system comprising:

an inertial measurement unit-enabled global position system (GPS/IMU) subsystem for obtaining global position system (GPS) position and heading data of a land vehicle operated by the driver as the vehicle travels along a road;

a street map subsystem for obtaining street map data of the GPS position of the vehicle using the GPS position and heading data as the vehicle travels along the road;

a three-dimensional (3D) object detector subsystem for detecting objects ahead of the vehicle and determining a 3D position data of each of the detected objects ahead of the vehicle; and

a driver alert subsystem;

wherein the street map subsystem merges the street map data, the GPS position and heading data of the vehicle and the 3D position data of the detected objects, to create real-time two-dimensional (2D) top-view representation of a traffic scene ahead of the vehicle, finds active roads ahead of the vehicle in the traffic scene, and finds each active road segment of the active roads ahead of the vehicle that is obstructed by one of the detected objects;

wherein the driver alert subsystem notifies the driver of the vehicle of each of the active road segments that is obstructed by one of the detected objects;

wherein the 3D object detector subsystem further determines a 3D size of each of the detected objects ahead of the vehicle; and

wherein the 3D object detector subsystem includes radar, at least one video camera, and a neural network, the radar providing two-dimensional (2D) position and speed data of each of the detected objects, the at least one video camera providing a video stream of the traffic scene obtained therewith and the neural network providing the type and scale of the object from the video stream, wherein the 3D size of each of the objects is determined from the type and scale of the object combined with approximate standard sizes appropriate of the recognized type of object.

3. The system of claim 1 , wherein the 3D object detector subsystem includes lidar providing 3D position and size of each of the detected objects.

4. The system of claim 3 , wherein the 3D object detector subsystem further includes at least one video camera, and a neural network, providing a video stream of the traffic scene obtained therewith and the neural network providing the type of the object from the video stream, wherein the type of the object is used to verify and validate lidar information.

5. The system of claim 1 , wherein the GPS/IMU subsystem and the street map subsystem comprise a GPS map unit.

6. The system of claim 1 , wherein the GPS/IMU comprises a GPS chip and the street map data is provided by an Internet provider.

7. The system of claim 1 , wherein the merging of the street map data, the GPS position and heading data of the vehicle and the 3D position data of the detected objects includes converting street map coordinates into relative distance from the vehicle's center of mass, making the 3D positions of the objects relative to the GPS/IMU subsystem, and projecting along a z axis a bounding box of the 3D object onto the street map.

8. The system of claim 1 , wherein the finding of the active roads ahead of the vehicle in the traffic scene is determined with the 3D position, speed and the heading of the vehicle, wherein based on the vehicle's position, the closest road on the street map is selected, wherein based on the vehicle's speed, a proportional active distance ahead of the selected road is calculated and a corresponding segment of the road is deemed active and further searched for intersecting roads, wherein for each intersection found, the roads leading to the intersection become active.

9. The system of claim 1 , wherein the finding of each active road segment of the active roads ahead of the vehicle that is obstructed by one of the detected objects comprises tracing rays from the drivers' eyes that pass by comers of each of the detected objects, and wherein segments formed by intersecting the rays with the active roads are hidden road segments obstructed by one of the detected objects.

10. The system of claim 1 , wherein the driver alert subsystem generates an audio-locating sound at a virtual position of the obstructing vehicle thereby alerting the driver.

11. The system of claim 1 , wherein the driver alert subsystem overlays a 3D bounding box of each of the obstructing objects onto a live video feed of the road ahead.

12. A system for a motorized land vehicle that detects objects obstructing a driver's view of an active road, the system comprising:

an inertial measurement unit-enabled global position system (GPS/IMU) subsystem for obtaining global position system (GPS) position and heading data of a land vehicle operated by the driver as the vehicle travels along a road;

a street map subsystem for obtaining street map data of the GPS position of the vehicle using the GPS position and heading data as the vehicle travels along the road;

a three-dimensional (3D) object detector subsystem for detecting objects ahead of the vehicle and determining a 3D position data of each of the detected objects ahead of the vehicle; and

a driver alert subsystem;

wherein the street map subsystem merges the street map data, the GPS position and heading data of the vehicle and the 3D position data of the detected objects, to create real-time two-dimensional (2D) top-view representation of a traffic scene ahead of the vehicle, finds active roads ahead of the vehicle in the traffic scene, and finds each active road segment of the active roads ahead of the vehicle that is obstructed by one of the detected objects;

wherein the driver alert subsystem notifies the driver of the vehicle of each of the active road segments that is obstructed by one of the detected objects;

wherein the driver alert subsystem generates a computer generated voice alerting the driver to the obstructing vehicle.

13. The system of claim 1 , wherein the driver alert subsystem generates a semi-transparent hologram directly on the windshield such that it overlays seamlessly with the live view through the windshield.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 20, 2016
From: NEC LABORATORIES AMERICA, INC.
To: NEC CORPORATION
Reel/Frame 040687/0340 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 20, 2015
From: COSATTO, ERIC; GRAF, HANS PETER
To: NEC LABORATORIES AMERICA, INC.
Reel/Frame 036377/0779 →
Continuity (2)
Provisional Application 62039523 · Aug 20, 2014
Related Publication 20160054452A1 · Feb 25, 2016