IP Library Granted Patent US 9,361,650
Granted Patent B2
US 9,361,650 · App. 14/057,435 · Granted Jun 7, 2016

Synchronization of vehicle sensor information

Inventors: Todd Binion (Bloomington, IL); Joe Harr (Bloomington, IL); Brian Fields (Normal, IL); Steven Cielocha (Bloomington, IL); Steve J. Balbach (Bloomington, IL)
Assignee: STATE FARM MUTUAL AUTOMOBILE INSURANCE COMPANY
G06Q40/08G06T13/80G07C5/008G07C5/085G07C5/0858G07C5/0866G07C5/0891G07C5/12G08G1/162G08G1/166B60R2021/0027B60R2300/302B60W2050/0089G01S13/02G01S15/02G01S17/02G08G1/01
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,361,650
App. No.
14/057,435
Filed
Oct 18, 2013
Granted
Jun 7, 2016
Kind
B2
Art Unit
3668
USPC
701/33.4
Abstract

A method includes receiving and storing sensor data including a first plurality of data points indicative of a plurality of respective states of the environment external to a vehicle at a plurality of respective times, operational data including a second plurality of data points indicative of a plurality of respective states of an operational parameter of the vehicle at a plurality of respective times, and synchronization data. The method also includes generating a virtual model of an event involving the vehicle using the stored data, at least by generating a first visual representation of the plurality of respective states of the external environment, generating a second visual representation of the plurality of respective states of the operational parameter, and using the synchronization data to cause the first visual representation to be displayed simultaneously with, and in a time-aligned manner with, the second visual representation.

Claims (78)

1. A method in an electronic processing system comprising a memory and one or more processors, the method comprising:

receiving, at the electronic processing system, sensor data representing information collected by a sensor (i) located on or in a vehicle and (ii) configured to sense an environment external to the vehicle, wherein receiving sensor data includes receiving a first plurality of data points indicative of a plurality of respective states of the environment external to the vehicle at a plurality of respective times;

receiving, at the electronic processing system, first operational data (i) generated by a first unit arranged within the vehicle and (ii) indicative of a first operational parameter of the vehicle, wherein receiving first operational data includes receiving a second plurality of data points indicative of a plurality of respective states of the first operational parameter at a plurality of respective times;

receiving, at the electronic processing system, external data that (i) was received by the vehicle from a mobile communication device of a pedestrian external to the vehicle via a wireless link and (ii) is indicative of objects sensed by the mobile communication device, wherein receiving external data includes receiving a third plurality of data points indicative of an additional plurality of respective states of the environment external to the vehicle at a plurality of respective times;

receiving, at the electronic processing system, synchronization data indicating a temporal relationship between the first plurality of data points, the second plurality of data points, and the third plurality of data points;

storing the received sensor data, the received first operational data, the received external data, and the received synchronization data in the memory; and

generating, via the one or more processors, a virtual model of an event involving the vehicle using the stored sensor data, the stored first operational data, the stored external data, and the stored synchronization data, wherein generating a virtual model includes

generating, using the first plurality of data points, a first visual representation of the plurality of respective states of the environment external to the vehicle,

generating, using the second plurality of data points, a second visual representation of the plurality of respective states of the first operational parameter,

generating, using the third plurality of data points, a third visual representation of the additional plurality of respective states of the environment external to the vehicle, and

using the synchronization data to cause the first visual representation to be displayed simultaneously with the second visual representation and the third visual representation such that the first visual representation is time-aligned with the second visual representation and the third visual representation.

2. The method of claim 1 , wherein receiving synchronization data includes receiving data indicating (i) which of the first plurality of data points corresponds to which of the second plurality of data points and (ii) which of the first plurality of data points corresponds to which of the third plurality of data points.

3. The method of claim 1 , wherein receiving synchronization data includes receiving:

a first plurality of time stamps each corresponding to a respective one of the first plurality of data points;

a second plurality of time stamps each corresponding to a respective one of the second plurality of data points; and

a third plurality of time stamps each corresponding to a respective one of the third plurality of data points.

4. The method of claim 1 , wherein receiving sensor data representing information collected by a sensor includes receiving sensor data representing information collected by:

a camera device;

a radar device;

a lidar device; or

a sonar device.

5. The method of claim 1 , wherein receiving first operational data indicative of a first operational parameter of the vehicle includes receiving first operational data indicative of:

velocity of the vehicle;

acceleration of the vehicle;

braking of the vehicle; or

steering of the vehicle.

6. The method of claim 1 , wherein receiving first operational data indicative of a first operational parameter of the vehicle includes receiving first operational data indicative of motion of the vehicle with respect to six degrees of freedom.

7. The method of claim 1 , wherein receiving first operational data indicative of a first operational parameter of the vehicle includes receiving first operational data indicative of status of an automated driving system of the vehicle.

8. The method of claim 1 , wherein receiving first operational data indicative of a first operational parameter of the vehicle includes receiving diagnostics information associated with the vehicle.

9. The method of claim 1 , wherein:

receiving sensor data representing information collected by a sensor includes receiving video data representing information collected by a video camera device;

generating a first visual representation of the plurality of respective states of the environment external to the vehicle includes generating a video corresponding to the video data;

generating a second visual representation of the plurality of respective states of the first operational parameter includes generating a visual indication of status of the first operational parameter of the vehicle; and

using the synchronization data to cause the first visual representation to be displayed simultaneously with the second visual representation includes using the synchronization data to cause the video to be displayed simultaneously with the visual indication of status of the first operational parameter of the vehicle.

10. The method of claim 1 , wherein generating a virtual model of an event involving the vehicle includes generating a virtual model of an accident involving the vehicle.

11. The method of claim 10 , further comprising at least one of:

determining fault for the accident based on the virtual model of the accident involving the vehicle; or

determining damages resulting from the accident based on the virtual model of the accident involving the vehicle.

12. The method of claim 1 , wherein:

the method further comprises

receiving, at the electronic processing system, second operational data (i) generated by a second unit arranged within the vehicle and (ii) indicative of a second operational parameter of the vehicle, wherein receiving second operational data includes receiving a fourth plurality of data points indicative of a plurality of respective states of the second operational parameter at a plurality of respective times, and

storing the received second operational data in the memory;

receiving synchronization data includes receiving synchronization data indicating a temporal relationship between the first plurality of data points, the second plurality of data points, the third plurality of data points, and the fourth plurality of data points;

generating a virtual model of an event involving the vehicle further includes generating a fourth visual representation of the plurality of respective states of the second operational parameter; and

using the synchronization data to cause the first visual representation to be displayed simultaneously with the second visual representation and the third visual representation further includes using the synchronization data to cause the first visual representation to be displayed simultaneously with the second visual representation, the third visual representation and the fourth visual representation, such that the first visual representation is time-aligned with the second visual representation, the third visual representation, and the fourth visual representation.

13. The method of claim 12 , wherein:

receiving sensor data representing information collected by a sensor includes receiving video data representing information collected by a video camera device;

generating a first visual representation of the plurality of respective states of the environment external to the vehicle includes generating a video corresponding to the video data;

generating a second visual representation of the plurality of respective states of the first operational parameter includes generating a visual indication of status of the first operational parameter of the vehicle;

generating a fourth visual representation of the plurality of respective states of the second operational parameter includes generating a visual indication of status of the second operational parameter of the vehicle; and

using the synchronization data to cause the first visual representation to be displayed simultaneously with the second visual representation, the third visual representation and the fourth visual representation includes using the synchronization data to cause the video to be displayed simultaneously with both the visual indication of status of the first operational parameter and the visual indication of status of the second operational parameter.

14. The method of claim 1 , wherein:

receiving sensor data includes receiving sensor data at a server remote from the vehicle via one of (i) a wireless communication network, or (ii) a portable memory device; and

receiving first operational data includes receiving first operational data at the server remote from the vehicle via one of (i) the wireless communication network, or (ii) the portable memory device.

15. A non-transitory, computer-readable medium storing instructions that, when executed by one or more processors of an electronic processing system, cause the one or more processors to retrieve from a memory of the electronic processing system sensor data representing information collected by a sensor (i) located on or in a vehicle and (ii) configured to sense an environment external to the vehicle, wherein the sensor data includes a first plurality of data points indicative of a plurality of respective states of the environment external to the vehicle at a plurality of respective times;

retrieve from the memory first operational data (i) generated by a first unit arranged within the vehicle and (ii) indicative of a first operational parameter of the vehicle, wherein the first operational data includes a second plurality of data points indicative of a plurality of respective states of the first operational parameter at a plurality of respective times;

retrieve from the memory external data that (i) was received by the vehicle from a source mobile communication device of a pedestrian external to the vehicle via a wireless link and (ii) is indicative of objects sensed by the mobile communication device, wherein the external data includes a third plurality of data points indicative of an additional plurality of respective states of the environment external to the vehicle at a plurality of respective times;

retrieve from the memory synchronization data indicating a temporal relationship between the first plurality of data points, the second plurality of data points, and the third plurality of data points; and

generate a virtual model of an event involving the vehicle using the retrieved first plurality of data points, the retrieved second plurality of data points, the retrieved third plurality of data points and the retrieved synchronization data, at least in part by

generating, using the first plurality of data points, a first visual representation of the plurality of respective states of the environment external to the vehicle,

generating, using the second plurality of data points, a second visual representation of the plurality of respective states of the first operational parameter,

generating, using the third plurality of data points, a third visual representation of the additional plurality of respective states of the environment external to the vehicle, and

using the synchronization data to cause the first visual representation to be displayed simultaneously with the second visual representation and the third visual representation such that the first visual representation is time-aligned with the second visual representation and the third visual representation.

16. The non-transitory, computer-readable medium of claim 15 , wherein the synchronization data includes at least one of:

data indicating (i) which of the first plurality of data points corresponds to which of the second plurality of data points and (ii) which of the first plurality of data points corresponds to which of the third plurality of data points; or

(i) a first plurality of time stamps each corresponding to a respective one of the first plurality of data points, (ii) a second plurality of time stamps each corresponding to a respective one of the second plurality of data points, and (iii) a third plurality of time stamps each corresponding to a respective one of the third plurality of data points.

17. The non-transitory, computer-readable medium of claim 15 , wherein the sensor data represents information collected by

a camera device,

a radar device,

a lidar device, or

a sonar device; and

the first operational data is indicative of

velocity of the vehicle,

acceleration of the vehicle,

braking of the vehicle,

steering of the vehicle,

diagnostics information associated with the vehicle, or

status of an automated driving system of the vehicle.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 22, 2013
From: BINION, TODD; HARR, JOE; FIELDS, BRIAN; CIELOCHA, STEVEN; BALBACH, STEVE J.
To: STATE FARM MUTUAL AUTOMOBILE INSURANCE COMPANY
Reel/Frame 031452/0889 →
Continuity (1)
Related Publication 20150112545A1 · Apr 23, 2015