IP Library Granted Patent US 10,992,844
Granted Patent B2
US 10,992,844 · App. 16/983,919 · Granted Apr 27, 2021

Video imaging and inertial sensor data coupling

Inventors: Joseph S. Virzi (Fremont, CA); Newfel Harrat (Burlingame, CA); Rakesh Ranjan (Foster City, CA)
Assignee: Nauto, Inc.
H04N5/12G06T1/20H04N5/05H04N5/06H04N5/144
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,992,844
App. No.
16/983,919
Granted
Apr 27, 2021
Kind
B2
Abstract

Introduced here are synchronization modules designed to integrate streams of data acquired from multiple sources in a precise, repeatable manner. To improve the usability of data acquired from multiple sources, a synchronization module can tightly couple streams of data received from these sources so that the data is temporally aligned. For example, a synchronization module may tightly couple motion data generated by a motion sensor with location data generated by a location sensor and image data generated by an image sensor in a manner that lessens the incurrence of software overhead.

Claims (61)

1. A device comprising:

a camera configured to

capture a series of images,

transfer the images to a processor, and

generate a first synchronization signal that includes a series of discrete values,

wherein each discrete value in the series of discrete values is associated with a transfer of a corresponding image in the series of images from the camera to the processor;

an inertial sampling circuit configured to

receive as input the first synchronization signal generated by the camera, and

generate as output a series of motion measurements indicative of motion of the device;

a Global Positioning System (GPS) receiver configured to generate a second synchronization signal; and

a microcontroller configured to

compute a phase difference between the first synchronization signal and the second synchronization signal,

append the phase difference to the motion measurements to create a series of fused data values, and

transfer the fused data values to the processor.

2. The device of claim 1 , wherein the images in the series form at least a part of a video file.

3. The device of claim 1 , wherein the GPS receiver is further configured to:

generate a series of location values indicative of location of the device, and

produce a series of speed values by establishing, for each location value in the series of location values, a speed at which the device is moving.

4. The device of claim 3 , wherein the processor is configured to receive the series of images from the camera, the series of fused data values from the microcontroller, and the series of speed values from the GPS receiver, and

wherein the processor is configured to associate each image in the series of images with (1) at least one of the motion measurements in the series of fused data values and (2) at least one of the speed values.

5. The device of claim 1 , wherein the inertial sampling circuit comprises an accelerometer, a gyroscope, a magnetometer, or any combination thereof.

6. The device of claim 1 , wherein the inertial sampling circuit includes an accelerometer and a gyroscope, and

wherein the series of motion measurements includes a first set of measurements generated by the accelerometer and a second set of measurements generated by the gyroscope.

7. The device of claim 1 , wherein each motion measurement in the series of motion measurements is generated by the inertial sampling circuit in response to receiving a corresponding one of the discrete values in the first synchronization signal from the camera.

8. The device of claim 7 , wherein the inertial sampling circuit is configured to generate the motion measurements at a frequency of at least 100 kilohertz (kHz).

9. The device of claim 1 , wherein the microcontroller is configured to determine the phase difference between the first synchronization signal and the second synchronization signal using a counter having a resolution of at least 32 nanoseconds (ns).

10. A synchronization module comprising:

a processor; and

a microcontroller configured to:

receive a synchronization signal that includes a series of discrete values from a camera,

wherein each discrete value in the series of discrete values is associated with a transfer of an image in a series of images generated by the camera,

receive a pulse-per-second (PPS) signal from a Global Positioning System (GPS) receiver,

compute a phase difference between the synchronization signal and the PPS signal,

append the phase difference to a series of motion measurements generated by a motion sensor configured to use the synchronization signal as input, and

transfer the motion measurements to the processor.

11. The synchronization module of claim 10 , wherein the motion sensor comprises an accelerometer.

12. The synchronization module of claim 10 , wherein the motion sensor is one of multiple motion sensors configured to use the synchronization signal as input; and

wherein the series of motion measurements include a first set of measurements generated by the one of the motion sensors, and a second set of measurements generated by another one of the motion sensors.

13. The synchronization module of claim 12 , wherein the microcontroller is configured to receive the first and second sets of measurements via different input pins.

14. The synchronization module of claim 10 , wherein the processor is configured to:

receive the series of images generated by the camera,

receive a series of speed values generated by the GPS receiver,

receive the series of motion measurements from the microcontroller, and

associate each image in the series of images with (1) at least one of the motion measurements and (2) at least one of the speed values.

15. The synchronization module of claim 14 , wherein the synchronization module is configured to store the series of images, the series of motion measurements, and the series of speed values in a profile.

16. The synchronization module of claim 15 , wherein the profile is associated with an individual captured in at least one image in the series of images.

17. A method for coupling streams of data to be analyzed in real time, the method comprising:

receiving, by a microcontroller, a synchronization signal that includes a series of discrete values,

wherein each discrete value in the series of discrete values is associated with a transfer of an image in a series of images generated by a camera;

receiving, by the microcontroller, a pulse-per-second (PPS) signal generated by a Global Positioning System (GPS) receiver;

computing, by the microcontroller, a phase difference between the synchronization signal and the PPS signal;

receiving, by the microcontroller, a series of motion measurements generated by an inertial sampling circuit configured to use the synchronization signal as input;

creating, by the microcontroller, a series of fused data values by appending the phase difference to the series of motion measurements;

transferring, by the microcontroller, the fused data values to a processor.

18. The method of claim 17 , wherein each fused data value in the series of fused data values includes a 16-bit signed integer indicative of a corresponding one of the motion measurements, and a 12-bit marker indicative of a time at which the synchronization signal was registered by the inertial sampling circuit.

19. The method of claim 17 , wherein the microcontroller employs a counter having a resolution of at least 32 nanoseconds (ns) to compute the phase difference between the synchronization signal and the PPS signal.

20. The method of claim 17 , further comprising:

receiving, by the processor, the series of images generated by the camera;

receiving, by the processor, a series of speed values generated by the GPS receiver;

receiving, by the processor, the series of fused data values from the microcontroller; and

associating, by the processor, each image in the series of images with (1) at least one of the fused data values and (2) at least one of the speed values.

Assignments (4)
RELEASE OF SECURITY INTEREST Recorded Jun 19, 2026
From: ORIX GROWTH CAPITAL, LLC
To: NAUTO, INC.
Reel/Frame 075016/0824 →
SECURITY INTEREST Recorded Aug 8, 2025
From: NAUTO, INC.
To: ORIX GROWTH CAPITAL, LLC, AS AGENT
Reel/Frame 071976/0818 →
SECURITY INTEREST Recorded Nov 10, 2022
From: NAUTO, INC.
To: SILICON VALLEY BANK
Reel/Frame 061722/0392 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 30, 2021
From: VIRZI, JOSEPH S.; HARRAT, NEWFEL; RANJAN, RAKESH
To: NAUTO, INC.
Reel/Frame 055775/0019 →
Continuity (2)
Provisional Application 62882314 · Aug 2, 2019
Related Publication 20210037167A1 · Feb 4, 2021