Granted Patent B2

US 10,614,581 · App. 15/880,364 · Granted Apr 7, 2020

Inventor: Nalin Senthamil (Santa Clara, CA)

Assignee: DAQRI, LLC

G06T7/337G06F16/29G06F16/583G06K9/00664G06N3/04G06N3/08G06N20/00G06T7/246G06T7/75G06T7/80G06T19/006G06T2207/10016G06T2207/20081G06T2207/30244

Abstract

Claims (76)

1. A server comprising:

a storage device configured to store a localization model that associates location and orientation data with each picture among a plurality of pictures; and

one or more hardware processor configured to perform operations comprising:

receiving a query for a location and an orientation of a device, the query including a first picture and a request to identify a geographic position and a pose of the device based on the first picture;

identifying a second picture, from the localization model, that corresponds to the first picture; and

determining the location and the orientation of the device based on the location and orientation data corresponding to the second picture, the location and orientation data indicating position, orientation, three-dimensional geometry, gyroscope measurement, and accelerometer measurement,

wherein the device comprises:

a camera configured to capture the picture and generate a plurality of video frames;

at least one inertial sensor configured to generate inertial data of the device; and

one or more hardware processors configured to perform operations comprising:

tracking at least one feature in the plurality of video frames;

synchronizing and aligning the plurality of video frames based on the inertial data;

computing a Visual Inertial Navigation (VIN) state of the device based on the synchronized plurality of video frames with the inertial data;

receiving VIN data and corresponding image data from a plurality of devices, the VIN data indicating VIN states and corresponding poses of the plurality of devices; and

training, using a neural network, the localization model based on the VIN data and corresponding image data from the plurality of devices, the image data including a plurality of images, the localization model correlating the VIN states and poses with each image among the plurality of images.

2. The server of claim 1 , wherein the operations further comprise:

receiving location and orientation data and corresponding pictures from a plurality of devices; and

forming the localization model based on the location and orientation data and the corresponding pictures from the plurality of devices.

3. The server of claim 2 , wherein the operations further comprise:

determining features in the plurality of pictures from the localization model; and

associating the features and relative positions of the features in each picture of the plurality of pictures with the corresponding location and orientation data.

4. The server of claim 3 , wherein the operations further comprise:

identifying features in the first picture;

comparing the features in the first picture with the features in the second picture; and

determining that the features in the first picture match the features in the second picture based on the comparison.

5. The server of claim 1 , wherein the location of the device identifies a geographic location of the device when the device generated the picture, and the orientation of the device identifies a pose of the device when the device generated the picture.

6. The server of claim 1 , wherein the operations at the device further comprise:

calibrating the camera off-line for at least one of focal length, principal point, pixel aspect ratio, or lens distortion;

calibrating the at least one inertial sensor for at least one of noise, scale, or bias; and

applying calibration information to the plurality of video frames and the inertial data.

7. The server of claim 1 , wherein the inertial data indicates an angular rate of change and a linear acceleration.

8. The server of claim 1 , wherein the features comprise predefined stationary interest points and line features.

9. A computer-implemented method comprising:

storing a localization model that associates location and orientation data with each picture among a plurality of pictures;

receiving a query for a location and an orientation of a device, the query including a first picture and a request to identify a geographic position and a pose of the device based on the first picture;

identifying a second picture, from the localization model, that corresponds to the first picture; and

determining the location and the orientation of the device based on the location and orientation data corresponding to the second picture, wherein the location and orientation data indicates position, orientation, three-dimensional geometry, gyroscope measurement, and accelerometer measurement,

wherein the device comprises:

a camera configured to capture the picture and generate a plurality of video frames;

at least one inertial sensor configured to generate inertial data of the device; and

one or more hardware processors configured to perform operations comprising:

tracking at least one feature in the plurality of video frames;

synchronizing and aligning the plurality of video frames based on the inertial data;

computing a Visual Inertial Navigation (VIN) state of the device based on the synchronized plurality of video frames with the inertial data;

receiving VIN data and corresponding image data from a plurality of devices, the VIN data indicating VIN states and corresponding poses of the plurality of devices; and

training, using a neural network, the localization model based on the VIN data and corresponding image data from the plurality of devices, the image data including a plurality of images, the localization model correlating the VIN states and poses with each image among the plurality of images.

10. The computer-implemented method of claim 9 , further comprising:

receiving location and orientation data and corresponding pictures from a plurality of devices; and

forming the localization model based on the location and orientation data and the corresponding pictures from the plurality of devices.

11. The computer-implemented method of claim 10 , further comprising:

determining features in the plurality of pictures from the localization model; and

associating the features and relative positions of the features in each picture of the plurality of pictures with the corresponding location and orientation data.

12. The computer-implemented method of claim of claim 11 , further comprising:

identifying features in the first picture;

comparing the features in the first picture with the features in the second picture; and

determining that the features in the first picture match the features in the second picture based on the comparison.

13. The computer-implemented method of claim of claim 9 , wherein the location of the device identifies a geographic location of the device when the device generated the picture, and the orientation of the device identifies a pose of the device when the device generated the picture.

14. The computer-implemented method of claim of claim 9 , further comprising:

calibrating the camera off-line for at least one of focal length, principal point, pixel aspect ratio, or lens distortion;

calibrating the at least one inertial sensor for at least one of noise, scale, or bias; and

applying calibration information to the plurality of video frames and the inertial data.

15. The computer-implemented method of claim of claim 9 , wherein the inertial data indicates an angular rate of change and a linear acceleration, and wherein the features comprise predefined stationary interest points and line features.

16. A non-transitory machine-readable storage medium, tangibly embodying a set of instructions that, when executed by at least one processor, causes the at least one processor to perform a set of operations comprising:

storing a localization model that associates location and orientation data with each picture among a plurality of pictures;

receiving a query for a location and an orientation of a device, the query including a first picture and a request to identify a geographic position and a pose of the device based on the first picture;

identifying a second picture, from the localization model, that corresponds to the first picture; and

determining the location and the orientation of the device based on the location and orientation data corresponding to the second picture, wherein the location and orientation data indicates position, orientation, three-dimensional geometry, gyroscope measurement, and accelerometer measurement,

wherein the device comprises:

a camera configured to capture the picture and generate a plurality of video frames;

at least one inertial sensor configured to generate inertial data of the device; and

one or more hardware processors configured to perform operations comprising:

tracking at least one feature in the plurality of video frames;

synchronizing and aligning the plurality of video frames based on the inertial data;

computing a Visual Inertial Navigation (VIN) state of the device based on the synchronized plurality of video frames with the inertial data;

receiving VIN data and corresponding image data from a plurality of devices, the VIN data indicating VIN states and corresponding poses of the plurality of devices; and

training, using a neural network, the localization model based on the VIN data and corresponding image data from the plurality of devices, the image data including a plurality of images, the localization model correlating the VIN states and poses with each image among the plurality of images.

Assignments (12)

Continuity (2)

Cited By (2)

US 12,450,823 US 12,475,590