IP Library Granted Patent US 11,810,322
Granted Patent B2
US 11,810,322 · App. 17/225,396 · Granted Nov 7, 2023

Camera pose estimation techniques

Inventors: Chenzhe Qian (San Diego, CA); Yi Luo (San Diego, CA); Ke Xu (San Diego, CA); Mingdong Wang (San Diego, CA); Xiaodi Hou (San Diego, CA)
Assignee: TUSIMPLE, INC.
G06T7/74G06T2207/30244G06T2207/30256
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,810,322
App. No.
17/225,396
Granted
Nov 7, 2023
Kind
B2
Abstract

Techniques are described for estimating pose of a camera located on a vehicle. An exemplary method of estimating camera pose includes obtaining, from a camera located on a vehicle, an image including a lane marker on a road on which the vehicle is driven, and estimating a pose of the camera such that the pose of the camera provides a best match according to a criterion between a first position of the lane marker determined from the image and a second position of the lane marker determined from a stored map of the road.

Claims (54)

1. A method of estimating camera pose, comprising:

obtaining, from a camera located on a vehicle, an image comprising a lane marker on a road on which the vehicle is driven; and

estimating a pose of the camera such that the pose of the camera provides a best match according to a criterion between a first position of the lane marker determined from the image and a second position of the lane marker determined from a stored map of the road,

wherein the first position corresponds to pixel locations associated with a corner of the lane marker,

wherein the second position corresponds to a three-dimensional (3D) world coordinates of the corner of the lane marker,

wherein the second position of the lane marker is determined by:

obtaining, from the stored map and based on a location of the vehicle, a first set of one or more lane markers that are located within a pre-determined distance from the vehicle;

obtaining a second set of one or more lane markers from the first set of one or more lane markers based on a direction in which the vehicle is driven;

obtaining a third set of one or more lane markers from the second set of one or more lane markers based on a pre-determined field of view (FOV) of the camera; and

obtaining the second position of the lane marker from the third set of one or more lane markers;

wherein the best match according to the criterion is determined by minimizing a function of a combination of a cost of misalignment term and of a cost of constraint term,

wherein the cost of misalignment term is determined by minimizing a distance from the 3D world coordinates of the corner of the lane marker to the pixel locations associated with the corner of the lane marker, and

wherein the cost of constraint term is determined by minimizing a difference between a first estimated camera pose at a first time when the image is obtained and a second estimated camera pose from a second time, wherein the second time precedes in time the first time.

2. The method of claim 1 , wherein the distance is minimized by minimizing a sum of squared distance between the pixel locations associated with the corner of the lane marker and the 3D world coordinates of the corner of the lane marker.

3. The method of claim 1 , further comprising:

generating a binary image from the image obtained from the camera; and

generating a gray-scale image from the binary image, wherein the gray-scale image comprises pixels with corresponding values, wherein a value of each pixel is a second function of a second distance between a pixel location in the gray-scale image and the first position of the corner of lane marker in the gray-scale image.

4. The method of claim 1 , wherein the third set of one or more lane markers excludes one or more lane markers determined to be obstructed by one or more objects.

5. The method of claim 1 , wherein the cost of constraint term is another function of a first set of rotation values of the camera associated with the first time, a second set of translation values of the camera associated with the first time, a second difference between the first set of rotation values and a third set of rotation values of the camera associated with the second time, and a third difference between the second set of translation values and a fourth set of translation values of the camera associated with the second time.

6. A system comprising:

a processor; and

a memory that stores instructions executable by the processor to:

obtain, from a camera located on a vehicle, an image comprising a lane marker on a road on which the vehicle is driven; and

estimate a pose of the camera such that the pose of the camera provides a best match according to a criterion between a first position of the lane marker determined from the image and a second position of the lane marker determined from a stored map of the road,

wherein the first position corresponds to pixel locations associated with a corner of the lane marker,

wherein the second position corresponds to a three-dimensional (3D) world coordinates of the corner of the lane marker,

wherein the second position of the lane marker is determined by the processor configured to:

obtain, from the stored map and based on a location of the vehicle, a first set of one or more lane markers that are located within a pre-determined distance from the vehicle;

obtain a second set of one or more lane markers from the first set of one or more lane markers based on a direction in which the vehicle is driven;

obtain a third set of one or more lane markers from the second set of one or more lane markers based on a pre-determined field of view (FOV) of the camera; and

obtain the second position of the lane marker from the third set of one or more lane markers;

wherein the best match according to the criterion is determined by minimizing a function of a combination of a cost of misalignment term and of a cost of constraint term,

wherein the cost of misalignment term is determined by minimizing a distance from the 3D world coordinates of the corner of the lane marker to the pixel locations associated with the corner of the lane marker, and

wherein the cost of constraint term is determined by minimizing a difference between a first estimated camera pose at a first time when the image is obtained and a second estimated camera pose from a second time, wherein the second time precedes in time the first time.

7. The system of claim 6 , wherein the cost of constraint term represents a constraint that limits a search space.

8. The system of claim 6 , wherein the cost of constraint term is another function of a first set of rotation values of the camera associated with the first time, a second set of translation values of the camera associated with the first time, a second difference between the first set of rotation values and a third set of rotation values of the camera associated with the second time, and a third difference between the second set of translation values and a fourth set of translation values of the camera associated with the second time.

9. A non-transitory computer readable storage medium having code stored thereon, the code, when executed by a processor, causing the processor to implement a method, comprising:

obtaining, from a camera located on a vehicle, an image comprising a lane marker on a road on which the vehicle is driven; and

estimating a pose of the camera such that the pose of the camera provides a best match according to a criterion between a first position of the lane marker determined from the image and a second position of the lane marker determined from a stored map of the road,

wherein the first position corresponds to pixel locations associated with a corner of the lane marker,

wherein the second position corresponds to a three-dimensional (3D) world coordinates of the corner of the lane marker,

wherein the second position of the lane marker is determined by:

obtaining, from the stored map and based on a location of the vehicle, a first set of lane markers that are located within a pre-determined distance from the vehicle;

obtaining a second set of lane markers from the first set of lane markers based on a direction in which the vehicle is driven;

obtaining a third set of lane markers from the second set of lane markers based on a pre-determined field of view (FOV) of the camera; and

obtaining the second position of the lane marker from the third set of lane markers;

wherein the best match according to the criterion is determined by minimizing a function of a combination of a cost of misalignment term and of a cost of constraint term,

wherein the cost of misalignment term is determined by minimizing a distance from the 3D world coordinates of the corner of the lane marker to the pixel locations associated with the corner of the lane marker, and

wherein the cost of constraint term is determined by minimizing a difference between a first estimated camera pose at a first time when the image is obtained and a second estimated camera pose from a second time, wherein the second time precedes in time the first time.

10. The non-transitory computer readable storage medium of claim 9 , wherein the method further comprises:

generating a gray-scale image from the image, wherein the gray-scale image comprises pixels with corresponding values, wherein a value of each pixel is a second function of a second distance between a pixel location in the gray-scale image and the first position of lane marker in the gray-scale image.

11. The non-transitory computer readable storage medium of claim 9 , wherein the distance is minimized by minimizing a sum of squared distance between the pixel locations associated with the corner of the lane marker and the 3D world coordinates of the corner of the lane marker.

12. The non-transitory computer readable storage medium of claim 9 , wherein the third set of lane markers excludes one or more lane markers determined to be obstructed by one or more objects.

13. The non-transitory computer readable storage medium of claim 9 , wherein the cost of constraint term is another function of a first set of rotation values of the camera associated with the first time, a second set of translation values of the camera associated with the first time, a second difference between the first set of rotation values and a third set of rotation values of the camera associated with the second time, and a third difference between the second set of translation values and a fourth set of translation values of the camera associated with the second time.

Assignments (2)
CHANGE OF NAME Recorded Dec 3, 2025
From: TUSIMPLE, INC.
To: CREATEAI, INC.
Reel/Frame 073832/0553 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 8, 2021
From: QIAN, CHENZHE; LUO, YI; XU, KE; WANG, MINGDONG; HOU, XIAODI
To: TUSIMPLE, INC.
Reel/Frame 055864/0665 →
Continuity (2)
Provisional Application 63007895 · Apr 9, 2020
Related Publication 20210319584A1 · Oct 14, 2021