IP Library › Granted Patent US 12670649
Granted Patent B2
US 12670649 · App. 18/421,241 · Granted Jun 30, 2026

Object scale utilizing away-facing images

Inventors: Gurunandan Krishnan Gorumkonda (Kirkland, WA); Rui Yu (State College, PA); Yicheng Wu (Boston, MA); Jian Wang (Pittsburgh, PA); Sizhuo Ma (Madison, WI)
G06T15/00G06T7/60G06T7/75G06T7/85G06T19/006G06T2207/30201G06T2207/30244
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12670649
App. No.
18/421,241
Granted
Jun 30, 2026
Kind
B2
Abstract

Methods, systems, mobile devices, and non-transitory computer-readable mediums for determining a scale of an object with a mobile device. The mobile device includes at least one object-facing camera (e.g., a monocular camera) and an away-facing stereo camera system. Information captured using the away-facing stereo camera system is used to estimate relative pose information for determining the scale of the object in images captured by the object-facing camera. The proposed system leverages the scene images surrounding the mobile device to resolve scale ambiguity and calculate the absolute scale.

Claims (80)

1 . A method for determining a scale of an object with a mobile device, the method comprising:

capturing an image of the object using an object-facing camera of the mobile device in each of a first position and a second position;

capturing a first image at the first position of the mobile device with an away-facing stereo camera system;

capturing a second image at the second position of the mobile device with the away-facing stereo camera system;

estimating an away-facing relative pose of the away-facing stereo camera system from the first and second images;

obtaining an object-facing relative pose of the object-facing camera using the away-facing relative pose; and

determining the scale of the object from the object-facing relative pose.

2 . The method of claim 1 , further comprising:

streaming video from the object-facing camera to a display of the mobile device;

developing an AR overlay including an item that is scaled using the scale of the object to overlay the object; and

presenting the AR overlay on the display with the scaled item on or near the object.

3 . The method of claim 1 , wherein capturing the image of the object, the first image, and the second image comprises:

capturing concurrently the image of the object in the first position of the mobile device and the first image; and

capturing concurrently the image of the object in the second position of the mobile device and the second image.

4 . The method of claim 1 , further comprising:

calibrating the object-facing camera of the mobile device with the away-facing stereo camera system.

5 . The method of claim 4 , wherein the away-facing stereo camera system includes a first away-facing camera and a second away-facing camera and wherein the calibrating comprises:

calibrating intrinsic parameters and distortion coefficients of each of the object-facing camera, the first away-facing camera, and the second away-facing camera;

calibrating an away-facing stereo camera system transformation between the first away-facing camera and the second away-facing camera; and

calibrating an object-facing to away-facing stereo camera system transformation between the object-facing camera and one of the first away-facing camera or the second away-facing camera.

6 . The method of claim 1 , further comprising:

correcting for motion of the object between the image of the object captured with the mobile device in the first position and the image of the object captured with the mobile device in the second position.

7 . The method of claim 1 , wherein the first image and the second image are two-dimensional (2D) and wherein the estimating the away-facing relative pose comprises:

reconstructing a true-scale three-dimensional (3D) point cloud for the first position of the mobile device using the first image and the second image;

matching keypoints between the first image and the second image to establish correspondence between the keypoints in the second image and the 3D point cloud for the first position to generate a 2D to 3D correspondence; and

estimating the away-facing relative pose based on the 2D to 3D correspondence.

8 . The method of claim 7 , wherein the estimating the away-facing relative pose based on the 2D to 3D correspondence comprises:

solving a perspective-n-point problem based on the 2D to 3D correspondence.

9 . The method of claim 1 , wherein the object is a face and wherein the obtaining the object-facing relative pose of the object-facing camera and the determining the scale of the object comprises:

transferring transformations between cameras of the away-facing stereo camera system to the object-facing camera with calibrated extrinsics between the away-facing stereo camera system and the object-facing camera, the transformations including a true-scale camera transformation;

detecting predefined face landmarks from the image of the object captured by the mobile device in the first position and the image of the object captured by the mobile device in the second position; and

reconstructing a true-scale three-dimensional (3D) face from the predefined face landmarks using the true-scale camera transformation.

10 . A mobile device for determining a scale of an object, the mobile device comprising:

an object-facing camera;

an away-facing stereo camera system; and

a processor coupled to the object-facing camera and the away-facing stereo camera system, the processor configured to:

capture an image of the object using the object-facing camera in each of a first position and a second position of the mobile device;

capture a first image at the first position of the mobile device with the away-facing stereo camera system;

capture a second image at the second position of the mobile device with the away-facing stereo camera system;

estimate an away-facing relative pose of the away-facing stereo camera system from the first and second images;

obtain an object-facing relative pose of the object-facing camera using the away-facing relative pose; and

determine the scale of the object from the object-facing relative pose.

11 . The mobile device of claim 10 , further comprising:

a display;

wherein the processor is coupled to the display and is further configured to:

stream video from the object-facing camera to the display;

develop an AR overlay including an item that is scaled using the scale of the object to overlay the object; and

present the AR overlay on the display with the scaled item on or near the object.

12 . The mobile device of claim 10 , wherein to capture the image of the object, the first image, and the second image, the processor is configured to:

capture concurrently the image of the object in the first position of the mobile device and the first image; and

capture concurrently the image of the object in the second position of the mobile device and the second image.

13 . The mobile device of claim 10 , wherein the processor is further configured to:

calibrate the object-facing camera of the mobile device with the away-facing stereo camera system.

14 . The mobile device of claim 13 , wherein the away-facing stereo camera system includes a first away-facing camera and a second away-facing camera and wherein to calibrate the object-facing camera of the mobile device with the away-facing stereo camera system the processor is configured to:

calibrate intrinsic parameters and distortion coefficients of each of the object-facing camera, the first away-facing camera, and the second away-facing camera;

calibrate an away-facing stereo camera system transformation between the first away-facing camera and the second away-facing camera; and

calibrate an object-facing to away-facing stereo camera system transformation between the object-facing camera and one of the first away-facing camera or the second away-facing camera.

15 . The mobile device of claim 10 , wherein the processor is further configured to:

correct for motion of the object between the image of the object captured with the mobile device in the first position and the image of the object captured with the mobile device in the second position.

16 . The mobile device of claim 10 , wherein the first image and the second image are two-dimensional (2D) and wherein to estimate the away-facing relative pose the processor is configured to:

reconstruct a true-scale three-dimensional (3D) point cloud for the first position of the mobile device using the first image and the second image;

match keypoints between the first image and the second image to establish correspondence between the keypoints in the second image and the 3D point cloud for the first position to generate a 2D to 3D correspondence; and

estimate the away-facing relative pose based on the 2D to 3D correspondence.

17 . The mobile device of claim 16 , wherein to estimate the away-facing relative pose based on the 2D to 3D correspondence the processor is configured to:

solve a perspective-n-point problem based on the 2D to 3D correspondence.

18 . The mobile device of claim 10 , wherein the object is a face and wherein to obtain the object-facing relative pose of the object-facing camera and to determine the scale of the object the processor is configured to:

transfer transformations between cameras of the away-facing stereo camera system to the object-facing camera with calibrated extrinsics between the away-facing stereo camera system and the object-facing camera, the transformations including a true-scale camera transformation;

detect predefined face landmarks from the image of the object captured by the mobile device in the first position and the image of the object captured by the mobile device in the second position; and

reconstruct a true-scale three-dimensional (3D) face from the predefined face landmarks using the true-scale camera transformation.

19 . A non-transitory computer-readable medium including instructions for determining a scale of an object with a mobile device, the instructions, when executed by a processor of the mobile device, configure the mobile device to:

capture an image of the object using an object-facing camera of the mobile device in each of a first position and a second position;

capture a first image at the first position of the mobile device with an away-facing stereo camera system;

capture a second image at the second position of the mobile device with the away-facing stereo camera system;

estimate an away-facing relative pose of the away-facing stereo camera system from the first and second images;

obtain an object-facing relative pose of the object-facing camera using the away-facing relative pose; and

determine the scale of the object from the object-facing relative pose.

20 . The non-transitory computer-readable medium of claim 19 , wherein the instructions, when executed by the processor of the mobile device, further configure the mobile device to:

stream video from the object-facing camera to a display of the mobile device;

develop an AR overlay including an item that is scaled using the scale of the object to overlay the object; and

present the AR overlay on the display with the scaled item on or near the object.