Granted Patent B2

US 12,322,129 · App. 17/592,500 · Granted Jun 3, 2025

Inventors: Sudipta Narayan Sinha (Kirkland, WA); Ondrej Miksik (Zurich, CH); Joseph Michael Degol (Seattle, WA); Tien Do (Minneapolis, MN)

Assignee: Microsoft Technology Licensing, LLC

G06T7/70G06V10/7747G06V10/7784G06V20/64G06T2207/10016G06T2207/20081

Abstract

Claims (29)

1. A method for camera localization within a scene, the method comprising:

receiving, at a processor, an image of a scene, the image captured by the camera;

inputting the image to a machine learning model which has been trained for the scene to detect a plurality of 3D scene landmarks, wherein the 3D scene landmarks are specified in a map of the scene, and receiving as output from the machine learning model a plurality of predictions, each prediction comprising a predicted 3D bearing vector originating at the camera and pointing towards a predicted 3D location of one of the 3D scene landmarks; and

using the predictions, computing an estimate of a position and orientation of the camera in the map of the scene.

2. The method of claim 1 wherein the camera is a mobile camera moving in the scene.

3. The method of claim 1 wherein the machine learning model has been trained by denoting for a 2D image point corresponding to a landmark, an associated unitized bearing vector in the camera coordinate as a point on a unit sphere.

4. The method of claim 1 wherein the method further comprises inputting the image to a second machine learning model, the second machine learning model configured to output predicted 2D locations in the image.

5. The method of claim 4 comprising determining pairs of the predicted 2D locations and the predicted 3D bearing vectors which relate to a same one of the 3D scene landmarks, and discarding the predicted 3D bearing vectors from the determined pairs prior to using the predictions to compute the estimate of the position and orientation of the camera.

6. The method of claim 1 wherein a number of the specified 3D landmarks is less than 250.

7. The method of claim 1 comprising automatically selecting the specified 3D landmarks in advance by selecting a specified number of 3D points in the scene which are the most robust, repeatedly detectable, unique in appearance, and generalizable.

8. The method of claim 7 comprising computing a saliency score to determine which are the most robust, repeatedly detectable, unique in appearance and generalizable 3D points in the scene.

9. The method of claim 1 wherein the machine learning model comprises a convolutional neural network which has been trained using supervised learning to predict the predicted 3D bearing vectors.

10. The method of claim 4 wherein the second machine learning model comprises a convolutional neural network and the input image is input into the convolutional neural network which computes feature maps, and wherein the feature maps are used to predict one heatmap for each 3D landmark, and a transposed convolution layer is used for performing up-sampling and generating a second set of heatmaps for each 3D landmark.

11. The method of claim 1 wherein the machine learning model computes the 3D bearing vectors using a plurality of multi-layered perceptrons, one multi-layered perceptron per 3D landmark.

12. The method of claim 11 wherein the machine learning model has been trained using supervised learning.

13. The method of claim 1 wherein the machine learning model has been trained using training data comprising observed videos of the scene and exhibiting appearance, illumination and geometric changes.

14. The method of claim 13 wherein the observed videos are captured over a time frame during which one or more objects in the scene move or during which ambient light in the scene changes.

15. The method of claim 13 wherein the training data is augmented with synthetic videos computed by applying homography and/or intensity changes to the observed videos of the scene.

16. The method of claim 1 comprising automatically selecting the specified 3D scene landmarks in advance by carrying out object recognition on images or videos depicting the scene, and using rules to place the 3D scene landmarks on recognized objects which are likely to be static.

17. The method of claim 1 further comprising:

removing one of the specified 3D scene landmarks in response to disappearance of a corresponding pattern in the scene or adding a new specified 3D scene landmark in response to a new salient object appearing in the scene; and

retraining the machine learning model.

18. The method of claim 1 wherein the machine learning model comprises a first neural network trained using data from multiple scenes and which feeds into additional neural network layers which are scene-specific.

19. The method of claim 1 wherein the scene is a large scene divided into a plurality of overlapping subregions, and where the method comprises using a different plurality of 3D scene landmarks for each subregion and using a different machine learning model for each subregion, using a classifier to classify the image into one of the subregions and inputting the image to the machine learning model associated with the subregion.

20. An apparatus for camera localization within a scene, the apparatus comprising a processor executing instructions from a non-transitory machine readable storage medium, the instructions being for:

receiving an image of a scene, the image captured by the camera;

executing first and second machine learning models, each of the machine learning models receiving the image as input and each of the machine learning models having been trained for the scene to detect a plurality of 3D scene landmarks, wherein the 3D scene landmarks are specified in a map of the scene;

receiving as output from the first machine learning model a first plurality of predictions and from the second machine learning model a second plurality of predictions, wherein each of the predictions of the first plurality of predictions and the second plurality of predictions indicates a location of one of the 3D scene landmarks; and

using the first plurality of predictions and the second plurality of predictions, computing an estimate of a position and orientation of the camera in the map of the scene.

Assignments (1)

Continuity (2)

References Cited (108)