IP Library Granted Patent US 10,672,186
Granted Patent B2
US 10,672,186 · App. 15/740,934 · Granted Jun 2, 2020

Method in constructing a model of a scenery and device therefor

Inventors: Yubin Kuang (Lund, SE); Pau Gargallo Piracés (Barcelona, ES); Jan Erik Solem (Bjärred, SE); Johan Gyllenspetz (West Hollywood, CA)
Assignee: MAPILLARY AB
G06T17/05G06T7/579G06T7/80G06T2207/10016G06T2207/20072G06T2207/30244
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,672,186
App. No.
15/740,934
Granted
Jun 2, 2020
Kind
B2
Abstract

A method in constructing a model of a scenery comprises: receiving ( 200 ) a plurality of images depicting the scenery, wherein each image is acquired by a camera ( 104 ) from a position at a rotational angle; creating ( 202 ) a plurality of sub-models, wherein each sub-model is created by: selecting ( 400 ) a basis image; determining ( 402 ) images related to the basis image; and determining ( 404 ) the sub-model based on the basis image and the related images, wherein the sub-model defines relative positions and rotational angles of the camera ( 104 ) for the basis image and the related images and the sub-model further defines three-dimensional structures of objects depicted in the plurality of images; and separately merging ( 204 ) the defined relative positions and rotational angles of cameras ( 104 ) for the plurality of sub-models.

Claims (28)

1. A method associated with constructing a model of a scenery, the method comprising:

receiving a plurality of images depicting the scenery, wherein each image is acquired by a camera from a position aiming at the scenery in a rotational angle;

creating a plurality of sub-models, wherein each of the plurality of sub-models is created by:

selecting a basis image among the plurality of images;

determining related images among the plurality of images, wherein each related image is related to the basis image based on a depicted part of the scenery in the related image and a depicted part of the scenery in the basis image being at least partly overlapping; and

determining the sub-model based on the basis image and the related images, wherein the sub-model defines relative positions and rotational angles of the camera for the basis image and the related images and the sub-model further defines three-dimensional structures of objects depicted in the plurality of images; and

separately merging the defined relative positions and rotational angles of the camera for the plurality of sub-models, wherein the plurality of sub-models are created using each image in the plurality of images as a basis image in a sub-model, a basis image of a first sub-model is a related image of a second sub-model, and the merging comprises calculating a solution of the relative position and rotational angle of the camera for the plurality of images, the calculation penalizing a deviation of a relative position or rotational angle of a camera determined by the sub-model as compared to the solution of the relative position and rotational angle of the camera.

2. The method according to claim 1 , further comprising receiving additional information for at least some of the plurality of images, the additional information providing an indication of the respective positions or rotational angles of the camera for the at least some of the plurality of images, and wherein the merging is further based on the additional information and comprises calculating absolute position and rotational angle of the camera for the plurality of images.

3. The method according to claim 2 , wherein the merging comprises weighting the plurality of sub-models and the additional information based on an accuracy of the additional information.

4. The method according to claim 1 , wherein the scenery comprises a pre-defined geographical area.

5. The method according to claim 4 , further comprising receiving further images, wherein each further image is acquired by the camera from a position within the pre-defined geographical area.

6. The method according to claim 5 , further comprising updating the model by updating the sub-models taking the further images into account and merging the defined relative positions and rotational angles of cameras for the plurality of updated sub-models.

7. The method according to claim 5 , wherein the updating is triggered when a predetermined number of further images are received.

8. The method according to claim 1 , wherein the plurality of images are represented as a graph, wherein each image is represented by a node in the graph and relations between images are represented by edges between nodes, and wherein the determining of related images comprises starting in a basis node of the graph representing the basis image and selecting nodes that are connected by an edge to the basis node, the selected nodes representing the related images.

9. A device associated with constructing a model of a scenery, the device comprising a processor configured to:

receive a plurality of images depicting the scenery, wherein each image is acquired by a camera from a position aiming at the scenery in a rotational angle;

create a plurality of sub-models, wherein each of the plurality of sub-models is created by:

selecting a basis image among the plurality of images;

determining related images among the plurality of images, wherein each related image is related to the basis image based on a depicted part of the scenery in the related image and a depicted part of the scenery in the basis image being at least partly overlapping; and

determining the sub-model based on the basis image and the related images, wherein the sub-model defines relative positions and rotational angles of the camera for the basis image and the related images and the sub-model further defines three-dimensional structures of objects depicted in the plurality of images; and

separately merge the defined relative positions and rotational angles of the camera for the plurality of sub-models, wherein the plurality of sub-models are created using each image in the plurality of images as a basis image in a sub-model, a basis image of a first sub-model is a related image of a second sub-model, and the merging comprises calculating a solution of the relative position and rotational angle of the camera for the plurality of images, the calculation penalizing a deviation of a relative position or rotational angle of a camera determined by the sub-model as compared to the solution of the relative position and rotational angle of the camera.

10. A non-transitory computer-readable medium comprising computer-readable instructions such that when executed on a processing unit the computer-readable instructions cause the processing unit to:

receive a plurality of images depicting the scenery, wherein each image is acquired by a camera from a position aiming at the scenery in a rotational angle;

create a plurality of sub-models, wherein each of the plurality of sub-models is created by:

selecting a basis image among the plurality of images;

determining related images among the plurality of images, wherein each related image is related to the basis image based on a depicted part of the scenery in the related image and a depicted part of the scenery in the basis image being at least partly overlapping; and

determining the sub-model based on the basis image and the related images, wherein the sub-model defines relative positions and rotational angles of the camera for the basis image and the related images and the sub-model further defines three-dimensional structures of objects depicted in the plurality of images; and

separately merge the defined relative positions and rotational angles of the camera for the plurality of sub-models, wherein the plurality of sub-models are created using each image in the plurality of images as a basis image in a sub-model, a basis image of a first sub-model is a related image of a second sub-model, and the merging comprises calculating a solution of the relative position and rotational angle of the camera for the plurality of images, the calculation penalizing a deviation of a relative position or rotational angle of a camera determined by the sub-model as compared to the solution of the relative position and rotational angle of the camera.

Assignments (3)
CHANGE OF NAME Recorded Nov 19, 2021
From: FACEBOOK, INC.
To: META PLATFORMS, INC.
Reel/Frame 058214/0351 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 11, 2020
From: MAPILLARY AB
To: FACEBOOK, INC.
Reel/Frame 054621/0274 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 21, 2019
From: KUANG, YUBIN; PIRACÉS, PAU GARGELLO; SOLEM, JAN ERIK; GYLLENSPETZ, JOHAN
To: MAPILLARY AB
Reel/Frame 049552/0328 →