IP Library › Granted Patent US 12,687,402

Granted Patent B2

US 12,687,402 · App. 18/656,970 · Granted Jul 21, 2026

Localization of user(s) in environment(s)

Inventors: Fatemeh Mirzaei (Scotts Valley, CA); Nicholas J. Karels, Jr. (Louisville, KY); Charlie Meredith (Louisville, KY)

Assignee: GOODMAPS INC.

G01C21/3476 G01C21/3667 G06V10/443 G06V20/52

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 12,687,402
App. No.: 18/656,970
Granted: Jul 21, 2026
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

Implementations described herein relate to various techniques for localization of user(s) in environment(s). In particular, processor(s) can utilize a multi-scan technique, a semantic segmentation technique, or a combination of these techniques. In utilizing the multi-scan technique, the processor(s) can initially determine a subset of candidate maps that are predicted to correspond to an environment of a user, and from a superset of candidate maps. Further, the processor(s) can obtain vision data that captures the environment, process the vision data to determine a narrower subset of candidate maps and to determine a given map, from the narrower subset of candidate maps corresponding to the environment. In utilizing the semantic segmentation technique, the processor(s) can additionally, or alternatively, determine semantic properties of the environment. The semantic properties can be utilized to initially constrain the subset of candidate maps and/or utilized in subsequently determining the given map.

Claims (73)

1 . A method implemented by one or more processors, the method comprising:

obtaining vision data that captures an environment of a user of a client device, the vision data being generated by one or more vision components of the client device of the user;

processing, using one or more machine learning (ML) models, the vision data to determine one or more semantic properties of the environment of the user;

determining, based on one or more of the semantic properties of the environment of the user, a subset of candidate maps, from a superset of candidate maps, that are predicted to correspond to the environment of the user;

determining, based on processing the vision data that captures the environment of the user or based on processing additional vision data that captures the environment of the user, and from among the subset of candidate maps, a given map corresponding to the environment of the user;

causing the given map corresponding to the environment of the user to be utilized as an operational map;

detecting one or more errors with respect to the given map being utilized as the operational map;

in response to detecting the one or more errors with respect to the given map being utilized as the operational map:

determining a further subset of candidate maps, from the superset of candidate maps, that are adjacent to the given map;

determining, from among the further subset of candidate maps, a further given map corresponding to the environment of the user; and

causing the further given map corresponding to the environment of the user to be utilized as the operational map.

2 . The method of claim 1 , wherein the one or more ML models comprise one or more of: an edge detection ML model, an object detection ML model, an object classification ML model, a door detection ML model, a lux detection ML model, a contrastive language-image pre-training (CLIP) model, or a visual language model (VLM).

3 . The method of claim 2 , wherein the one or more semantic properties are dependent on the one or more ML models that are utilized to process the vision data.

4 . The method of claim 3 , wherein the one or more semantic properties include one or more of:

edges detected in the environment of the user;

objects detected in the environment of the user;

sizes and/or orientations of the objects detected in the environment of the user;

classifications of the objects detected in the environment of the user;

one or more doors detected in the environment of the user;

sizes and/or orientations of the one or more doors detected in the environment of the user;

a quantity of the one or more doors detected in the environment of the user; or

a lux of one or more lights in the environment of the user.

5 . The method of claim 1 , wherein the one or more semantic properties are assigned to corresponding points of interest included in each map, included in the superset of candidate maps, when each of the maps are generated.

6 . The method of claim 5 , wherein each of the maps, included in the superset of candidate maps, are generated prior to obtaining the vision data that captures the environment of the user.

7 . The method of claim 1 , wherein determining the given map corresponding to the environment of the user based on processing the vision data that captures the environment of the user or based on processing additional vision data that captures the environment of the user, and from among the subset of candidate maps, comprises:

processing, using a machine learning (ML) model, the vision data to generate output, wherein the ML model is a feature extraction model, and wherein the output generated based on processing the vision data using the ML model comprises: global features of the environment of the user, local features of the environment of the user, and corresponding keypoint detection scores associated with the global features of the environment of the user and the local features of the environment of the user;

determining, based on at least a portion of the output corresponding to the global features of the environment of the user and the corresponding keypoint detection scores associated with the global features of the environment of the user, and from among the subset of candidate maps, a narrower subset of candidate maps; and

determining, based on at least an additional portion of the output corresponding to the local features of the environment of the user and the corresponding keypoint detection scores associated with the local features of the environment of the user, and from among the narrower subset of candidate maps, the given map.

8 . The method of claim 7 , wherein determining the narrower subset of candidate maps based on at least the portion of the output, and from among the subset of candidate maps, comprises:

processing, using a rough feature matching algorithm or model, the global features of the environment of the user and the corresponding keypoint detection scores associated with the global features of the environment of the user to determine, from among the subset of candidate maps, the narrower subset of candidate maps.

9 . The method of claim 8 , wherein determining the given map based on at least the additional portion of the output, and from among the narrower subset of candidate maps, comprises:

processing, using a fine feature matching algorithm or model, the local features of the environment of the user and the corresponding keypoint detection scores associated with the local features of the environment of the user to determine, from among the narrower subset of candidate maps, the given map.

10 . The method of claim 9 , wherein a quantity of the local features that are processed using the fine feature matching algorithm or model is greater than a quantity of the global features that are processed using the rough feature matching algorithm or model.

11 . The method of claim 9 , wherein the processing of the local features of the environment of the user and the corresponding keypoint detection scores associated with the local features of the environment of the user using the fine feature matching algorithm or model is more computationally intensive than the processing of the global features of the environment of the user and the corresponding keypoint detection scores associated with the global features of the environment of the user.

12 . The method of claim 9 , wherein the rough feature matching algorithm or model is one of: a k-nearest neighbors (kNN) algorithm, or one or more distance functions, and wherein the fine feature matching algorithm or model is one of: a structure from motion (SfM) model, or a visual simultaneous localization and mapping (visual SLAM) algorithm.

13 . The method of claim 1 , wherein causing the given map corresponding to the environment of the user to be utilized as the operational map comprises:

causing the operational map to be utilized by the client device of the user to provide the user with navigational directions from a current location of the user in the environment and to a given point of interest in the environment that is specified by the user via the client device of the user.

14 . The method of claim 1 , wherein causing the given map corresponding to the environment of the user to be utilized as the operational map comprises:

causing the operational map to be utilized by the client device of the user to provide the user with information related to a plurality of points of interest in the environment.

15 . The method of claim 1 , wherein the environment of the user corresponds to an indoor environment of a building, and wherein each of the subset of candidate maps are associated with: the building, a corresponding floor of the building, a portion of the corresponding floor of the building, or an outdoor environment that surrounds the building.

16 . The method of claim 1 , wherein determining the further given map corresponding to the environment of the user and from among the further subset of candidate maps is based on processing further additional vision data that captures the environment of the user.

17 . A system comprising:

at least one processor; and

memory storing instructions that, when executed, cause the at least one processor to be operable to:

obtain vision data that captures an environment of a user of a client device, the vision data being generated by one or more vision components of the client device of the user;

process, using one or more machine learning (ML) models, the vision data to determine one or more semantic properties of the environment of the user;

determine, based on one or more of the semantic properties of the environment of the user, a subset of candidate maps, from a superset of candidate maps, that are predicted to correspond to the environment of the user;

determine, based on processing the vision data that captures the environment of the user or based on processing additional vision data that captures the environment of the user, and from among the subset of candidate maps, a given map corresponding to the environment of the user, wherein the instructions to determine the given map corresponding to the environment of the user based on processing the vision data that captures the environment of the user or based on processing additional vision data that captures the environment of the user, and from among the subset of candidate maps, comprise instructions to:

process, using a machine learning (ML) model, the vision data to generate output, wherein the ML model is a feature extraction model, and wherein the output generated based on processing the vision data using the ML model comprises: global features of the environment of the user, local features of the environment of the user, and corresponding keypoint detection scores associated with the global features of the environment of the user and the local features of the environment of the user;

determine, based on at least a portion of the output corresponding to the global features of the environment of the user and the corresponding keypoint detection scores associated with the global features of the environment of the user, and from among the subset of candidate maps, a narrower subset of candidate maps; and

determine, based on at least an additional portion of the output corresponding to the local features of the environment of the user and the corresponding keypoint detection scores associated with the local features of the environment of the user, and from among the narrower subset of candidate maps, the given map; and

cause the given map corresponding to the environment of the user to be utilized as an operational map.

18 . The system of claim 17 , wherein the at least one processor is further operable to:

detect one or more errors with respect to the given map being utilized as the operational map;

in response to detecting the one or more errors with respect to the given map being utilized as the operational map:

determine a further subset of candidate maps, from the superset of candidate maps, that are adjacent to the given map;

determine, from among the further subset of candidate maps, a further given map corresponding to the environment of the user; and

cause the further given map corresponding to the environment of the user to be utilized as the operational map.

19 . A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to be operable to perform operations, the operations comprising: