IP Library Granted Patent US 11,669,995
Granted Patent B2
US 11,669,995 · App. 16/764,751 · Granted Jun 6, 2023

Orientation determination for mobile computing devices

Inventor: Daniel Joseph Filip (San Jose, CA)
Assignee: GOOGLE LLC
G06T7/73G06V10/12G06V10/24G06V10/60G06V10/82G06V20/50G06T2207/20081G06T2207/20084G06T2207/30244G06T2207/30252G06V2201/07G06V2201/08
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,669,995
App. No.
16/764,751
Granted
Jun 6, 2023
Kind
B2
Abstract

Methods, systems, devices, and tangible non-transitory computer readable media for determining orientation are provided. The disclosed technology can include capturing images of an environment visible in a field of view of the mobile computing device. Location data associated with a location of the mobile computing device can be received. Image data including information associated with the images can be generated. Based on the image data and one or more machine-learned models, features of one or more objects in the environment can be determined. Based on the location data and the features of the objects, respective orientations of the objects relative to the location of the mobile computing device can be determined. Furthermore, orientation data that includes a geographic orientation of the mobile computing device can be generated based on the respective (geographic) orientations of the objects.

Claims (53)

1. A computer-implemented method of determining orientation, the computer-implemented method comprising:

capturing, by a mobile computing device comprising one or more processors, one or more images of an environment visible in a field of view of the mobile computing device;

receiving, by the mobile computing device, location data associated with a location of the mobile computing device;

generating, by the mobile computing device, image data comprising information associated with the one or more images;

determining, by the mobile computing device, based at least in part on the image data and one or more machine-learned models, one or more object classifications of one or more objects in the environment, wherein the one or more machine-learned models are customized using user-specific data;

determining, by the mobile computing device, based at least in part on the location data and the one or more object classifications of the one or more objects, one or more respective orientations of the one or more objects relative to the location of the mobile computing device; and

generating, by the mobile computing device, orientation data comprising a geographic orientation of the mobile computing device based at least in part on the one or more respective orientations of the one or more objects.

2. The computer-implemented method of claim 1 , wherein the determining, by the mobile computing device, based at least in part on the location data and the one or more object classifications of the one or more objects, one or more respective orientations of the one or more objects relative to the location of the mobile computing device comprises:

determining, by the mobile computing device, based at least in part on the one or more object classifications, one or more sides of the one or more objects; and

determining, by the mobile computing device, the one or more respective orientations of the one or more sides of the one or more objects relative to the mobile computing device.

3. The computer-implemented method of claim 2 , wherein the one or more sides of one or more objects comprise one or more sides of one or more vehicles.

4. The computer-implemented method of claim 1 , wherein the one or more object classifications are associated with one or more vehicles, and wherein the one or more object classifications comprise one or more headlight features, one or more taillight features, one or more front windshield features, one or more back windshield features, one or more front grille features, one or more wheel features, one or more side window features, one or more door features, or one or more bumper features.

5. The computer-implemented method of claim 1 , wherein the determining, by the mobile computing device, based at least in part on the location data and the one or more object classifications of the one or more objects, one or more respective orientations of the one or more objects relative to the location of the mobile computing device comprises:

determining, by the mobile computing device, based at least in part on the one or more object classifications, one or more vehicle lights associated with the one or more objects; and

determining, by the mobile computing device, one or more respective orientations of the one or more vehicle lights relative to the location of the mobile computing device.

6. The computer-implemented method of claim 1 , wherein the determining, by the mobile computing device, based at least in part on the location data and the one or more object classifications of the one or more objects, one or more respective orientations of the one or more objects relative to the location of the mobile computing device comprises:

determining, by the mobile computing device, based at least in part on the one or more object classifications, a travel path associated with the environment; and

determining, by the mobile computing device, the orientation of the travel path relative to the location of the mobile computing device.

7. The computer-implemented method of claim 6 , wherein the one or more object classifications comprise one or more indications associated with a direction of travel along the travel path.

8. The computer-implemented method of claim 1 , wherein the determining, by the mobile computing device, based at least in part on the location data and the one or more object classifications of the one or more objects, one or more respective orientations of the one or more objects relative to the location of the mobile computing device comprises:

determining, by the mobile computing device, based at least in part on the one or more object classifications, one or more directions of travel of the one or more objects; and

determining, by the mobile computing device, the one or more orientations of the one or more objects relative to the location of the mobile computing device based at least in part on the one or more directions of travel of the one or more objects.

9. The computer-implemented method of claim 1 , wherein the determining, by the mobile computing device, based at least in part on the location data and the one or more object classifications of the one or more objects, one or more respective orientations of the one or more objects relative to the location of the mobile computing device comprises:

determining, by the mobile computing device, based at least in part on the one or more object classifications, one or more location identifiers associated with the one or more objects; and

determining, by the mobile computing device, the one or more respective orientations of the one or more location identifiers relative to the mobile computing device.

10. The computer-implemented method of claim 9 , wherein the orientation of the mobile computing device is based at least in part on one or more locations respectively associated with the one or more location identifiers.

11. The computer-implemented method of claim 9 , wherein the one or more location identifiers comprise one or more street numbers, one or more street names, or one or more signs associated with a geographic location.

12. The computer-implemented method of claim 1 , wherein the determining, by the mobile computing device, based at least in part on the location data and the one or more object classifications of the one or more objects, one or more respective orientations of the one or more objects relative to the location of the mobile computing device comprises:

determining, by the mobile computing device, based at least in part on one or more features of the one or more objects, an amount of sunlight at one or more portions of the environment; and

determining, by the mobile computing device, the one or more respective orientations of the one or more objects relative to the location of the mobile computing device based at least in part on the amount of sunlight at the one or more portions of the environment.

13. The computer-implemented method of claim 12 , wherein the one or more respective orientations of the one or more objects relative to the location of the mobile computing device are based at least in part on a time of day at which the amount of sunlight at the one or more portions of the environment was detected.

14. The computer-implemented method of claim 1 , wherein the one or more machine-learned models are trained based at least in part on training data comprising one or more images of one or more vehicles at one or more distances and one or more angles with respect to an image capture device that captured the one or more images of the one or more vehicles.

15. One or more tangible non-transitory computer-readable media storing computer-readable instructions that when executed by one or more processors cause the one or more processors to perform operations, the operations comprising:

capturing one or more images of an environment visible in a field of view of a mobile computing device;

receiving location data associated with a location of the mobile computing device;

generating, by the mobile computing device, image data comprising information associated with the one or more images;

determining, based at least in part on the image data and one or more machine-learned models, one or more object classifications of one or more objects in the environment, wherein the one or more machine-learned models are customized using user-specific data;

determining, based at least in part on the location data and the one or more object classifications of the one or more objects, one or more respective orientations of the one or more objects relative to the location of the mobile computing device; and

generating orientation data comprising a geographic orientation of the mobile computing device based at least in part on the one or more respective orientations of the one or more objects.

16. The one or more tangible non-transitory computer-readable media of claim 15 , further comprising:

sending the orientation data to one or more augmented reality applications associated with the mobile computing device, wherein the orientation of the mobile computing device is used to establish one or more locations of one or more augmented reality objects relative to the mobile computing device.

17. The one or more tangible non-transitory computer-readable media of claim 15 , wherein the one or more machine-learned models are trained based at least in part on training data associated with one or more images of one or more different geographic regions, and wherein each of the one or more geographic regions is associated with a respective plurality of traffic regulations.

18. A mobile computing device comprising:

one or more processors;

one or more non-transitory computer-readable media storing instructions that when executed by the one or more processors cause the one or more processors to perform operations comprising:

capturing one or more images of an environment visible in a field of view of the mobile computing device;

receiving location data associated with a location of the mobile computing device;

generating image data comprising information associated with the one or more images;

determining, based at least in part on the image data and one or more machine-learned models, one or more object classifications of one or more objects in the environment wherein the one or more machine-learned models are customized using user-specific data;

determining, based at least in part on the location data and the one or more object classifications of the one or more objects, one or more respective orientations of the one or more objects relative to the location of the mobile computing device; and

generating orientation data comprising a geographic orientation of the mobile computing device based at least in part on the one or more respective orientations of the one or more objects.

19. The mobile computing device of claim 18 , wherein the one or more objects comprise one or more vehicles.

20. The mobile computing device claim 18 , wherein the one or more machine-learned models are configured to identify one or more visual features associated with the image data.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 21, 2020
From: FILIP, DANIEL JOSEPH
To: GOOGLE LLC
Reel/Frame 052724/0148 →
Continuity (1)
Related Publication 20220319039A1 · Oct 6, 2022