IP Library › Granted Patent US 12,242,704
Granted Patent B2
US 12,242,704 · App. 18/317,587 · Granted Mar 4, 2025

System and method of indicating the distance or the surface of an image of a geographical object

Inventors: Daniel Joseph Filip (San Jose, CA); Andrew Timothy Szybalski (San Francisco, CA)
Assignee: GOOGLE LLC
G06F3/04812G01C21/367G06F3/048G06F16/29
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,242,704
App. No.
18/317,587
Granted
Mar 4, 2025
Kind
B2
Abstract

A system and method is provided that displays cursors for street level images, where the cursor changes appearance based on the objects in the image, such as the geographic distance between the objects and the camera position and the surface of the objects. For example, the cursor may appear to lie flat against the objects in the image change size based on the distance between the camera and object's surface.

Claims (65)

1. A method, comprising:

receiving an image including a plurality of geographical objects including a first geographical object and a second geographical object;

providing the image for presentation on an electronic display of a computing device;

when a cursor associated with the computing device is provided for display on the image and positioned on the first geographical object, providing the cursor with a first visual characteristic based on a first distance which is associated with a distance between a camera which captured the image and the first geographical object; and

when the cursor is provided for display on the image and positioned on the second geographical object, providing the cursor with a second visual characteristic based on a second distance which is associated with a distance between the camera which captured the image and the second geographical object.

2. The method of claim 1 , wherein

the first visual characteristic includes a first shape of the cursor, and

the second visual characteristic includes a second shape of the cursor, the second shape being different from the first shape.

3. The method of claim 2 , wherein

the first geographical object is a building, and

the second geographical object is a travel route.

4. The method of claim 3 , wherein

the first shape includes a polygon that has a size based on the first distance, and

the second shape includes a circle that has a size based on the second distance.

5. The method of claim 2 , wherein

the first shape of the cursor is based on a type of the first geographical object.

6. The method of claim 2 , wherein

the first shape of the cursor is based on an orientation of the first geographical object.

7. The method of claim 6 , wherein

when the orientation of the first geographical object is perpendicular to a ground plane in the image, the first shape of the cursor corresponds to a polygon, and

when the orientation of the first geographical object is parallel to the ground plane in the image, the first shape of the cursor corresponds to a circle.

8. The method of claim 1 , wherein

the first visual characteristic includes at least one of a first shape, a first size, or a first color.

9. The method of claim 1 , wherein the plurality of geographical objects include a third geographic object which at least partially overlaps the first geographical object, and the method further comprising:

when the cursor is provided for display on the image and positioned on the third geographical object and a size of the third geographical object is less than a threshold value, providing the cursor with the first visual characteristic based on the first distance.

10. A method, comprising:

receiving an image including a plurality of geographical objects including a first geographical object and a second geographical object;

providing the image for presentation on an electronic display of a computing device from a first viewpoint having a first orientation;

when a cursor associated with the computing device is provided for display on the image and positioned on the first geographical object, providing the cursor with a first visual characteristic which is indicative of a first operation to be executed in response to receiving a first user input while the cursor is positioned on the first geographical object; and

when the cursor is provided for display on the image and positioned on the second geographical object, providing the cursor with a second visual characteristic which is indicative of a second operation to be executed in response to receiving a second user input while the cursor is positioned on the second geographical object.

11. The method of claim 10 , further comprising:

receiving the first user input while the cursor is positioned on the first geographical object; and

executing the first operation, wherein executing the first operation comprises providing another image for presentation on the electronic display of the computing device from a second viewpoint having the first orientation.

12. The method of claim 11 , further comprising:

receiving the second user input while the cursor is positioned on the second geographical object; and

executing the second operation, wherein executing the second operation comprises providing another image for presentation on the electronic display of the computing device from the first viewpoint having a second orientation, different from the first viewpoint.

13. The method of claim 10 , wherein the first visual characteristic includes a first shape that indicates whether the first orientation will change in response to the first operation being executed in response to receiving the first user input while the cursor is positioned on the first geographical object.

14. The method of claim 10 , wherein

the first visual characteristic includes a first shape, and

the first shape of the cursor is based on a type of the first geographical object and the first operation to be executed in response to receiving the first user input while the cursor is positioned on the first geographical object.

15. The method of claim 10 , wherein

the first visual characteristic includes a first shape, and

the first shape of the cursor is based on a first distance which is associated with a distance between a camera which captured the image and the first geographical object, and the first operation to be executed in response to receiving the first user input while the cursor is positioned on the first geographical object.

16. A computing system, comprising:

a display device configured to display an image and to display a cursor on the image;

an input device configured to control movement of the cursor;

one or more non-transitory computer-readable media configured to store instructions; and

one or more processors configured to execute the instructions to perform operations, the operations comprising:

receiving the image, the image including a plurality of geographical objects including a first geographical object and a second geographical object,

providing the image for presentation on the display device,

when the cursor is positioned, via the input device, on the first geographical object, providing the cursor with a first visual characteristic based on a first distance which is associated with a distance between a camera which captured the image and the first geographical object, and

when the cursor is positioned, via the input device, on the second geographical object, providing the cursor with a second visual characteristic based on a second distance which is associated with a distance between the camera which captured the image and the second geographical object.

17. The computing system of claim 16 , wherein

the first visual characteristic includes a first shape of the cursor, and

the second visual characteristic includes a second shape of the cursor, the second shape being different from the first shape.

18. The computing system of claim 17 , wherein

the first shape includes a polygon that has a size based on the first distance, and

the second shape includes a circle that has a size based on the second distance.

19. The computing system of claim 17 , wherein

the first shape of the cursor is based on a type of the first geographical object or an orientation of the first geographical object.

20. The computing system of claim 16 , wherein the operations further comprise:

the first visual characteristic is further based on a first operation to be executed in response to receiving a first user input, via the input device, while the cursor is positioned on the first geographical object,

wherein

the image is provided for presentation on the display device from a first viewpoint having a first orientation, and

the first visual characteristic includes a first shape that indicates whether the first orientation will change in response to the first operation being executed in response to receiving the first user input while the cursor is positioned on the first geographical object.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 6, 2023
From: FILIP, DANIEL; SZYBALSKI, ANDREW
To: GOOGLE INC.
Reel/Frame 063865/0818 →
CHANGE OF NAME Recorded Jun 6, 2023
From: GOOGLE INC.
To: GOOGLE LLC
Reel/Frame 063881/0768 →
Continuity (4)
Continuation 17496462 · Oct 7, 2021
Continuation 15292226 · Oct 13, 2016
Continuation 12415102 · Mar 31, 2009
Related Publication 20230289026A1 · Sep 14, 2023
References Cited (110)
US 5559707A · DeLorme et al. · 1996 [cited by applicant]
US 6009190A · Szeliski et al. · 1999 [cited by applicant]
US 6097393A · Prouty, IV et al. · 2000 [cited by applicant]
US 6157747A · Szeliski et al. · 2000 [cited by applicant]
US 6285317B1 · Ong · 2001 [cited by applicant]
US 6346938B1 · Chan et al. · 2002 [cited by applicant]
US 6477268B1 · Chiang et al. · 2002 [cited by applicant]
US 6563529B1 · Jongerius · 2003 [cited by applicant]
US 6717608B1 · Mancuso et al. · 2004 [cited by applicant]
US 6687753B2 · Schneider · 2004 [cited by applicant]
US 6771304B1 · Mancuso et al. · 2004 [cited by applicant]
US 6885392B1 · Mancuso et al. · 2005 [cited by applicant]
US 6895126B2 · Bernardo et al. · 2005 [cited by applicant]
US 7006707B2 · Peterson · 2006 [cited by applicant]
US 7050102B1 · Vincent · 2006 [cited by applicant]
US 7096428B2 · Foote et al. · 2006 [cited by applicant]
US 7158878B2 · Rasmussen et al. · 2007 [cited by applicant]
US 7239760B2 · Di Bernardo et al. · 2007 [cited by applicant]
US 7161604B2 · Higgins et al. · 2007 [cited by applicant]
US 7424133B2 · Schultz et al. · 2008 [cited by applicant]
US 7460953B2 · Herbst et al. · 2008 [cited by applicant]
US 7526718B2 · Samadani et al. · 2009 [cited by applicant]
US 7574381B1 · Lin-Hendel · 2009 [cited by applicant]
US 7577316B2 · Di Bernardo et al. · 2009 [cited by applicant]
US 7643673B2 · Rohlf et al. · 2010 [cited by applicant]
US 7787659B2 · Schultz et al. · 2010 [cited by applicant]
US 7805025B2 · Di Bernardo et al. · 2010 [cited by applicant]
US 7813596B2 · Di Bernardo et al. · 2010 [cited by applicant]
US 10540294B2 · Zhu et al. · 2020 [cited by applicant]
US 20010033283A1 · Liang et al. · 2001 [cited by applicant]
US 20020010734A1 · Ebersole et al. · 2002 [cited by applicant]
US 20020130838A1 · Feierbach · 2002 [cited by applicant]
US 20020138338A1 · Trauth et al. · 2002 [cited by applicant]
US 20030080978A1 · Navab et al. · 2003 [cited by applicant]
US 20030142115A1 · Endo et al. · 2003 [cited by applicant]
US 20030210228A1 · Ebersole · 2003 [cited by applicant]
US 20040196282A1 · Oh · 2004 [cited by applicant]
US 20040239699A1 · Uyttendaele et al. · 2004 [cited by applicant]
US 20040257384A1 · Park et al. · 2004 [cited by applicant]
US 20050004749A1 · Park · 2005 [cited by applicant]
US 20050116964A1 · Kotake et al. · 2005 [cited by applicant]
US 20060073853A1 · Cighir et al. · 2006 [cited by applicant]
US 20060074549A1 · Takahashi et al. · 2006 [cited by applicant]
US 20060087507A1 · Urano et al. · 2006 [cited by applicant]
US 20060132482A1 · Oh · 2006 [cited by applicant]
US 20060238383A1 · Kimchi et al. · 2006 [cited by applicant]
US 20060271287A1 · Gold et al. · 2006 [cited by applicant]
US 20070030396A1 · Zhou et al. · 2007 [cited by applicant]
US 20070035563A1 · Biocca et al. · 2007 [cited by applicant]
US 20070070069A1 · Samarasekera et al. · 2007 [cited by applicant]
US 20070110338A1 · Snavely et al. · 2007 [cited by applicant]
US 20070146325A1 · Poston et al. · 2007 [cited by applicant]
US 20070273758A1 · Mendoza et al. · 2007 [cited by applicant]
US 20080043020A1 · Snow et al. · 2008 [cited by applicant]
US 20080089558A1 · Vadon et al. · 2008 [cited by applicant]
US 20080143709A1 · Fassero et al. · 2008 [cited by applicant]
US 20080189031A1 · Meadow et al. · 2008 [cited by applicant]
US 20080195315A1 · Hu et al. · 2008 [cited by applicant]
US 20080235630A1 · Kenney · 2008 [cited by applicant]
US 20080309668A1 · Borovikov · 2008 [cited by applicant]
US 20090237403A1 · Horii et al. · 2009 [cited by applicant]
US 20100142748A1 · Oldroyd · 2010 [cited by applicant]
US 20100199225A1 · Coleman · 2010 [cited by examiner]
US 20130135288A1 · King et al. · 2013 [cited by applicant]
EP 1426904 · 2004 [cited by applicant]
JP 2004342004 · 2004 [cited by applicant]
JP 2005250560 · 2005 [cited by applicant]
JP 2008520052 · 2008 [cited by applicant]
WO WO2010114878 · 2010 [cited by applicant]
Agarwal, “How to Embed FLV Flash Videos in your Blog?”, Digital Inspiration, http://labnol.blogspot.com/2006/08/how-to-embed-fly-flash-videos-in-your.html, retrieved on Jun. 27, 2022, 1 page. [cited by applicant]
Bay et al., “SURF: Speeded Up Robust Features”, Computer Vision—European Conference on Computer Vision 2006, Jul. 26, 2006, pp. 1-14. [cited by applicant]
Chen, “QuickTime VR—An Image-Based Approach to Virtual Environment Navigation”, Proceedings of the Twenty Second Annual Conference on Computer Graphics and Interactive Techniques Aug. 6-11, 1995, Los Angeles, California… [cited by applicant]
Cobzas et al., “A Panoramic Model for Remote Robot Environment Mapping and Predictive Display”, International Journal of Robotics and Automation, 2005, 11 pages. [cited by applicant]
Darken et al., “Map Usage in Virtual Environments: Orientation Issues,” Institute of Electrical and Electronics Engineers Virtual Reality Conference, Houston, Texas, United States, Mar. 13-17, 1999, pp. 133-140. [cited by applicant]
Dykes et al., “Virtual Environments for Student Fieldwork Using Networked Components,” International Journal of Geographical Information Science, vol. 13, No. 4, pp. 397-416, Taylor & Francis Ltd., 1999. [cited by applicant]
Dykes, “An Approach to Virtual Environments for Visualization Using Linked Georeferenced Panoramic Imagery,” Computers, Environment and Urban Systems, 24, 2000, pp. 127-152. [cited by applicant]
Endo et al., “Image-Based Walk-Through System for Large-Scale Scenes”, Fourth International Conference on Virtual Systems and Multimedia, Nov. 18-20, 1988, Gifu, Japan, 6 pages. [cited by applicant]
EP Patent Office, Partial International Search Report, International Application No. PCT/US2008/006683, date of mailing: Nov. 7, 2008, 5 pages. [cited by applicant]
First Office Action, dated Apr. 1, 2013, Japanese Patent Application No. 2010-548720, Japanese Patent Office, 7 pages (English language translation appended). [cited by applicant]
Hirose, “Image-Based Virtual World Generation,” Institute of Electrical and Electronics Engineers MultiMedia, vol. 4, No. 1, 1997, pp. 27-33. [cited by applicant]
http://labnol.blogspot.com/2006/08/how-to-embed-flv-flash-videos-in-your.html, retrieved on Jun. 27, 2022, 1 page. [cited by applicant]
International Preliminary Report on Patentability for Application No. PCT/US2010/29337, mailed Oct. 4, 2011, 8 pages. [cited by applicant]
International Preliminary Report on Patentability, dated Sep. 10, 2010, International Patent Application No. PCT/US2009/001216, The International Bureau of WIPO, Geneva, Switzerland, 11 pages. [cited by applicant]
International Search Report and Written Opinion for International Patent Application PCT/US2009/001216, European Patent Office, Netherlands, completed Sep. 8, 2009, mailed Sep. 21, 2009, 20 pages. [cited by applicant]
International Search Report and Written Opinion, PCT/US2010/29337, dated Jun. 1, 2010, 9 pages. [cited by applicant]
Kadobayashi, et al., “Combined Use of 2D Images and 3D Models for Retrieving and Browsing Digital Archive Contents”, Videometrics VIIL Proceedings of the SPIE—The International Society for Optical Engineering, San Jose,… [cited by applicant]
Kang et al., “Virtual Navigation of Complex Scenes Using Clusters of Cylindrical Panoramic Images”, Cambridge Research Laboratory—Technical Report Series, CRL 97/5, Sep. 1997, 30 pages. [cited by applicant]
Kato et al., “Town Digitizing: Recording of Street Views by Using Omnidirectional Vision Sensors”, Twenty-Sixth Annual Conference of the Institute of Electrical and Electronics Engineers Industrial Electronics Society, … [cited by applicant]
Kimber et al., “FlyAbout: Spatially Indexed Panoramic Video”, Ninth Association for Computing Machinery International Conference on Multimedia, Sep. 30-Oct. 5, 2001, Ottawa, Ontario, Canada, pp. 339-347. [cited by applicant]
Kourogi et al., “A Panorama-Base Technique for Annotation Overlay and its Real-Time Implementation”, Institute of Electrical and Electronics Engineers International Conference on Multimedia and Expo, Jul. 30-Aug. 2, 200… [cited by applicant]
Kourogi et al., “Improvement of Panorama-Based Annotation Overlay Using Omnidirectional Vision and Inertial Sensors”, Fourth Institute of Electrical and Electronics Engineers International Symposium on Wearable Computer… [cited by applicant]
Kulju et al., “Route Guidance Using a 3D City Mobile Device”, Mobile HCI02 Workshop: Mobile Tourism Support Systems, 2002, 6 pages. [cited by applicant]
Lippman, “Movie-Maps: An Application of the Optical Videodisc to Computer Graphics,” Association for Computing Machinery SIGGRAPH Computer Graphics, vol. 14, No. 3, 1980, pp. 32-42. [cited by applicant]
Lowe et al., “Fitting Parameterized Three-Dimensional Models to Images,” Institute of Electrical and Electronics Engineers Transactions on Pattern Analysis and Machine Intelligence, 13, May 5, 1991, pp. 441-450. [cited by applicant]
Murphy et al., “Air-Mobile Ground Surveillance and Security System (AMGSSS) Project Summary Report”, Technical Document 2914, Defense Special Weapons Agency (DSWA) and Physical Security Equipment Management Office (PSEM… [cited by applicant]
Nguyen et al., “Virtual Systems: Aspects of the Air-Mobile Ground Security and Surveillance System Prototype,” Unmanned Systems, vol. 14, No. 1, 11 pages, Winter 1996. [cited by applicant]
Notification of the First Office Action dated Mar. 30, 2012, Chinese Patent Application 200980114885.5, The State Intellectual Property Office of the People's Republic of China, 7 pages. (English language translation ap… [cited by applicant]
Notification of the Third Office Action dated Feb. 25, 2013, Chinese Patent Application 200980114885.5, The State Intellectual Property Office of the People's Republic of China, 7 pages. (English language translation ap… [cited by applicant]
Snavely et al., “Photo Tourism: Exploring Photo Collections in 3D,” Association for Computing Machinery Transactions on Graphics (SIGGRAPH Proceedings), 25(3), 2006, pp. 835-846. [cited by applicant]
Tanikawa et al., “Building a Photo-Realistic Virtual World Using View-Dependent Images and Models,” Institute of Electrical and Electronics Engineers, 1999, pp. VI-98 to VI-103. [cited by applicant]
The State Intellectual Property Office of The People's Republic of China, “Notification of the Second Office Action,” Application No. 200980114885.5, Date of Issuing: Sep. 12, 2012, pp. 1/52-10/52 (English language tran… [cited by applicant]
U.S. Appl. No. 11/754,265. [cited by applicant]
U.S. Appl. No. 11/754,266. [cited by applicant]
U.S. Appl. No. 11/754,267. [cited by applicant]
U.S. Appl. No. 12/038,325. [cited by applicant]
Web Archive Capture (Feb. 26, 2008), http://phototour.cs.washington.edu/, retrieved on Jun. 22, 2022, 2 pages. [cited by applicant]
Wei et al., “A Panoramic Image-Based Approach to Virtual Scene Modeling”, Multiconference on “Computational Engineering in Systems Applications”, Oct. 4-6, 2006, Beijing, China, pp. 1449-1453. [cited by applicant]
Wei et al., “Color Anaglyphs for Panorama Visualizations,” Communication and Information Technology Research Technical Report (“CITR-TR”), No. 19, Feb. 1998, pp. 2-15. [cited by applicant]
Wikipedia, “QuickTime VR,” Nov. 7, 2007, https://en.wikipedia.org/wiki/QuickTime_VR on Jun. 22, 2022, 5 pages. [cited by applicant]
Zheng et al., “Panoramic Representation of Scenes for Route Understanding,” Institute of Electrical and Electronics Engineers, 1990, pp. 161-167. [cited by applicant]