IP Library Granted Patent US 12,339,897
Granted Patent B2
US 12,339,897 · App. 18/377,989 · Granted Jun 24, 2025

Displaying objects based on a plurality of models

Inventor: Janne Matias Kontkanen (San Francisco, CA)
Assignee: Google Inc.
G06F16/5838G06T15/00G06T15/04G06T15/20G06T15/205G06T15/503G06T17/00G06V20/30
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,339,897
App. No.
18/377,989
Granted
Jun 24, 2025
Kind
B2
Abstract

A system and method is provided for displaying surfaces of an object from a vantage point different from the vantage point from which imagery of the object was captured. In some aspects, imagery may be generated for display by combining visual characteristics from multiple source images and applying greater weight to the visual characteristics of some of the source images relative to the other source images. The weight may be based on the orientation of the surface relative to the location from which the image was captured and the location from which the object will be displayed.

Claims (49)

1. A method of providing an image for display, the method comprising:

receiving a request for an image depicting an object from a requested vantage point;

accessing information associated with a first source image of the object captured from a first vantage point;

accessing information associated with a second source image of the object captured from a second vantage point;

generating the requested image using the information from the first source image and the information from the second source image and the requested vantage point; and

providing the requested image for display.

2. The method of claim 1 , wherein generating the requested image comprises accessing at least one model that is based on at least one of the information from the first source image or the information from the second source image.

3. The method of claim 2 , further comprising:

determining a first visual characteristic from a first model; and

determining a second visual characteristic from a second model;

wherein generating the requested image includes generating the requested image to have a third visual characteristic that is based on the first visual characteristic and the second visual characteristic.

4. The method of claim 3 , further comprising determining an orientation associated with the requested vantage point, wherein the first visual characteristic of the first model is based on the orientation, and the second visual characteristic of the second model is based on the orientation.

5. The method of claim 3 , wherein the third visual characteristic of the third image is based on a first orientation associated with the first model and a second orientation associated with the second model.

6. The method of claim 3 , wherein the third visual characteristic of the third image is based on first location information associated with the first model and second location information associated with the second model.

7. The method of claim 3 , wherein:

the first visual characteristic is a first texel generated by projecting image data captured from the first vantage point onto the first model; and

the second visual characteristic is a second texel generated by projecting image data captured from the second vantage point onto the second model.

8. The method of claim 3 , wherein the first visual characteristic and second visual characteristics are associated with colors, and wherein the third visual characteristic of the third image is based on blending a first color associated with the first visual characteristic and blending a second color associated with the second visual characteristic.

9. The method of claim 3 , wherein the object has a surface defining a plane, the method further comprises selecting a point on the plane, and the first visual characteristic and the second visual characteristic are determined based on a respective angle between the plane and a line extending from the first vantage point and the second vantage point to the selected point on the plane.

10. A system comprising:

one or more computing devices; and

memory storing instructions, the instructions being executable by the one or more computing devices;

wherein the instructions comprise:

receiving a request for an image depicting an object from a requested vantage point;

accessing information associated with a first source image of the object captured from a first vantage point;

accessing information associated with a second source image of the object captured from a second vantage point;

generating the requested image using the information from the first source image and the information from the second source image and the requested vantage point; and

providing the requested image for display.

11. The system of claim 10 , wherein generating the requested image comprises accessing at least one model that is based on at least one of the information from the first source image or the information from the second source image.

12. The system of claim 11 , further comprising:

determining a first visual characteristic from a first model; and

determining a second visual characteristic from a second model;

wherein generating the requested image includes generating the requested image to have a third visual characteristic that is based on the first visual characteristic and the second visual characteristic.

13. The system of claim 12 , further comprising determining an orientation associated with the requested vantage point, wherein the first visual characteristic of the first model is based on the orientation, and the second visual characteristic of the second model is based on the orientation.

14. The system of claim 12 , wherein the third visual characteristic of the third image is based on a first orientation associated with the first model and a second orientation associated with the second model.

15. The system of claim 12 , wherein the third visual characteristic of the third image is based on first location information associated with the first model and second location information associated with the second model.

16. The system of claim 12 , wherein the first visual characteristic and second visual characteristics are associated with colors, and wherein the third visual characteristic of the third image is based on blending a first color associated with the first visual characteristic and blending a second color associated with the second visual characteristic.

17. A non-transitory computing-device readable storage medium on which computing-device readable instructions of a program are stored, the instructions, when executed by one or more computing devices, causing the one or more computing devices to perform a method, the method comprising:

receiving a request for an image depicting an object from a requested vantage point;

accessing information associated with a first source image of the object captured from a first vantage point;

accessing information associated with a second source image of the object captured from a second vantage point;

generating the requested image using the information from the first source image and the information from the second source image and the requested vantage point; and

providing the requested image for display.

18. The medium of claim 17 , wherein generating the requested image comprises accessing at least one model that is based on at least one of the information from the first source image or the information from the second source image.

19. The medium of claim 18 , further comprising:

determining a first visual characteristic from a first model; and

determining a second visual characteristic from a second model;

wherein generating the requested image includes generating the requested image to have a third visual characteristic that is based on the first visual characteristic and the second visual characteristic.

20. The medium of claim 19 , further comprising determining an orientation associated with the requested vantage point, wherein the first visual characteristic of the first model is based on the orientation, and the second visual characteristic of the second model is based on the orientation.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 10, 2023
From: KONTKANEN, JANNE MATIAS
To: GOOGLE INC.
Reel/Frame 065170/0394 →
CHANGE OF NAME Recorded Oct 10, 2023
From: GOOGLE INC.
To: GOOGLE LLC
Reel/Frame 065197/0523 →
Continuity (5)
Continuation 17320648 · May 14, 2021
Continuation 16575941 · Sep 19, 2019
Continuation 15692548 · Aug 31, 2017
Continuation 14877368 · Oct 7, 2015
Related Publication 20240054155A1 · Feb 15, 2024
References Cited (73)
US 6697062B1 · Cabral et al. · 2004 [cited by applicant]
US 7085409B2 · Sawhney et al. · 2006 [cited by applicant]
US 8244025B2 · Davis et al. · 2012 [cited by applicant]
US 8842162B2 · Goldner et al. · 2014 [cited by applicant]
US 8963916B2 · Reitan · 2015 [cited by applicant]
US 9626798B2 · Zavesky · 2017 [cited by applicant]
US 11086927B2 · Kontkanen · 2021 [cited by applicant]
US 11809487B2 · Kontkanen · 2023 [cited by examiner]
US 20020061131A1 · Sawhney et al. · 2002 [cited by applicant]
US 20110178783A1 · Smith et al. · 2011 [cited by applicant]
US 20130057644A1 · Stefanoski et al. · 2013 [cited by applicant]
US 20130127988A1 · Wang et al. · 2013 [cited by applicant]
US 20140320535A1 · Ying · 2014 [cited by applicant]
US 20150043636A1 · Jung et al. · 2015 [cited by applicant]
US 20150055821A1 · Fotland · 2015 [cited by applicant]
US 20150130799A1 · Holzer et al. · 2015 [cited by applicant]
US 20160062573A1 · Dascola et al. · 2016 [cited by applicant]
US 20160260244A1 · Moravec et al. · 2016 [cited by applicant]
US 20170084293A1 · Holzer et al. · 2017 [cited by applicant]
CA 2644032A1 · 2007 [cited by applicant]
CN 1607551A · 2005 [cited by applicant]
CN 101321299A · 2008 [cited by applicant]
CN 101398936A · 2009 [cited by applicant]
CN 101673394A · 2010 [cited by applicant]
CN 102027504A · 2011 [cited by applicant]
CN 102640502A · 2012 [cited by applicant]
CN 104885107A · 2015 [cited by applicant]
EP 1313329A2 · 2003 [cited by applicant]
JP 2004537082A · 2004 [cited by applicant]
JP 2006350553A · 2006 [cited by applicant]
JP 2010020487A · 2010 [cited by applicant]
WO 2002045001A1 · 2002 [cited by applicant]
WO 02069272A2 · 2002 [cited by applicant]
WO 2010114878A1 · 2010 [cited by applicant]
Office Action for Japanese Patent Application No. 2023-41182 dated Jun. 24, 2024. 3 pages. [cited by applicant]
Brian Curless and Marc Levoy; A Volumetric Method for Building Complex Models from Range Images; Stanford University; 1996; 10 pages. [cited by applicant]
Ned Greene and Paul Heckbert; Creating Raster Omnimax Images from Multiple Perspective Views Using the Elliptical Weighted Average Filters; New York Institute of Technology; 1986; 7 pages. [cited by applicant]
Steven Maxwell Seitz; Image-Based Transformation of Viewpoint and Scene Appearance; University of Wisconsin—Madison; 1997; 28 pages. [cited by applicant]
William R. Mark; Post-Rendering 3D Image Warping: Visibility, Reconstructions, and Performance for Depth-Image Warping; Apr. 21, 1999; 13 pages. [cited by applicant]
Per H. Christensen; David M. Laur; Julian Fong; Wayne L. Wooten; Dana Batali; Ray Differentials and Multiresolution Geometry Caching for Distribution Ray Tracing in Complex Scenes; Eurographics 2003 / P. Burnet and D. F… [cited by applicant]
Chris Buehler; Michael Bosse; Leonard McMillan; Steven Gortler; Michael Cohen; Unstructured Lumigraph Rendering; 2001; 8 pages. [cited by applicant]
Kari Pulli; Michael Cohen; Tom Duchamp; Hugues Hoppe; Linda Shapiro; Werner Stuetzle; View-Based Rendering: Visualizing Real Objects from Scanned Range and Color Data; 1997; 13 pages. [cited by applicant]
Paul S. Heckbert; Fundamentals of Texture Mapping and Image Warping; Jun. 17, 1989; 94 pages. [cited by applicant]
Renato Pajarola; Miguel Sainz and Yu Meng; Dmesh: Fast Depth-Image Meshing and Warping: Jul. 31, 2003; 29 pages. [cited by applicant]
Paul Debevec et al.: “Efficient View-Dependent Image-Based Rendering with Projective Texture-Mapping” In: “Rendering Techniques '98”, Jan. 1, 1998 (Jan. 1, 1998), Springer Vienna, Vienna, XP55164217, ISSN: 0946-2767 ISB… [cited by applicant]
Narayanan, P. J. et al.: “Virtual worlds using computer vision”, Computer Vision for Virtual Reality Based Human Communications, 1998. Proceedings, 1998 IEEE and ATR Workshop on Bombay, India Jan. 3, 1998, Los Alamitos,… [cited by applicant]
Akash Kushal et al.: “Modeling 3D Objects from Stereo Views and Recognizing Them in Photographs”, Computer Vision—ECCV 2006 Lecture Notes in Computer Science; LNCS, Springer, Berlin, DE, vol. 3952, Jan. 1, 2006 (Jan. 1,… [cited by applicant]
Invitation to Pay Additional Fees and Communication Relating to the Results of the Partial International Search for PCT Application No. PCT/US2016/055323, dated Dec. 2, 2016. [cited by applicant]
International Search Report and Written Opinion for PCT Application No. PCT/US2016/055323, dated Jan. 23, 2017. [cited by applicant]
International Preliminary Report on Patentability dated Apr. 19, 2018 for International Patent Application No. PCT/US2016/055323. 14 pages. [cited by applicant]
Office Action issued in Japanese Patent Application 2018-500421, dated Jan. 21, 2019, 2 pages. [cited by applicant]
Office Action for Korean Patent Application No. 10-2018-7000631 dated Aug. 5, 2019. [cited by applicant]
Notice of Allowance for Korean Patent Application No. 10-2018-7000631 dated Feb. 13, 2020. [cited by applicant]
Office Action for Korean Patent Application No. 10-2020-7013013 dated Jul. 2, 2020. 4 pages. [cited by applicant]
Szeliski, Richard. Computer Vision: Algorithms and Applications. Jan. 31, 2012. Tsinghua University Press, 3 pages. [cited by applicant]
Sonka, Milan et al. Image Processing, Analysis, and Machine Vision. Third Edition. Jan. 31, 2011. Tsinghua University Press, 4 pages. [cited by applicant]
Notification of the First Office Action for Chinese Patent Application No. 201680039889.1 dated Sep. 14, 2020. 10 pages. [cited by applicant]
First Examination Report for Indian Patent Application No. 201847000299 dated Oct. 26, 2020. 8 pages. [cited by applicant]
Notice of Reasons for Rejection for Japanese Patent Application No. 2019-143176 dated Oct. 26, 2020. 4 pages. [cited by applicant]
Notice of Allowance for Korean Patent Application No. 10-2020-7013013 dated Dec. 22, 2020. 2 pages. [cited by applicant]
Examination Report for United Kingdom Patent Application No. 1800102.4 dated Feb. 9, 2021. 4 pages. [cited by applicant]
Office Action for Korean Patent Application No. 10-2021-7008457 dated Apr. 5, 2021. 4 pages. [cited by applicant]
Examination Report for European Patent Application No. 16788840.3 dated Apr. 19, 2021. 11 pages. [cited by applicant]
Notification of the Second Office Action for Chinese Patent Application No. 201680039889.1 dated May 13, 2021. 6 pages. [cited by applicant]
Result of Consultation for European Patent Application No. 16788840.3 dated Aug. 12, 2021. 3 pages. [cited by applicant]
Notice of Allowance for Korean Patent Application No. 10-2021-7008457 dated Sep. 27, 2021. 2 pages. [cited by applicant]
Combined Search and Examination Report for UK Patent Applicaton No. 2112089.4 dated Nov. 1, 2021. 4 pages. [cited by applicant]
Notice of Reasons for Rejection for Japanese Patent Application No. 2021-115686 dated Jul. 11, 2022. 3 pages. [cited by applicant]
Office Action for German Patent Application No. 112016003134.3 dated Dec. 19, 2022. 14 pages. [cited by applicant]
Notice of Allowance for Japanese Patent Application No. 2021-115686 dated Feb. 13, 2023. 3 pages. [cited by applicant]
Office Action for European Patent Application No. 16788840.3 dated Apr. 24, 2023. 6 pages. [cited by applicant]
Notice of Allowance for Japanese Patent Application No. 2023-041182 dated Oct. 28, 2024. 3 pages. [cited by applicant]
Notice of Grant for Chinese Patent Application No. 202210415621.0 dated Apr. 29, 2025. 4 pages. [cited by applicant]