IP Library › Granted Patent US 12,608,878
Granted Patent B2
US 12,608,878 · App. 17/539,089 · Granted Apr 21, 2026

Arbitrary view generation

Inventors: Clarence Chui (Los Altos Hills, CA); Manu Parmar (Sunnyvale, CA)
Assignee: Outward, Inc.
G06T15/205G06F16/58G06T5/50G06T7/32
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,608,878
App. No.
17/539,089
Granted
Apr 21, 2026
Kind
B2
Abstract

Techniques for generating an arbitrary view of an asset are disclosed. In some embodiments, arbitrary view generation includes storing a set of images associated with an asset in a database and generating an image comprising a requested view of the asset using at least a subset of the set of images associated with the asset that are stored in the database.

Claims (60)

1 . A system, comprising:

a database configured to store a set of images associated with an asset; and

a processor configured to generate an image comprising a requested view of the asset using at least a subset of the set of images associated with the asset that have different views than the requested view and corresponding metadata associated with the subset of the set of images and based at least in part on processing resources at a client requesting the image comprising the requested view.

2 . The system of claim 1 , wherein the image comprising the requested view is generated based at least in part on network bandwidth for communication with the client.

3 . The system of claim 1 , wherein the processor facilitates resource adaptive rendering.

4 . The system of claim 1 , wherein the processor facilitates resource adaptive rendering by trading off quality of the generated image comprising the requested view with responsiveness.

5 . The system of claim 1 , wherein the processor facilitates resource adaptive rendering at least in part by adaptively selecting a number of images of the asset having different views to include in the subset.

6 . The system of claim 1 , wherein the processor facilitates resource adaptive rendering at least in part by adaptively scaling quality of one or more of the set of images and including the scaled images in the subset.

7 . The system of claim 6 , wherein scaling quality comprises scaling one or both of resolution and bit depth.

8 . The system of claim 6 , wherein scaling quality comprises scaling one or both of pixel density and pixel precision.

9 . The system of claim 1 , wherein the set of images comprises reference views of the asset and images encoding metadata.

10 . The system of claim 1 , wherein the requested view comprises one or more of arbitrary camera characteristics, lighting, and textures.

11 . The system of claim 1 , wherein the set of images includes versions of images having different qualities.

12 . The system of claim 1 , wherein the set of images includes versions of images having one or both of different resolutions and different bit depths.

13 . The system of claim 1 , wherein the generated image comprising the requested view has a quality that is based at least in part on a number of reference images of the asset having different perspectives included in the subset.

14 . The system of claim 1 , wherein the generated image comprising the requested view has a quality that is based at least in part on resolutions of images comprising the subset.

15 . The system of claim 1 , wherein the generated image comprising the requested view has a quality that is based at least in part on bit depths of images comprising the subset.

16 . The system of claim 1 , wherein the set of images associated with the asset comprises a highest available quality.

17 . The system of claim 1 , wherein the database stores corresponding sets of images of a plurality of assets and wherein the image comprising the requested view comprises an asset ensemble including the asset that is generated using at least subsets of sets of images associated with assets comprising the ensemble.

18 . The system of claim 1 , wherein the processor is further configured to deliver the generated image comprising the requested view with progressive delivery of quality.

19 . A method, comprising:

storing a set of images associated with an asset in a database; and

generating an image comprising a requested view of the asset using at least a subset of the set of images associated with the asset that have different views than the requested view and corresponding metadata associated with the subset of the set of images and based at least in part on processing resources at a client requesting the image comprising the requested view.

20 . The method of claim 19 , wherein the image comprising the requested view is generated based at least in part on network bandwidth for communication with the client.

21 . The method of claim 19 , wherein the image comprising the requested view is generated based on resource adaptive rendering.

22 . The method of claim 19 , wherein the image comprising the requested view is generated based on resource adaptive rendering by trading off quality of the generated image comprising the requested view with responsiveness.

23 . The method of claim 19 , wherein the image comprising the requested view is generated based on resource adaptive rendering at least in part by adaptively selecting a number of images of the asset having different views to include in the subset.

24 . The method of claim 19 , wherein the image comprising the requested view is generated based on resource adaptive rendering at least in part by adaptively scaling quality of one or more of the set of images and including the scaled images in the subset.

25 . The method of claim 24 , wherein scaling quality comprises scaling one or both of resolution and bit depth.

26 . The method of claim 24 , wherein scaling quality comprises scaling one or both of pixel density and pixel precision.

27 . The method of claim 19 , wherein the set of images comprises reference views of the asset and images encoding metadata.

28 . The method of claim 19 , wherein the requested view comprises one or more of arbitrary camera characteristics, lighting, and textures.

29 . The method of claim 19 , wherein the set of images includes versions of images having different qualities.

30 . The method of claim 19 , wherein the set of images includes versions of images having one or both of different resolutions and different bit depths.

31 . The method of claim 19 , wherein the generated image comprising the requested view has a quality that is based at least in part on a number of reference images of the asset having different perspectives included in the subset.

32 . The method of claim 19 , wherein the generated image comprising the requested view has a quality that is based at least in part on resolutions of images comprising the subset.

33 . The method of claim 19 , wherein the generated image comprising the requested view has a quality that is based at least in part on bit depths of images comprising the subset.

34 . The method of claim 19 , wherein the set of images associated with the asset comprises a highest available quality.

35 . The method of claim 19 , wherein the database stores corresponding sets of images of a plurality of assets and wherein the image comprising the requested view comprises an asset ensemble including the asset that is generated using at least subsets of sets of images associated with assets comprising the ensemble.

36 . The method of claim 19 , wherein the generated image comprising the requested view is delivered with progressive delivery of quality.

37 . A computer program product, the computer program product embodied in a non-transitory computer readable medium and comprising computer instructions which when executed cause a computer to:

store a set of images associated with an asset in a database; and

generate an image comprising a requested view of the asset using at least a subset of the set of images associated with the asset that have different views than the requested view and corresponding metadata associated with the subset of the set of images and based at least in part on processing resources at a client requesting the image comprising the requested view.

38 . The computer program product of claim 37 , wherein the image comprising the requested view is generated based at least in part on network bandwidth for communication with the client.

39 . The computer program product of claim 37 , wherein the image comprising the requested view is generated based on resource adaptive rendering.

40 . The computer program product of claim 37 , wherein the image comprising the requested view is generated based on resource adaptive rendering by trading off quality of the generated image comprising the requested view with responsiveness.

41 . The computer program product of claim 37 , wherein the image comprising the requested view is generated based on resource adaptive rendering at least in part by adaptively selecting a number of images of the asset having different views to include in the subset.

42 . The computer program product of claim 37 , wherein the image comprising the requested view is generated based on resource adaptive rendering at least in part by adaptively scaling quality of one or more of the set of images and including the scaled images in the subset.

43 . The computer program product of claim 42 , wherein scaling quality comprises scaling one or both of resolution and bit depth.

44 . The computer program product of claim 42 , wherein scaling quality comprises scaling one or both of pixel density and pixel precision.

45 . The computer program product of claim 37 , wherein the set of images comprises reference views of the asset and images encoding metadata.

46 . The computer program product of claim 37 , wherein the requested view comprises one or more of arbitrary camera characteristics, lighting, and textures.

47 . The computer program product of claim 37 , wherein the set of images includes versions of images having different qualities.

48 . The computer program product of claim 37 , wherein the set of images includes versions of images having one or both of different resolutions and different bit depths.

49 . The computer program product of claim 37 , wherein the generated image comprising the requested view has a quality that is based at least in part on a number of reference images of the asset having different perspectives included in the subset.

50 . The computer program product of claim 37 , wherein the generated image comprising the requested view has a quality that is based at least in part on resolutions of images comprising the subset.

51 . The computer program product of claim 37 , wherein the generated image comprising the requested view has a quality that is based at least in part on bit depths of images comprising the subset.

52 . The computer program product of claim 37 , wherein the set of images associated with the asset comprises a highest available quality.

53 . The computer program product of claim 37 , wherein the database stores corresponding sets of images of a plurality of assets and wherein the image comprising the requested view comprises an asset ensemble including the asset that is generated using at least subsets of sets of images associated with assets comprising the ensemble.

54 . The computer program product of claim 37 , wherein the generated image comprising the requested view is delivered with progressive delivery of quality.

Continuity (6)
Continuation 16523886 · Jul 26, 2019
Continuation In Part 16181607 · Nov 6, 2018
Continuation 15721426 · Sep 29, 2017
Continuation In Part 15081553 · Mar 25, 2016
Provisional Application 62541607 · Aug 4, 2017
Related Publication 20220084280A1 · Mar 17, 2022
References Cited (61)
US 6222947B1 · Koba · 2001 [cited by applicant]
US 6362822B1 · Randel · 2002 [cited by applicant]
US 6377257B1 · Borrel · 2002 [cited by applicant]
US 8655052B2 · Spooner · 2014 [cited by applicant]
US 9407904B2 · Sandrew · 2016 [cited by applicant]
US 10909749B2 · Chui · 2021 [cited by applicant]
US 20050018045A1 · Thomas · 2005 [cited by applicant]
US 20060280368A1 · Petrich · 2006 [cited by applicant]
US 20070242284A1 · Schalkwijk · 2007 [cited by examiner]
US 20080143715A1 · Moden · 2008 [cited by applicant]
US 20090028403A1 · Bar-Aviv · 2009 [cited by applicant]
US 20090276105A1 · Lacaze · 2009 [cited by applicant]
US 20110001826A1 · Hongo · 2011 [cited by applicant]
US 20120120240A1 · Muramatsu · 2012 [cited by applicant]
US 20120140027A1 · Curtis · 2012 [cited by applicant]
US 20120163672A1 · Mckinnon · 2012 [cited by applicant]
US 20120314937A1 · Kim · 2012 [cited by applicant]
US 20130100290A1 · Sato · 2013 [cited by applicant]
US 20130222369A1 · Huston · 2013 [cited by applicant]
US 20130259448A1 · Stankiewicz · 2013 [cited by applicant]
US 20140063061A1 · Reitan · 2014 [cited by applicant]
US 20140198182A1 · Ward · 2014 [cited by applicant]
US 20140254908A1 · Strommer · 2014 [cited by applicant]
US 20140267343A1 · Arcas · 2014 [cited by applicant]
US 20150015581A1 · Lininger · 2015 [cited by applicant]
US 20150169982A1 · Perry · 2015 [cited by applicant]
US 20150317822A1 · Haimovitch-Yogev · 2015 [cited by applicant]
US 20170103512A1 · Mailhe · 2017 [cited by applicant]
US 20170277979A1 · Allen · 2017 [cited by applicant]
US 20170278251A1 · Peeper · 2017 [cited by applicant]
US 20170304732A1 · Velic · 2017 [cited by applicant]
US 20170334066A1 · Levine · 2017 [cited by applicant]
US 20170372193A1 · Mailhe · 2017 [cited by applicant]
US 20180012330A1 · Holzer · 2018 [cited by applicant]
US 20190325621A1 · Wang · 2019 [cited by applicant]
CN 101283375 · 2008 [cited by applicant]
CN 101281640 · 2012 [cited by applicant]
CN 103152518 · 2013 [cited by applicant]
CN 103179339 · 2013 [cited by applicant]
CN 103828359 · 2014 [cited by applicant]
CN 203870604 · 2014 [cited by applicant]
JP 2000137815 · 2000 [cited by applicant]
JP 2003187261 · 2003 [cited by applicant]
JP 2004287517 · 2004 [cited by applicant]
JP 2009211335 · 2009 [cited by applicant]
JP 2010140097 · 2010 [cited by applicant]
JP 2017212593 · 2017 [cited by applicant]
JP 2018081672 · 2018 [cited by applicant]
KR 20060029140 · 2006 [cited by applicant]
KR 20120137295 · 2012 [cited by applicant]
KR 20140021766 · 2014 [cited by applicant]
KR 20190094254 · 2019 [cited by applicant]
WO 2018197984 · 2018 [cited by applicant]
WO 2019167453 · 2019 [cited by applicant]
Daniel Scharstein. “A Survey of Image-Based Rendering and Stereo”. In: “View Synthesis Using Stereo Vision”, Lecture Notes in Computer Science, vol. 1583, Jan. 1, 1999, pp. 23-39. [cited by applicant]
Inamoto et al. “Virtual Viewpoint Replay for a Soccer Match by View Interpolation from Multiple Cameras”. IEEE Transactions on Multimedia, vol. 9 No. 6, Oct. 1, 2007, pp. 1155-1166. [cited by applicant]
Sun et al. “An overview of free viewpoint Depth-Image-Based Rendering (DIBR).” Proceedings of the Second APSIPA Annual Summit and Conference. Dec. 14, 2010, pp. 1-8. [cited by applicant]
Bouwmans et al.: “Deep neural network concepts for background subtraction:A systematic review and comparative evaluation”, Neural Networks, Elsevier Science Publishers, Barking, GB, vol. 117, May 15, 2019 (May 15, 2019)… [cited by applicant]
Ledig et al., “Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network”, Sep. 15, 2016 (Sep. 15, 2016), Retrieved from the Internet: URL:https://arxiv.org/pdf/1609.04802.pdf. [cited by applicant]
Stamatios Lefkimmiatis: “Non-local Color Image Denoising with Convolutional Neural Networks”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017), Jul. 1, 2017 (Jul. 1, 2017), pp. 5882-5891, ISBN: 978… [cited by applicant]
Sheng et al., “Virtue Plane Mapping: A Method of Rendering into Depth Images”, Journal of Software, vol. 19, No. 7, Jul. 2008, pp. 1806-1816. [cited by applicant]