IP Library Granted Patent US 12,296,264
Granted Patent B2
US 12,296,264 · App. 18/359,357 · Granted May 13, 2025

Systems and methods for rendering a virtual content object in an augmented reality environment

Inventor: Nicholas T. Hariton (Trabuco Canyon, CA)
A63F13/53A63F13/216A63F13/26A63F13/52G06F3/1454G06F3/147G06T15/20G06T19/006G06T19/20H04L67/131A63F2003/00996A63F13/822A63F2300/8076G06T13/20G06T2200/24G06T2215/16G06T2219/024G07C15/006G09G2320/0261
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,296,264
App. No.
18/359,357
Granted
May 13, 2025
Kind
B2
Abstract

Systems and methods for rendering of a virtual content object in an augmented reality environment based on a physical marker are discussed herein. Virtual content objects may be rendered by a display device in an augmented reality environment based on the field of view seen through the display device and a position of a marker (and one or more linkage points associated with the marker) in the real world. When rendered in the augmented reality environment, the virtual content objects may be visualized from any angle, from the exterior or interior of the object, and manipulated in response to user input. Virtual content objects and/or user visualizations of virtual content objects may be shared with other users (local and/or remote), enabling multiple users to potentially build, modify, and/or interact with a virtual content object simultaneously and/or cooperatively.

Claims (30)

1. A system configured to convey virtual content in an augmented reality environment, via a presentation device, by causing superimposition of images of the virtual content over physical objects and/or physical surroundings visible within a field of view of a user as if the images of the virtual content were present in a real world, wherein the augmented reality environment includes (i) the physical objects and/or the physical surroundings visible within the field of view of the user and (ii) the images of the virtual content presented in conjunction with the physical objects and/or the physical surroundings, the system comprising:

electronic storage that stores information related to virtual surfaces that are applied to physical surfaces in the augmented reality environment visible within the field of view of the user; and

one or more processors configured by machine-readable instructions to:

generate a user interface that enables the user to provide user input related to the physical objects visible within the field of view of the user;

obtain, via the user interface, the user input, wherein the user input specifies (i) sets of linkage points on the physical surfaces of some of the physical objects within the field of view of the user upon which the virtual surfaces are to be applied to, and (ii) surface definitions that define the virtual surfaces;

store the sets of linkage points and the surface definitions to the electronic storage;

receive, via the user interface, selections of one of the physical surfaces visible within the field of view of the user and one of the surface definitions from the electronic storage;

generate, based on the selections, an image of a virtual surface for the selected physical surface to be presented in the augmented reality environment; and

cause the image to be presented in the augmented reality environment via the presentation device so that the virtual surface is covering the selected physical surface in accordance with the set of linkage points for the selected physical surface.

2. The system of claim 1 , wherein the electronic storage stores templates, wherein individual ones of the templates include one of the sets of linkage points and one of the surface definitions.

3. The system of claim 1 , wherein the surface definitions include values to virtual surface parameters.

4. The system of claim 3 , wherein the virtual surface parameters include one or more color, texture, shape, uploaded images, transparency, font, and/or font size.

5. The system of claim 3 , wherein the one or more processors are further configured to:

obtain supplemental user input from the user indicating a request to modify the virtual surface presented in the augmented reality environment;

obtain values of the virtual surface parameter defining a modified version of the virtual surface presented in the augmented reality environment based on the supplemental user input; and

facilitate presentation of the modified version of the virtual surface via the presentation device of the user and/or presentation devices of other users.

6. A method, implemented using one or more processors, for conveying virtual content in an augmented reality environment, via a presentation device, by causing superimposition of images of the virtual content over physical objects and/or physical surroundings visible within a field of view of a user as if the images of the virtual content were present in a real world, wherein the augmented reality environment includes (i) the physical objects and/or the physical surroundings visible within the field of view of the user and (ii) the images of the virtual content presented in conjunction with the physical objects and/or the physical surroundings, the method comprising:

generating a user interface that enables the user to provide user input related to the physical objects visible within the field of view of the user;

obtaining, via the user interface and using the one or more processors, the user input, wherein the user input specifies (i) sets of linkage points on physical surfaces of some of the physical objects in the augmented reality environment visible within the field of view of the user upon which virtual surfaces are to be applied to, and (ii) surface definitions that define the virtual surfaces;

storing the sets of linkage points and the surface definitions to electronic storage;

receiving, via the user interface, selections of one of the physical surfaces visible within the field of view of the user and one of the surface definitions from the electronic storage;

generating, using the one or more processors, an image of a virtual surface for the selected physical surface to be presented in the augmented reality environment based on the selections; and

causing, using the one or more processors, the image to be presented in the augmented reality environment via the presentation device so that the virtual surface is covering the selected physical surface in accordance with the set of linkage points for the selected physical surface.

7. The method of claim 6 , wherein the electronic storage, wherein the electronic storage stores templates, wherein individual ones of the templates include one of the sets of linkage points and one of the surface definitions.

8. The method of claim 6 , wherein the surface definitions include values to virtual surface parameters.

9. The method of claim 8 , wherein the virtual surface parameters include one or more color, texture, shape, uploaded images, transparency, font, and/or font size.

10. The method of claim 8 , further comprising:

obtaining, using the one or more processors, supplemental user input from the user indicating a request to modify the virtual surface presented in the augmented reality environment;

obtaining, using the one or more processors, values of the virtual surface parameter defining a modified version of the virtual surface presented in the augmented reality environment based on the supplemental user input; and

facilitating, using the one or more processors, presentation of the modified version of the virtual surface via the presentation device of the user and/or presentation devices of other users.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 23, 2026
From: HARITON, NICHOLAS T.
To: AR2 PROJECT, LLC
Reel/Frame 073558/0362 →
Continuity (5)
Continuation 17537027 · Nov 29, 2021
Continuation 16855306 · Apr 22, 2020
Continuation 16162947 · Oct 17, 2018
Continuation 15796716 · Oct 27, 2017
Related Publication 20230364512A1 · Nov 16, 2023
References Cited (163)
US 2015027A · Finley · 1935 [cited by applicant]
US 6335731B1 · Yamamoto · 2002 [cited by applicant]
US 7274380B2 · Navab · 2007 [cited by applicant]
US 7373377B2 · Altieri · 2008 [cited by applicant]
US 7774027B2 · Parikh · 2010 [cited by applicant]
US 8139067B2 · Anguelov · 2012 [cited by applicant]
US 8228325B2 · Barbaro Altieri · 2012 [cited by applicant]
US 8275590B2 · Szymczyk · 2012 [cited by applicant]
US 8451266B2 · Hertenstein · 2013 [cited by applicant]
US 8458028B2 · Altieri · 2013 [cited by applicant]
US 8462198B2 · Lin · 2013 [cited by applicant]
US 8473835B2 · Hariton · 2013 [cited by applicant]
US 8490007B1 · Hoffman · 2013 [cited by applicant]
US 8847953B1 · Cho · 2014 [cited by applicant]
US 9001118B2 · Molyneaux · 2015 [cited by applicant]
US 9041622B2 · McCulloch · 2015 [cited by applicant]
US 9230367B2 · Stroila · 2016 [cited by applicant]
US 9251626B2 · Wang · 2016 [cited by applicant]
US 9317670B2 · Gudlavenkatasiva · 2016 [cited by applicant]
US 9323325B2 · Perez · 2016 [cited by applicant]
US 9324318B1 · Bunn · 2016 [cited by applicant]
US 9345957B2 · Geisner · 2016 [cited by applicant]
US 9354702B2 · Mullins · 2016 [cited by applicant]
US 9374788B2 · Singamsetti · 2016 [cited by applicant]
US 9400593B2 · Hariton · 2016 [cited by applicant]
US 9709983B2 · Seydoux · 2017 [cited by applicant]
US 9792584B2 · Hariton · 2017 [cited by applicant]
US 10102659B1 · Hariton · 2018 [cited by applicant]
US 10105601B1 · Hariton · 2018 [cited by applicant]
US 10198871B1 · Hariton · 2019 [cited by applicant]
US 10565767B2 · Hariton · 2020 [cited by applicant]
US 10586396B1 · Hariton · 2020 [cited by applicant]
US 10593121B2 · Hariton · 2020 [cited by applicant]
US 10636188B2 · Hariton · 2020 [cited by applicant]
US 10661170B2 · Hariton · 2020 [cited by applicant]
US 10672170B1 · Hariton · 2020 [cited by applicant]
US 10679427B1 · Hariton · 2020 [cited by applicant]
US 10796467B2 · Hariton · 2020 [cited by applicant]
US 10818096B1 · Hariton · 2020 [cited by applicant]
US 10839409B1 · Aquilla · 2020 [cited by applicant]
US 10846931B1 · Hariton · 2020 [cited by applicant]
US 10861245B2 · Hariton · 2020 [cited by applicant]
US 10867424B2 · Hariton · 2020 [cited by applicant]
US 11107282B1 · Boissière · 2021 [cited by applicant]
US 11120596B2 · Hariton · 2021 [cited by applicant]
US 11145136B2 · Hariton · 2021 [cited by applicant]
US 11185775B2 · Hariton · 2021 [cited by applicant]
US 11198064B2 · Hariton · 2021 [cited by applicant]
US 11200748B2 · Hariton · 2021 [cited by applicant]
US 11249714B2 · Spivack · 2022 [cited by applicant]
US 11487353B2 · Harvey · 2022 [cited by applicant]
US 20040162638A1 · Solomon · 2004 [cited by applicant]
US 20080284777A1 · Altieri · 2008 [cited by applicant]
US 20090288150A1 · Toomim · 2009 [cited by applicant]
US 20110018903A1 · Lapstun · 2011 [cited by applicant]
US 20110292076A1 · Wither · 2011 [cited by applicant]
US 20120001938A1 · Sandberg · 2012 [cited by applicant]
US 20120075430A1 · Ito · 2012 [cited by applicant]
US 20120077584A1 · Sarmenta · 2012 [cited by applicant]
US 20120113223A1 · Hilliges · 2012 [cited by applicant]
US 20120122570A1 · Baronoff · 2012 [cited by applicant]
US 20120209714A1 · Douglas · 2012 [cited by applicant]
US 20120218263A1 · Meier · 2012 [cited by applicant]
US 20130083173A1 · Geisner · 2013 [cited by applicant]
US 20130201215A1 · Martellaro · 2013 [cited by applicant]
US 20130278635A1 · Maggiore · 2013 [cited by applicant]
US 20130326364A1 · Latta · 2013 [cited by applicant]
US 20130342572A1 · Poulos · 2013 [cited by applicant]
US 20140007247A1 · Carter · 2014 [cited by applicant]
US 20140035901A1 · Jiawen · 2014 [cited by applicant]
US 20140091984A1 · Ashbrook · 2014 [cited by applicant]
US 20140122737A1 · Silberstein · 2014 [cited by applicant]
US 20140248950A1 · Tosas Bautista · 2014 [cited by applicant]
US 20140368537A1 · Salter · 2014 [cited by applicant]
US 20150050994A1 · Mangold · 2015 [cited by applicant]
US 20150081417A1 · Golden · 2015 [cited by applicant]
US 20150235423A1 · Tobita · 2015 [cited by applicant]
US 20150274294A1 · Dahlstrom · 2015 [cited by applicant]
US 20150346722A1 · Herz · 2015 [cited by applicant]
US 20150356781A1 · Miller · 2015 [cited by applicant]
US 20150371447A1 · Yasutake · 2015 [cited by applicant]
US 20160026253A1 · Bradski · 2016 [cited by applicant]
US 20160110922A1 · Haring · 2016 [cited by applicant]
US 20160179455A1 · Liu · 2016 [cited by applicant]
US 20160187654A1 · Border · 2016 [cited by applicant]
US 20160203586A1 · Chang · 2016 [cited by applicant]
US 20160210780A1 · Paulovich · 2016 [cited by applicant]
US 20160314716A1 · Grubbs · 2016 [cited by applicant]
US 20170024932A1 · Sugaya · 2017 [cited by applicant]
US 20170038829A1 · Lanier · 2017 [cited by applicant]
US 20170092002A1 · Mullins · 2017 [cited by applicant]
US 20170132842A1 · Morrison · 2017 [cited by applicant]
US 20170277264A1 · Grant · 2017 [cited by applicant]
US 20170337858A1 · Vartanian · 2017 [cited by applicant]
US 20170339372A1 · Valli · 2017 [cited by applicant]
US 20170354875A1 · Marks · 2017 [cited by applicant]
US 20180018825A1 · Kim · 2018 [cited by applicant]
US 20180096507A1 · Valdivia · 2018 [cited by applicant]
US 20180123813A1 · Milevski · 2018 [cited by applicant]
US 20180126620A1 · Talgorn · 2018 [cited by applicant]
US 20180201370A1 · Yang · 2018 [cited by applicant]
US 20180341386A1 · Inomata · 2018 [cited by applicant]
US 20180350144A1 · Rathod · 2018 [cited by applicant]
US 20180373413A1 · Sawaki · 2018 [cited by applicant]
US 20190023418A1 · Ro · 2019 [cited by applicant]
US 20190087995A1 · Hariton · 2019 [cited by applicant]
US 20190126149A1 · Hariton · 2019 [cited by applicant]
US 20190206131A1 · Kamal · 2019 [cited by examiner]
US 20190217202A1 · Komori · 2019 [cited by applicant]
US 20190244404A1 · Goslin · 2019 [cited by applicant]
US 20190248485A1 · Ulaganathan · 2019 [cited by applicant]
US 20190251720A1 · Hariton · 2019 [cited by applicant]
US 20190324539A1 · Wu · 2019 [cited by applicant]
US 20190329136A1 · Koyama · 2019 [cited by applicant]
US 20190333277A1 · Hariton · 2019 [cited by applicant]
US 20190391724A1 · Holz · 2019 [cited by applicant]
US 20200092287A1 · Cano · 2020 [cited by applicant]
US 20200097246A1 · McCoy · 2020 [cited by applicant]
US 20200110560A1 · Hariton · 2020 [cited by applicant]
US 20200151930A1 · Hariton · 2020 [cited by applicant]
US 20200175772A1 · Hariton · 2020 [cited by applicant]
US 20200222805A1 · Hariton · 2020 [cited by applicant]
US 20200226809A1 · Hariton · 2020 [cited by applicant]
US 20200226810A1 · Hariton · 2020 [cited by applicant]
US 20200246699A1 · Hariton · 2020 [cited by applicant]
US 20200349767A1 · Hariton · 2020 [cited by applicant]
US 20200349771A1 · Hariton · 2020 [cited by applicant]
US 20210019927A1 · Hariton · 2021 [cited by applicant]
US 20210019950A1 · Hariton · 2021 [cited by applicant]
US 20210035372A1 · Hariton · 2021 [cited by applicant]
US 20210090340A1 · Hariton · 2021 [cited by applicant]
US 20210104084A1 · Hariton · 2021 [cited by applicant]
US 20210220737A1 · Fish · 2021 [cited by applicant]
US 20210233323A1 · Guerra Filho · 2021 [cited by examiner]
US 20210407208A1 · Hariton · 2021 [cited by applicant]
US 20220051461A1 · Hariton · 2022 [cited by applicant]
US 20220080311A1 · Hariton · 2022 [cited by applicant]
US 20220096929A1 · Hariton · 2022 [cited by applicant]
US 20220101616A1 · Hariton · 2022 [cited by applicant]
US 20230093062A1 · Hariton · 2023 [cited by applicant]
US 20230368433A1 · Hariton · 2023 [cited by applicant]
WO 2014147686 · 2014 [cited by applicant]
WO 2018055459 · 2018 [cited by applicant]
Carolyn Giardina, New Venture Aims to Creat AI-Driven Digital “Copies” of Hollywood Stars; published Jun. 28, 2017; [retrieved Sep. 18, 2017] retrieved from the Internet http://www.hollywoodreporter.com/behind-screen/ne… [cited by applicant]
David Kariuki, 7 firms with face tracking tech for better VR avatars, Hypergrid Business; Nov. 20, 2016, retrieved from the internet http://www.hypergridbusiness.com/2016/11/ai-emotion-and-facial-tracking-to-help-make-v… [cited by applicant]
E. Matchar, “When Work Becomes a Game,” published Feb. 16, 2016, downloaded from https://www.smithsonianmag.com/ innovation/when-work-becomes-game-180958145/ (Year: 2016) 6 pages. [cited by applicant]
Josh Constine, Augmented Reality for Trying on Makeup Is a Booming Business Posted Jan. 19, 2016 by Josh Constine (@joshconstine)—TechCrunch, retrieved from the Internet https://techcrunch.com/2016/01/19/facial-precogni… [cited by applicant]
Kato, et al., Marker Tracking and HMD Calibration for a Video-based Augmented Reality Conferencing System, in Proceedings of the 2nd International Workshop on Augmented Reality (IWAR 99). Oct. 1999, [retrieved Dec. 7, 2… [cited by applicant]
Mashalkar, et al., Personalized Animatable Avatars from Depth Data, Joint Virtual Reality Conference of EGVE—EuroVR (2013), [retrieved Sep. 18, 2017] retrieved from the internet https://www.cse.iitb.ac.in/˜paragc/pubs/p… [cited by applicant]
Mindshow promotional video, Mindshow Official Trailer—Create Animated Movies in Virtual Reality, YouTube, Published Apr. 14, 2017 [retrieved Sep. 18, 2017] retrieved from the Internet https://www.youtube.com/watch?v=UCV… [cited by applicant]
Mindshow Webpage, About | Mindshow, [retrieved Sep. 18, 2017] retrieved from the Internet https://mindshow.com/about/ (6 pages). [cited by applicant]
Mindshow Webpage, How it Works, [retrieved Sep. 18, 2017] retrieved from the Internet https://mindshow.com/#how-it-works (4 pages). [cited by applicant]
Neal Stephenson, Snow Crash; New York : Del Rey, 1992, (pp. 44-50, 66, 70, 76, 79, 562, and 563). [cited by applicant]
R. Schuler, “How Does the Internet work,” published 2002, downloaded from https://web.stanford.edu/class/msande91si/www-spr04/readings/week1/InternetWhitepaper.htm (Year: 2002); 5 pages. [cited by applicant]
Rekimoto, et al., CyberCode: Designing Augmented Reality Environments with Visual Tags, Interaction Laboratory, Sony Computer Science Laboratories, Inc. Mar. 14, 2013, retrieved from the Internet http://citeseerx.ist.ps… [cited by applicant]
Roma: Interactive Fabrication with Augmented Reality and a Robotic Printer, YouTube video, published Feb. 11, 2018, downloaded from https://www.youtube.com/watch?time_continue=148&v=K_wWuYD1Fkg&feature=emb_logo (Year: 2… [cited by applicant]
T. Klosowski, The Psychology of Gamification: Can Apps Keep You Motivated?, Lifehacker website, published Feb. 13, 2014, 16 pages, downloaded at https://lifehacker.com/the-psychology-of-gamification-can-apps-keep-you-mo… [cited by applicant]
Techopedia website, “Machine Code,” published Jun. 16, 2017, downloaded from https://www.techopedia.com/definition/8179/ machine-code-mc (Year: 2017). [cited by applicant]
Wolfgang Broll, “Populating the Internet: supporting multiple users and shared applications with VRML.” Proceedings of the second symposium on Virtual reality modeling language. 1997. (Year: 1997) 8 pages. [cited by applicant]
Woodrow Barfield, Fundamentals of Wearable Computers and Augmented Reality, Second Edition (p. ix). CRC Press. Kindle Edition. Chapel Hill, North Carolina, Jul. 29, 2015, (pp. 59, 60, 195, 196, 227). [cited by applicant]
YouTube video, “Drone Grafitti 2013-2016,” published Jul. 31, 2016, downloaded from https://www.youtube.com/watch?v= Ep5tPWphKFY (Year: 2016). [cited by applicant]
A. Oppermann, “What is Nanorobotics?”, builtin.com website, dated Mar. 3, 2023, downloaded from https://builtin.com/robotics/ nanorobotics (Year: 2023). [cited by applicant]
R. Walters, “Real life Construction quadcopter robots being developed,” Extremetech website, dated Dec. 2, 2011, downloaded from https://www.extremetech.com/extreme/107217-real-life-constructicon-quadcopter-robots-being… [cited by applicant]