IP Library › Granted Patent US 12,289,561
Granted Patent B2
US 12,289,561 · App. 18/500,445 · Granted Apr 29, 2025

Parallel video call and artificial reality spaces

Inventors: Michael James Lebeau (Amsterdam, NL); Björn Wanbo (London, GB); Fábio Resende (San Francisco, CA); Mark Rabkin (Menlo Park, CA); Vesa Petteri Rantanen (London, GB)
Assignee: Meta Platforms Technologies, LLC
H04N7/157G06Q10/1093H04L12/1818H04L12/1822
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,289,561
App. No.
18/500,445
Granted
Apr 29, 2025
Kind
B2
Abstract

Aspects of the present disclosure are directed to a VC/XR connection system that can establish and administer an XR space for a video call. The VC/XR connection system allows users to easily transition from a typical video call experience to the XR space, simply by putting on her artificial reality device. The VC/XR connection system can identify calendared video call events, establish corresponding XR spaces, and create a link between the video call and the XR space. Invitees to the video call that don an artificial reality device can be automatically taken into the XR space. The XR space can A) connect to the video call as a call participant, allowing the video call participants to see into the XR space and B) show a feed of the video call in the XR space, allowing users in the XR space to see the video call participants.

Claims (55)

1. A method for integrating a video call with an XR space, provided via an XR device, the method comprising:

receiving, at the XR device, an indication of the video call, wherein two or more video call users participate in the video call, and wherein at least one of the two or more video call users participate in the video call via a non-XR device; and

adding, to a defined layout of the XR space, representations of the two or more video call users, wherein adding the representations comprises:

receiving one or more feeds including the representations of the two or more video call users; and

arranging, in the defined layout of the XR space, separate virtual elements for each respective video call user, of the two or more video call users, wherein each separate virtual element corresponds to a respective representation of the representations from the one or more feeds.

2. The method of claim 1 , further comprising:

providing a feed representing a portion of the XR space to the non-XR device, wherein the feed, when displayed on a display of the non-XR device, allows a first video call user associated with the non-XR device to see the portion of the XR space.

3. The method of claim 1 , wherein the defined layout is based on a contextual factor associated with the video call.

4. The method of claim 3 , wherein the contextual factor is based on the number of video call users.

5. The method of claim 1 , further comprising:

receiving, at the XR device, an input by a user of the XR device,

wherein the input is in response to an interaction, by the XR device user, associated with the indication, and

wherein the representations are added to the defined layout in response to the received input.

6. The method of claim 1 , further comprising:

adding, to the defined layout of the XR space, a representation of a new user who is added to the video call.

7. The method of claim 6 , wherein the new user joins the video call via another XR device.

8. The method of claim 1 , further comprising:

moving a first virtual element associated with a first video call user to another location in the XR space.

9. A non-transitory computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process for integrating a video call with an XR space, provided via an XR device, the process comprising:

receiving, at the XR device, an indication of the video call, wherein two or more video call users participate in the video call, and wherein at least one of the two or more video call users participate in the video call via a non-XR device; and

adding, to a defined layout of the XR space, representations of the two or more video call users, wherein adding the representations comprises:

receiving one or more feeds including the representations of the two or more video call users; and

arranging, in the defined layout of the XR space, separate virtual elements for each respective video call user, of the two or more video call users, wherein each separate virtual element corresponds to a respective representation of the representations from the one or more feeds.

10. The non-transitory computer-readable storage medium of claim 9 , wherein the process further comprises:

providing a feed representing a portion of the XR space to the non-XR device, wherein the feed, when displayed on a display of the non-XR device, allows a first video call user associated with the non-XR device to see the portion of the XR space.

11. The non-transitory computer-readable storage medium of claim 9 , wherein the process further comprises:

receiving, at the XR device, an input by a user of the XR device,

wherein the input is in response to an interaction, by the XR device user, associated with the indication, and

wherein the representations are added to the defined layout in response to the received input.

12. The non-transitory computer-readable storage medium of claim 9 , wherein the process further comprises:

adding, to the defined layout of the XR space, a representation of a new user who is added to the video call.

13. The non-transitory computer-readable storage medium of claim 9 , wherein the process further comprises:

moving a first virtual element associated with a first video call user to another location in the XR space.

14. A computing system for integrating a video call with an XR space, provided via an XR device, the computing system comprising:

one or more processors; and

one or more memories storing instructions that, when executed by at least one of the one or more processors, cause the computing system to:

receive, at the XR device, an indication of the video call, wherein two or more video call users participate in the video call, and wherein at least one of the two or more video call users participate in the video call via a non-XR device; and

add, to a defined layout of the XR space, representations of the two or more video call users, wherein adding the representations comprises causing the computing system to:

receive one or more feeds including the representations of the two or more video call users; and

arrange, in the defined layout of the XR space, separate virtual elements for each respective video call user, of the two or more video call users, wherein each separate virtual element corresponds to a respective representation of the representations from the one or more feeds.

15. The computing system of claim 14 , wherein the instructions, when executed by at least one of the one or more processors, further cause the computing system to:

provide a feed representing a portion of the XR space to the non-XR device, wherein the feed, when displayed on a display of the non-XR device, allows a first video call user associated with the non-XR device to see the portion of the XR space.

16. The computing system of claim 14 ,

wherein the defined layout is based on a contextual factor associated with the video call, and

wherein the contextual factor is based on the number of video call users.

17. The computing system of claim 14 , wherein the instructions, when executed by at least one of the one or more processors, further cause the computing system to:

receive, at the XR device, an input by a user of the XR device,

wherein the input is in response to an interaction, by the XR device user, associated with the indication, and

wherein the representations are added to the defined layout in response to the received input.

18. The computing system of claim 14 , wherein the instructions, when executed by at least one of the one or more processors, further cause the computing system to:

add, to the defined layout of the XR space, a representation of a new user who is added to the video call,

wherein the new user joins the video call via another XR device.

19. The non-transitory computer-readable storage medium of claim 18 , wherein the defined layout is based on a contextual factor associated with the video call.

20. The computing system of claim 14 , wherein the instructions, when executed by at least one of the one or more processors, further cause the computing system to:

move a first virtual element associated with a first video call user to another location in the XR space.

Continuity (2)
Continuation 17466528 · Sep 3, 2021
Related Publication 20240214524A1 · Jun 27, 2024
References Cited (157)
US 6559863B1 · Megiddo · 2003 [cited by examiner]
US 8537196B2 · Hegde et al. · 2013 [cited by applicant]
US 8581958B2 · Baker et al. · 2013 [cited by applicant]
US 8675067B2 · Chou et al. · 2014 [cited by applicant]
US 9338404B1 · Egeler et al. · 2016 [cited by applicant]
US 9401095B2 · Kubota et al. · 2016 [cited by applicant]
US 9460340B2 · Kauffmann et al. · 2016 [cited by applicant]
US 9841814B1 · Kallmeyer et al. · 2017 [cited by applicant]
US 9959676B2 · Barzuza et al. · 2018 [cited by applicant]
US 10182210B1 · Goetzinger, Jr. · 2019 [cited by examiner]
US 10212428B2 · Trepte · 2019 [cited by applicant]
US 10298587B2 · Hook et al. · 2019 [cited by applicant]
US 10311383B2 · Holmes et al. · 2019 [cited by applicant]
US 10403050B1 · Beall · 2019 [cited by examiner]
US 10499033B2 · Pesonen · 2019 [cited by applicant]
US 10554931B1 · Zavesky et al. · 2020 [cited by applicant]
US 10582191B1 · Marchak, Jr. et al. · 2020 [cited by applicant]
US 10664772B1 · Poel et al. · 2020 [cited by applicant]
US 10701318B2 · Valli · 2020 [cited by applicant]
US 10917613B1 · Chiarella et al. · 2021 [cited by applicant]
US 10952006B1 · Krol et al. · 2021 [cited by applicant]
US 11140361B1 · Krol et al. · 2021 [cited by applicant]
US 11302063B2 · Cabral et al. · 2022 [cited by applicant]
US 11302085B2 · LeBeau et al. · 2022 [cited by applicant]
US 11394925B1 · Faulkner et al. · 2022 [cited by applicant]
US 11456887B1 · McCracken et al. · 2022 [cited by applicant]
US 11475634B2 · Atlas et al. · 2022 [cited by applicant]
US 11528312B2 · Oyman · 2022 [cited by applicant]
US 11556172B1 · Stevens et al. · 2023 [cited by applicant]
US 11563779B2 · Copley et al. · 2023 [cited by applicant]
US 11831814B2 · Lebeau et al. · 2023 [cited by applicant]
US 11886625B1 · Trzynadlowski et al. · 2024 [cited by applicant]
US 11921970B1 · Hoover et al. · 2024 [cited by applicant]
US 20020158873A1 · Williamson · 2002 [cited by applicant]
US 20070279484A1 · Derocher et al. · 2007 [cited by applicant]
US 20090288007A1 · Leacock et al. · 2009 [cited by applicant]
US 20090300516A1 · Jerrard-Dunne et al. · 2009 [cited by applicant]
US 20100085416A1 · Hegde · 2010 [cited by examiner]
US 20110107270A1 · Wang et al. · 2011 [cited by applicant]
US 20120192088A1 · Sauriol et al. · 2012 [cited by applicant]
US 20120204118A1 · Lefar · 2012 [cited by examiner]
US 20120246582A1 · Leacock et al. · 2012 [cited by applicant]
US 20120317501A1 · Milou · 2012 [cited by applicant]
US 20130024785A1 · Van Wie · 2013 [cited by applicant]
US 20130031475A1 · Maor et al. · 2013 [cited by applicant]
US 20140085316A1 · Narayanan · 2014 [cited by applicant]
US 20140085406A1 · Narayanan · 2014 [cited by applicant]
US 20140096036A1 · Mohler · 2014 [cited by applicant]
US 20150215581A1 · Barzuza et al. · 2015 [cited by applicant]
US 20150271220A1 · Kleiner et al. · 2015 [cited by applicant]
US 20150279044A1 · Kim et al. · 2015 [cited by applicant]
US 20160050394A1 · Segal · 2016 [cited by applicant]
US 20160093108A1 · Mao et al. · 2016 [cited by applicant]
US 20160132221A1 · Lasser et al. · 2016 [cited by applicant]
US 20160155187A1 · Paulrajan et al. · 2016 [cited by applicant]
US 20160156584A1 · Hum et al. · 2016 [cited by applicant]
US 20160163070A1 · Leacock et al. · 2016 [cited by applicant]
US 20160353062A1 · Ono et al. · 2016 [cited by applicant]
US 20170257405A1 · Lo et al. · 2017 [cited by applicant]
US 20170357917A1 · Holmes et al. · 2017 [cited by applicant]
US 20180063480A1 · Luks et al. · 2018 [cited by applicant]
US 20180095635A1 · Valdivia et al. · 2018 [cited by applicant]
US 20180098059A1 · Valdivia et al. · 2018 [cited by applicant]
US 20180101989A1 · Frueh et al. · 2018 [cited by applicant]
US 20180139246A1 · Rosenberg · 2018 [cited by applicant]
US 20180144212A1 · Burgos et al. · 2018 [cited by applicant]
US 20180158246A1 · Grau et al. · 2018 [cited by applicant]
US 20180227138A1 · Faulkner et al. · 2018 [cited by applicant]
US 20180234671A1 · Yang et al. · 2018 [cited by applicant]
US 20180356885A1 · Ross et al. · 2018 [cited by applicant]
US 20190042640A1 · Valeski · 2019 [cited by applicant]
US 20190045157A1 · Venshtain et al. · 2019 [cited by applicant]
US 20190058870A1 · Rowell et al. · 2019 [cited by applicant]
US 20190130629A1 · Chand et al. · 2019 [cited by applicant]
US 20190171354A1 · Dascola et al. · 2019 [cited by applicant]
US 20190253667A1 · Valli · 2019 [cited by examiner]
US 20190279424A1 · Clausen et al. · 2019 [cited by applicant]
US 20190310757A1 · Lee · 2019 [cited by examiner]
US 20190325658A1 · Park et al. · 2019 [cited by applicant]
US 20190327392A1 · Sarkar · 2019 [cited by examiner]
US 20190340828A1 · Harvey · 2019 [cited by applicant]
US 20190346522A1 · Botnar et al. · 2019 [cited by applicant]
US 20190371060A1 · Energin et al. · 2019 [cited by applicant]
US 20190379750A1 · Zamora Duran et al. · 2019 [cited by applicant]
US 20200090350A1 · Cho et al. · 2020 [cited by applicant]
US 20200099891A1 · Valli et al. · 2020 [cited by applicant]
US 20200118342A1 · Varshney et al. · 2020 [cited by applicant]
US 20200128033A1 · Kawakita · 2020 [cited by applicant]
US 20200142475A1 · Paez et al. · 2020 [cited by applicant]
US 20200154166A1 · Rakshit et al. · 2020 [cited by applicant]
US 20200279411A1 · Atria et al. · 2020 [cited by applicant]
US 20200294000A1 · Sexauer · 2020 [cited by examiner]
US 20200296327A1 · Karafin et al. · 2020 [cited by applicant]
US 20200314385A1 · Kang · 2020 [cited by examiner]
US 20200329214A1 · Ahn et al. · 2020 [cited by applicant]
US 20200371665A1 · Clausen et al. · 2020 [cited by applicant]
US 20200372140A1 · Jaber · 2020 [cited by examiner]
US 20200396266A1 · Goel · 2020 [cited by applicant]
US 20210019541A1 · Wang et al. · 2021 [cited by applicant]
US 20210029326A1 · Oyman et al. · 2021 [cited by applicant]
US 20210099433A1 · Soryal et al. · 2021 [cited by applicant]
US 20210120066A1 · Oyman et al. · 2021 [cited by applicant]
US 20210149627A1 · Jang et al. · 2021 [cited by applicant]
US 20210248727A1 · Fisher et al. · 2021 [cited by applicant]
US 20210263593A1 · Lacey · 2021 [cited by applicant]
US 20210281802A1 · Kirisken · 2021 [cited by examiner]
US 20210336784A1 · Athlur · 2021 [cited by examiner]
US 20210385412A1 · Matula et al. · 2021 [cited by applicant]
US 20210390767A1 · Johnson et al. · 2021 [cited by applicant]
US 20210392175A1 · Gronau et al. · 2021 [cited by applicant]
US 20210392296A1 · Rabinovich et al. · 2021 [cited by applicant]
US 20210407520A1 · Neckermann et al. · 2021 [cited by applicant]
US 20220084288A1 · LeBeau et al. · 2022 [cited by applicant]
US 20220086167A1 · LeBeau et al. · 2022 [cited by applicant]
US 20220086205A1 · LeBeau et al. · 2022 [cited by applicant]
US 20220103386A1 · Jensen · 2022 [cited by examiner]
US 20220109810A1 · Kancharlawar et al. · 2022 [cited by applicant]
US 20220116431A1 · Mayfield · 2022 [cited by examiner]
US 20220150083A1 · Faulkner · 2022 [cited by applicant]
US 20220157342A1 · Kliushkin et al. · 2022 [cited by applicant]
US 20220172444A1 · LeBeau et al. · 2022 [cited by applicant]
US 20220174108A1 · Oyman · 2022 [cited by applicant]
US 20220200979A1 · Wenzel · 2022 [cited by applicant]
US 20220229535A1 · Evangelista · 2022 [cited by examiner]
US 20220385490A1 · Lin · 2022 [cited by examiner]
US 20220417308A1 · Oyman · 2022 [cited by applicant]
US 20230082461A1 · Gal · 2023 [cited by examiner]
US 20230198745A1 · Vansteenkiste et al. · 2023 [cited by applicant]
US 20230244799A1 · Sharma · 2023 [cited by applicant]
US 20230281929A1 · Atlas et al. · 2023 [cited by applicant]
US 20230412724A1 · Ma et al. · 2023 [cited by applicant]
US 20240070957A1 · Cross · 2024 [cited by examiner]
CA 3039347A1 · 2018 [cited by applicant]
WO 2020117657A1 · 2020 [cited by applicant]
Gunkel et al., 360-Degree Photo-realistic VR Conferencing, 2019, IEEE, 2 pages. (Year: 2019). [cited by examiner]
Jikadra et al., Video Calling with Augmented Reality Using WebRTC API, 2019, IEEE, 5 pages (Year: 2019). [cited by examiner]
Gupta K., et al., “Do You See What I See? The Effect ofGaze Tracking on Task Space Remote Collaboration,” IEEE Transactions onVisualization and Computer Graphics, Nov. 2016, vol. 22, No. 11, pp. 2413-2422, DOI: 10.1109/… [cited by applicant]
International Search Report and Written Opinion for International Application No. PCT/US2021/038992, mailed Oct. 29, 2021, 16 pages. [cited by applicant]
International Search Report and Written Opinion for International Application No. PCT/US2023/012192, mailed May 4, 2023, 8 pages. [cited by applicant]
Jikadra R., et al., “Video Calling with Augmented Reality Using WebRTC API,” 2019 6th International Conference on Computing for Sustainable Global Development (INDIACom), Mar. 13, 2019, 5 pages. [cited by applicant]
Shanmugam M., et al., “Research Opportunities on Virtual Reality and Augmented Reality: A Survey,” 2019 IEEE International Conference on System, Computation, Automation and Networking (ICSCAN), Mar. 29, 2019, 6 pages. [cited by applicant]
Unknown., “A Better Way to Meet Online,” Gather, https://www.gather.town/ , Last Accessed Oct. 11, 2021. [cited by applicant]
Baxter D.B., “Creating Treemap Chart Using JavaScript,” AnyChart News, Aug. 1, 2019, 12 Pages. [cited by applicant]
eBook Reader, “Viewing Popular Highlights on Kindles,” [Online], Feb. 15, 2018 [Retrieved on Aug. 5, 2020], 11 Pages, Retrieved from Internet: URL: https://blog.the-ebook-reader.com/2018/02/15/viewing-popular-highlights… [cited by applicant]
Garofalakis J., et al., “Personalized Web Search by Constructing Semantic Clusters of User Profiles,” Springer Link [Online], 2008 [Retrieved on Aug. 5, 2020], 6 Pages, Retrieved on Internet: URL: https://link.springer.… [cited by applicant]
Gauch S., et al., “Ontology-Based User Profiles for Search and Browsing,” [Online], 2002 [Retrieved on Aug. 5, 2020], 6 Pages, Retrieved from Internet: URL: https://www.semanticscholar.org/paper/Ontology-Based-User-Prof… [cited by applicant]
International Preliminary Report on Patentability for International Application No. PCT/US2022/042452, mailed Mar. 14, 2024, 6 pages. [cited by applicant]
“Pinterest Communities What are they and How can you Join?,” Ivory Mix, Aug. 5, 2020, 19 Pages, Retrieved from Internet: URL: https://ivorymix.com/pinterest-communities/. [cited by applicant]
Schmeiser L., “Microsoft Analytics Aims to Analyze Worker Habits, Point to More Productivity,” [Online], Jul. 5, 2017 [Retrieved on Aug. 5, 2020], 3 Pages, Retrieved from Internet: URL: https://www.itprotoday.com/office… [cited by applicant]
Semantic Arts, “White Paper: Semantic Profiling,” [Online], Jan. 21, 2015 [Retrieved on Aug. 5, 2020], 11 Pages, Retrieved from Internet: https://www.semanticarts.com/white-paper-semantic-profiling/. [cited by applicant]
Song X., et al., “A Scalable and Statistically Robust Beam Alignment Technique for mm-Wave Systems,” May 8, 2018, 13 pages. [cited by applicant]
Tarpani G., “What is Google Wave?,” Macworld [Online], Feb. 22, 2010 [Retrieved on Aug. 5, 2020], 7 Pages, Retrieved from Internet: URL: https://www.macworld.com/article/1146555/whatisgooglewave.html. [cited by applicant]
Wanders I., “Build your Own Knowledge Graph: From Unstructured Dark Data to Valuable Business Insights,” Vector Consulting [Online], Oct. 18, 2018 [Retrieved on 2020-80-05], 8 Pages, Retrieved from Internet: https://med… [cited by applicant]
Wikipedia, The Free Encyclopedia, “Google Wave,” [Online], Aug. 5, 2020, 1 Page, Retrieved from Internet: URL: https://simple.wikipedia.org/wiki/Google_Wave. [cited by applicant]
Wikipedia, The Free Encyclopedia, “Treemapping,” Aug. 5, 2020, 13 Pages, Retrieved from Internet: URL: https://en.wikipedia.org/wiki/Treemapping. [cited by applicant]
Wikipedia: The Free Encyclopedia, “Unconference,” Aug. 5, 2020, 3 Pages, Retrieved from Internet: URL: https://en.wikipedia.org/wiki/Unconference. [cited by applicant]
International Preliminary Report on Patentability for International Application No. PCT/US2023/012192, mailed Aug. 15, 2024, 6 pages. [cited by applicant]