IP Library › Granted Patent US 12,608,075
Granted Patent B2
US 12,608,075 · App. 18/500,619 · Granted Apr 21, 2026

VR partial passthrough using contour approximation

Inventors: Sho Ayuba (Tokyo, JP); Chisa Koriyama (Bunkyo-ku, JP); Kazuki Matsumaru (Higashimurayama, JP); Akinobu Morishima (Tokyo, JP)
Assignee: International Business Machines Corporation
G06F3/011G06T19/006G06T19/20G06V10/752G06V20/64G06T2219/2004
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,608,075
App. No.
18/500,619
Granted
Apr 21, 2026
Kind
B2
Abstract

According to at least one embodiment, a method, a computer system, and a computer program product for virtual reality is provided. The present invention may include detecting a selection of a real-world object via a passthrough function of a virtual reality headset; analyzing a contour of the selected real-world object via a contour approximation method; creating a first layer that maintains passthrough function within the analyzed contour of the selected real-world object; creating a second layer that comprises a virtual simulated environment and one or more virtual objects; and displaying the first layer on top of the second layer by overwrapping the first layer and the second layer.

Claims (43)

1 . A processor-implemented method for virtual reality, the method comprising:

detecting a selection of a real-world object, by a user, via a passthrough function of a virtual reality headset;

analyzing a contour of the selected real-world object via a contour approximation method;

creating a first layer that maintains passthrough function within the analyzed contour of the selected real-world object, wherein the creating the first layer includes cutting an image of the selected real-world object precisely around a border of the selected real-world object during the analyzing;

creating a second layer that comprises a virtual simulated environment and one or more virtual objects;

displaying the first layer on top of the second layer by overwrapping the first layer and the second layer; and

continuously displaying the selected real-world object in the virtual simulated environment, wherein the continuous display of the selected real-world object comprises matching a position of the selected real-world object in the virtual simulated environment to a corresponding position in a real-world surrounding environment.

2 . The method of claim 1 , further comprising:

adjusting a position of a virtual ground of the second layer, upon detection of a mismatch between the virtual ground of the second layer and a real-world ground of the real-world surrounding environment, wherein the adjusting changes the virtual ground of the second layer to match the real-world ground.

3 . The method of claim 1 , further comprising:

identifying one or more real-world objects in a real-world surrounding environment using object recognition.

4 . The method of claim 1 , wherein the detecting of the selection of a real-world object via the passthrough function of the virtual reality headset further comprises a selection of a pre-registered real-world object.

5 . The method of claim 1 , wherein the continuously displaying of the selected real-world object in the virtual simulated environment further comprises displaying one or more body parts of a virtual headset wearer.

6 . The method of claim 1 , wherein the continuously displaying of the selected real-world object in the virtual simulated environment further comprises highlighting the selected real-world object in the passthrough layer using pattern recognition.

7 . A computer system for virtual reality, the computer system comprising:

one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage medium, and program instructions stored on at least one of the one or more tangible storage medium for execution by at least one of the one or more processors via at least one of the one or more memories, wherein the computer system is capable of performing a method comprising:

detecting a selection of a real-world object, by a user, via a passthrough function of a virtual reality headset;

analyzing a contour of the selected real-world object via a contour approximation method;

creating a first layer that maintains passthrough function within the analyzed contour of the selected real-world object, wherein the creating the first layer includes cutting an image of the selected real-world object precisely around a border of the selected real-world object during the analyzing;

creating a second layer that comprises a virtual simulated environment and one or more virtual objects;

displaying the first layer on top of the second layer by overwrapping the first layer and the second layer; and

continuously displaying the selected real-world object in the virtual simulated environment, wherein the continuous display of the selected real-world object comprises matching a position of the selected real-world object in the virtual simulated environment to a corresponding position in a real-world surrounding environment.

8 . The computer system of claim 7 , further comprising:

adjusting a position of a virtual ground of the second layer, upon detection of a mismatch between the virtual ground of the second layer and a real-world ground of the real-world surrounding environment, wherein the adjusting changes the virtual ground of the second layer to match the real-world ground.

9 . The computer system of claim 7 , further comprising:

identifying one or more real-world objects in a real-world surrounding environment using object recognition.

10 . The computer system of claim 7 , wherein the detecting of the selection of a real-world object via the passthrough function of the virtual reality headset further comprises a selection of a pre-registered real-world object.

11 . The computer system of claim 7 , wherein the continuously displaying of the selected real-world object in the virtual simulated environment further comprises displaying one or more body parts of a virtual headset wearer.

12 . The computer system of claim 7 , wherein the continuously displaying of the selected real-world object in the virtual simulated environment further comprises highlighting the selected real-world object in the passthrough layer using pattern recognition.

13 . A computer program product for virtual reality, the computer program product comprising:

one or more computer-readable tangible storage medium and program instructions stored on at least one of the one or more tangible storage medium, the program instructions executable by a processor to cause the processor to perform a method comprising:

detecting a selection of a real-world object, by a user, via a passthrough function of a virtual reality headset;

analyzing a contour of the selected real-world object via a contour approximation method;

creating a first layer that maintains passthrough function within the analyzed contour of the selected real-world object, wherein the creating the first layer includes cutting an image of the selected real-world object precisely around a border of the selected real-world object during the analyzing;

creating a second layer that comprises a virtual simulated environment and one or more virtual objects;

displaying the first layer on top of the second layer by overwrapping the first layer and the second layer; and

continuously displaying the selected real-world object in the virtual simulated environment, wherein the continuous display of the selected real-world object comprises matching a position of the selected real-world object in the virtual simulated environment to a corresponding position in a real-world surrounding environment.

14 . The computer program product of claim 13 , further comprising:

adjusting a position of a virtual ground of the second layer, upon detection of a mismatch between the virtual ground of the second layer and a real-world ground of the real-world surrounding environment, wherein the adjusting changes the virtual ground of the second layer to match the real-world ground.

15 . The computer program product of claim 13 , further comprising:

identifying one or more real-world objects in a real-world surrounding environment using object recognition.

16 . The computer program product of claim 13 , wherein the detecting of the selection of a real-world object via the passthrough function of the virtual reality headset further comprises a selection of a pre-registered real-world object.

17 . The computer program product of claim 13 , wherein the continuously displaying of the selected real-world object in the virtual simulated environment further comprises displaying one or more body parts of a virtual headset wearer.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 2, 2023
From: AYUBA, SHO; KORIYAMA, CHISA; MATSUMARU, KAZUKI; MORISHIMA, AKINOBU
To: INTERNATIONAL BUSINESS MACHINES CORPORATION
Reel/Frame 065437/0726 →
Continuity (1)
Related Publication 20250147573A1 · May 8, 2025
References Cited (25)
US 11009716B2 · Kiemele · 2021 [cited by applicant]
US 20160027215A1 · Burns · 2016 [cited by applicant]
US 20180004286A1 · Chen · 2018 [cited by applicant]
US 20190020843A1 · Reif · 2019 [cited by applicant]
US 20190385372A1 · Cartwright · 2019 [cited by applicant]
US 20210132683A1 · Herling · 2021 [cited by examiner]
US 20240104849A1 · Nie · 2024 [cited by examiner]
US 20240303940A1 · Bataille · 2024 [cited by examiner]
US 20250104363A1 · Faaborg · 2025 [cited by examiner]
US 20250111619A1 · Singh · 2025 [cited by examiner]
WO 2017201191A1 · 2017 [cited by applicant]
WO 2020219601A1 · 2020 [cited by applicant]
Intel, “Demystifying the Virtual Reality Landscape,” Intel [online], [accessed on Jul. 24, 2023], 4 pages, Retrieved from the Internet: <URL: https://www.intel.com/content/www/us/en/tech-tips-and-tricks/virtual-reality-… [cited by applicant]
Melnick, “Microsoft Files Patent to Bring Real-World Objects to VR,” VRScout [news], May 19, 2021 [accessed on Jul. 24, 2023], 8 pages, Retrieved from the Internet: <URL: https://vrscout.com/news/microsoft-patent-real-w… [cited by applicant]
Nakano, et al., “Ukemochi: A Video See-through Food Overlay System for Eating Experience in the Metaverse,” CHI EA '22: Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems, Apr. 2022, 8 p… [cited by applicant]
Nishida, I was surprised at the practicality when I wrote a manuscript in a VR space Advantages and problems of “virtual work”, ITmedia News [online], Nov. 26, 2021 [accessed on Jul. 24, 2023], 6 pages, Retrieved from t… [cited by applicant]
Oculus VR, “Mixed Reality with Passthrough,” Meta Quest [online], Jul. 23, 2021 [Jul. 24, 2023], 4 pages, Retrieved from the Internet: <URL: https://developer.oculus.com/blog/mixed-reality-with-passthrough/>. [cited by applicant]
Pai, “Oculus Passthrough: Surface Projected and Selective Passthrough,” immersive insiders [online blog], Jun. 2022 [accessed on Jul. 24, 2023], 17 pages, Retrieved from the Internet: <URL: https://blog.immersive-inside… [cited by applicant]
Palli, “Real World Meets Virtual World? Meta to Introduce Mixed Reality Soon,” Mashable India [online], Feb. 22, 2022 [accessed on Jul. 24, 2023], 7 pages, Retrieved from the Internet: <URL: https://in.mashable.com/tech… [cited by applicant]
Rubin, “Meta (Facebook)'s VR conferencing app “Horizon Workrooms” will be the third pillar of the Metaverse,” Wired [online], Nov. 10, 2021 [accessed on Jul. 24, 2023], 6 pages, Retrieved from Internet: <URL: https://wi… [cited by applicant]
Screen captures from YouTube video clip entitled “OpenCV GSOC 2014,” 2 pages, uploaded on Mar. 23, 2015 by OpenCV, Retrieved from the Internet: <URL: https://www.youtube.com/watch?v=3f76HCHJJRA>. [cited by applicant]
Strange, “Disney experiment lets you catch real objects while in VR,” Mashable [online], Mar. 20, 2017 [accessed on Jul. 24, 2023], 8 pages, Retrieved from the Internet: <URL: https://mashable.com/article/vr-catching-re… [cited by applicant]
Taylor, et al., “Interacting with Real Objects in Virtual Worlds,” Real VR—Immersive Digital Reality: How to Import the Real World into Head-Mounted Immersive Displays [chapter], Mar. 3, 2020, pp. 337-353, Springer Inte… [cited by applicant]
Wikipedia, “Mirror world”, Wikipedia, the free encyclopedia, [accessed on Jul. 24, 2023], 4 Pages, Retrieved from the Internet: <URL: https://en.wikipedia.org/wiki/Mirror_world>. [cited by applicant]
Tolya Talamanov, “opencv,” (Version 438.0), Jul. 30, 2023, 2 Pages, URL: https://github.com/opencv/opencv. [cited by applicant]