IP Library Granted Patent US 11,755,275
Granted Patent B2
US 11,755,275 · App. 16/915,684 · Granted Sep 12, 2023

Generating augmented reality experiences utilizing physical objects to represent analogous virtual objects

Inventor: Andrew Lovitt (Redmond, WA)
Assignee: Meta Platforms Technologies, LLC
G06F3/165G06T11/00
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,755,275
App. No.
16/915,684
Granted
Sep 12, 2023
Kind
B2
Abstract

This disclosure describes embodiments of methods, non-transitory computer-readable media, and systems for detecting that a physical space includes a physical object corresponding to an analogous virtual object from an augmented reality experience and rendering (or otherwise modifying) the augment reality experience to integrate the physical object as part of the experience. In particular, the disclosed systems can determine that a physical object within a physical environment corresponds to an analogous virtual object of an augmented reality experience. Based on this correspondence, the disclosed systems can modify one or more of the virtual graphics, sound, or other features corresponding to the augmented reality experience to represent the virtual object using the physical object. For example, the disclosed systems can modify acoustic features of a sound for the augmented reality experience to simulate the sound originating from (or being affected by) the physical object.

Claims (77)

1. A computer-implemented method comprising:

capturing a data stream corresponding to a physical environment utilizing an augmented-reality-computing device;

identifying a physical object within the physical environment that corresponds to an analogous virtual object of an augmented reality experience;

analyzing the analogous virtual object within the augmented reality experience to determine a function of one or more acoustic features of the analogous virtual object;

modifying, by the augmented-reality-computing device, one or more acoustic features of a sound for the augmented reality experience based on the function of one or more acoustic features of the analogous virtual object to simulate that the sound for the augmented reality experience originates from a location of the physical object;

modifying, by the augmented-reality-computing device, one or more visual features of the augmented reality experience by applying a visual overlay to the location of the physical object; and

presenting, by the augmented-reality-computing device, the augmented reality experience with the modified one or more acoustic features and the analogous visual object.

2. The computer-implemented method as recited in claim 1 , further comprising:

mapping the physical environment to determine a location of the physical object relative to the augmented-reality-computing device, wherein the augmented-reality-computing device comprises a mixed reality headset; and

wherein modifying the one or more acoustic features of the sound for the augmented reality experience comprises modifying the sound for the augmented reality experience to simulate the sound originating from the location of the physical object relative to the augmented-reality-computing device.

3. The computer-implemented method as recited in claim 1 , further comprising modifying the one or more acoustic features of the sound for the augmented reality experience by performing one or more of: modifying an acoustic feature of the sound for the augmented reality experience based on a distance between the location of the physical object and the augmented-reality-computing device; modifying the acoustic feature of the sound for the augmented reality experience based on spectral localization cues from the location of the physical object relative to the augmented-reality-computing device; or modifying the acoustic feature of the sound for the augmented reality experience based on a visual characteristic of the physical object.

4. The computer-implemented method as recited in claim 1 , further comprising modifying the one or more acoustic features of the sound for the augmented reality experience by performing one or more of:

modifying one or more audio streams corresponding to the sound for the augmented reality experience; or

consolidating two or more audio streams corresponding to the sound for the augmented reality experience.

5. The computer-implemented method as recited in claim 4 , further comprising modifying the one or more acoustic features of the sound for the augmented reality experience by:

identifying a sound profile associated with the analogous virtual object; and

modifying an acoustic feature of the sound for the augmented reality experience based on the sound profile associated with the analogous virtual object.

6. The computer-implemented method as recited in claim 1 , further comprising:

identifying a visual characteristic of the analogous virtual object;

generating a virtual graphic overlay based on the visual characteristic; and

presenting the augmented reality experience by superimposing the virtual graphic overlay over a portion of the physical object or over an entirety of the physical object.

7. The computer-implemented method as recited in claim 6 , further comprising:

detecting a user interaction with the location of the physical object on which the virtual graphic overlay is superimposed;

generating a new virtual graphic overlay based on the user interaction with the location of the physical object; and

rendering the new virtual graphic overlay superimposed over the portion of the physical object or over the entirety of the physical object.

8. The computer-implemented method as recited in claim 1 , further comprising:

identifying that the sound for the augmented reality experience corresponds to an additional virtual object from the augmented reality experience;

identifying a sound effect for the sound based on the analogous virtual object;

determining a physical characteristic of the physical object; and

modifying the one or more acoustic features of the sound to simulate the sound effect based on the physical characteristic of the physical object.

9. The computer-implemented method as recited in claim 8 , wherein determining the physical characteristic of the physical object comprises determining one or more of: a thickness of the physical object, a mass of the physical object, a size of the physical object, a shape of the physical object, or a density of the physical object.

10. The computer-implemented method as recited in claim 1 , wherein determining that the physical object within the physical environment corresponds to the analogous virtual object of the augmented reality experience comprises:

generating an object-matching score indicating a degree to which one or more characteristics of the physical object match one or more characteristics of the analogous virtual object; and

determining the object-matching score satisfies an object-matching threshold.

11. A system comprising:

at least one processor; and

at least one non-transitory computer-readable storage medium storing instructions thereon that, when executed by the at least one processor, cause the system to:

capture a data stream corresponding to a physical environment utilizing an augmented-reality-computing device;

identify that a physical object within the physical environment that corresponds to an analogous virtual object of an augmented reality experience;

analyze the analogous virtual object within the augmented reality experience to determine a function of one or more acoustic features of the analogous virtual object;

modifying, by the augmented-reality-computing device, one or more acoustic features of a sound for the augmented reality experience based on the function of one or more acoustic features of the analogous virtual object to simulate that the sound for the augmented reality experience originates from a location of the physical object;

modifying, by the augmented-reality-computing device, one or more visual features of the augmented reality experience by applying a visual overlay to the location of the physical object; and

presenting, by the augmented-reality-computing device, the augmented reality experience with the modified one or more acoustic features and the analogous visual object.

12. The system as recited in claim 11 , further storing instructions thereon that, when executed by the at least one processor, cause the system to:

map the physical environment to determine a location of the physical object relative to the augmented-reality-computing device wherein the augmented-reality-computing device comprises a mixed reality headset; and

modify the one or more acoustic features of the sound for the augmented reality experience by modifying the sound to simulate the sound originating from the location of the physical object relative to the augmented-reality-computing device.

13. The system as recited in claim 11 , further storing instructions thereon that, when executed by the at least one processor, cause the system to modify the one or more acoustic features of the sound for the augmented reality experience by performing one or more of:

modifying an acoustic feature of the sound for the augmented reality experience based on a distance between the location of the physical object and the augmented-reality-computing device;

modifying the acoustic feature of the sound for the augmented reality experience based on spectral localization cues from the location of the physical object relative to the augmented-reality-computing device; or

modifying the acoustic feature of the sound for the augmented reality experience based on a visual characteristic of the physical object.

14. The system as recited in claim 11 , further storing instructions thereon that, when executed by the at least one processor, cause the system to modify the one or more acoustic features of the sound for the augmented reality experience by performing one or more of:

modifying one or more audio streams corresponding to the sound for the augmented reality experience; or

consolidating two or more audio streams corresponding to the sound for the augmented reality experience.

15. The system as recited in claim 11 , further storing instructions thereon that, when executed by the at least one processor, cause the system to modify the one or more acoustic features of the sound for the augmented reality experience by:

identifying a sound profile associated with the analogous virtual object; and

modifying an acoustic feature of the sound based on the sound profile associated with the analogous virtual object.

16. A non-transitory computer-readable medium storing instructions thereon that, when executed by at least one processor, cause the at least one processor to:

capture a data stream corresponding to a physical environment utilizing an augmented-reality-computing device;

identify that a physical object within the physical environment that corresponds to an analogous virtual object of an augmented reality experience;

analyze the analogous virtual object within the augmented reality experience to determine a function of one or more acoustic features of the analogous virtual object;

modifying, by the augmented-reality-computing device, one or more acoustic features of a sound for the augmented reality experience based on the function of one or more acoustic features of the analogous virtual object to simulate that the sound for the augmented reality experience originates from a location of the physical object;

modifying, by the augmented-reality-computing device, one or more visual features of the augmented reality experience by applying a visual overlay to the location of the physical object; and

presenting, by the augmented-reality-computing device, the augmented reality experience with the modified one or more acoustic features and the analogous visual object.

17. The non-transitory computer-readable medium as recited in claim 16 , further storing instructions thereon that, when executed by the at least one processor, cause the at least one processor to:

identify a visual characteristic of the analogous virtual object;

generate a virtual graphic overlay based on the visual characteristic; and

present the augmented reality experience by superimposing the virtual graphic overlay over a portion of the physical object or over an entirety of the physical object.

18. The non-transitory computer-readable medium as recited in claim 17 , further storing instructions thereon that, when executed by the at least one processor, cause the at least one processor to:

detect a user interaction with an area of the physical object on which the virtual graphic overlay is superimposed;

generate a new virtual graphic overlay based on the user interaction; and

render the new virtual graphic overlay superimposed over the portion of the physical object or over the entirety of the physical object.

19. The non-transitory computer-readable medium as recited in claim 16 , further storing instructions thereon that, when executed by the at least one processor, cause the at least one processor to:

identify that the sound for the augmented reality experience corresponds to an additional virtual object from the augmented reality experience, wherein the augmented-reality-computing device comprises a mixed reality headset;

identify a sound effect for the sound based on the analogous virtual object;

determine a physical characteristic of the physical object; and

modify the one or more acoustic features of the sound for the augmented reality experience to simulate the sound effect based on the physical characteristic of the physical object.

20. The non-transitory computer-readable medium as recited in claim 19 , further storing instructions thereon that, when executed by the at least one processor, cause the at least one processor to further determine the physical characteristic of the physical object by determining that the physical object displays one or more images or produces audio.

Assignments (2)
CHANGE OF NAME Recorded May 19, 2022
From: FACEBOOK TECHNOLOGIES, LLC
To: META PLATFORMS TECHNOLOGIES, LLC
Reel/Frame 060121/0676 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 2, 2020
From: LOVITT, ANDREW
To: FACEBOOK TECHNOLOGIES, LLC
Reel/Frame 053110/0291 →
Cited By (1)
US 12,572,326