Masking method for augmented reality effects
To provide masks for augmented reality (AR) effects in real-time, the embodiments described herein capture a plurality of images of an environment that includes one or more real-world objects after initiating an AR session with an AR device located in the environment. Upon detecting a trigger to generate an AR effect targeting a first real-world object of the one or more real-world objects, a first virtual representation of the first real-world object is generated, based on the plurality of images of the environment, and a second virtual representation of a second real-world object of the one or more real-world objects is generated based on the plurality of images of the environment, wherein the second real-world object is at least partially occluded by the first real-world object.
1. A computer-implemented method, comprising:
initiating an augmented reality (AR) session within an environment comprising one or more real-world objects, wherein an environmental map of the environment is unavailable for generating AR effects for the AR session at a time of the initiation of the AR session;
after initiating the AR session, capturing a plurality of images of the environment upon detecting at least a first trigger in the environment; and
upon detecting a second trigger to generate an AR effect targeting a first real-world object of the one or more real-world objects:
generating a first virtual representation of the first real-world object, based on the plurality of images;
upon determining that image data of a second real-world object of the one or more real-world objects partially occluded by the first real-world object is unavailable within the plurality of images:
identifying at least one third real-world object of the one or more real-world objects behind the first real-world object, based on the plurality of images;
determining one or more first attributes of the at least one third real-world object; and
predicting one or more second attributes of the second real-world object based on the one or more first attributes; and
generating a second virtual representation of the second real-world object, based on the plurality of images and the one or more second attributes.
2. The computer-implemented method of claim 1 , further comprising:
manipulating the first virtual representation, based on input from a user; and
masking the first real-world object with the second virtual representation, while the first virtual representation is manipulated.
3. The computer-implemented method of claim 1 , wherein:
the AR session is initiated by an AR device; and
the first trigger comprises movement of a user of the AR device.
4. The computer-implemented method of claim 3 , wherein each image of the plurality of images is an image of the environment from a different viewpoint of the AR device.
5. The computer-implemented method of claim 3 , wherein the AR session is initiated upon powering on the AR device.
6. The computer-implemented method of claim 3 , wherein the plurality of images are captured using one or more visual sensors of the AR device.
7. The computer-implemented method of claim 3 , wherein the AR device comprises an all in one head mounted AR device.
8. A non-transitory computer-readable storage medium comprising computer-readable program code embodied therewith, the computer-readable program code is configured to perform, when executed by a processor, an operation, the operation comprising:
initiating an augmented reality (AR) session within an environment comprising one or more real-world objects, wherein an environmental map of the environment is unavailable for generating AR effects for the AR session at a time of the initiation of the AR session;
after initiating the AR session, capturing a plurality of images of the environment upon detecting at least a first trigger in the environment; and
upon detecting a second trigger to generate an AR effect targeting a first real-world object of the one or more real-world objects:
generating a first virtual representation of the first real-world object, based on the plurality of images;
upon determining that image data of a second real-world object of the one or more real-world objects partially occluded by the first real-world object is unavailable within the plurality of images:
identifying at least one third real-world object of the one or more real-world objects behind the first real-world object, based on the plurality of images;
determining one or more first attributes of the at least one third real-world object; and
predicting one or more second attributes of the second real-world object based on the one or more first attributes; and
generating a second virtual representation of the second real-world object, based on the plurality of images and the one or more second attributes.
9. The non-transitory computer-readable storage medium of claim 8 , the operation further comprising:
manipulating the first virtual representation, based on input from a user; and
masking the first real-world object with the second virtual representation, while the first virtual representation is manipulated.
10. The non-transitory computer-readable storage medium of claim 8 , wherein:
the AR session is initiated by an AR device; and
the first trigger comprises movement of a user of the AR device.
11. The non-transitory computer-readable storage medium of claim 10 , wherein each image of the plurality of images is an image of the environment from a different viewpoint of the AR device.
12. The non-transitory computer-readable storage medium of claim 10 , wherein the AR session is initiated upon powering on the AR device.
13. A system, comprising:
at least one visual sensor;
a processor; and
a memory comprising instructions which, when executed on the processor, performs an operation, the operation comprising:
initiating an augmented reality (AR) session within an environment comprising one or more real-world objects, wherein an environmental map of the environment is unavailable for generating AR effects for the AR session at a time of the initiation of the AR session;
after initiating the AR session, capturing, via the at least one visual sensor, a plurality of images of the environment upon detecting at least a first trigger in the environment; and
upon detecting a second trigger to generate an AR effect targeting a first real-world object of the one or more real-world objects:
generating a first virtual representation of the first real-world object, based on the plurality of images;
upon determining that image data of a second real-world object of the one or more real-world objects partially occluded by the first real-world object is unavailable within the plurality of images:
identifying at least one third real-world object of the one or more real-world objects behind the first real-world object, based on the plurality of images;
determining one or more first attributes of the at least one third real-world object; and
predicting one or more second attributes of the second real-world object based on the one or more first attributes; and
generating a second virtual representation of the second real-world object, based on the plurality of images and the one or more second attributes.
14. The system of claim 13 , the operation further comprising:
manipulating the first virtual representation, based on input from a user; and
masking the first real-world object with the second virtual representation, while the first virtual representation is manipulated.
15. The system of claim 13 , wherein the first trigger comprises movement of a user of the system.
16. The system of claim 15 , wherein each image of the plurality of images is an image of the environment from a different viewpoint of the system.
17. The system of claim 15 , wherein the AR session is initiated upon powering on the system.
18. The system of claim 15 , wherein the system comprises an all in one head mounted AR device.