Applying pregenerated virtual experiences in new location
Aspects of the present disclosure involve a system for providing virtual experiences. The system performs operations including selecting, by a messaging application, a virtual experience that represents a previously captured real-world environment at a first location; accessing an image representing a new real-world environment at a second location, the image depicting a plurality of real-world objects; receiving input that selects a first real-world object from the plurality of real-world objects depicted in the image; and modifying the image, accessed at the second location, based on the virtual experience to depict the previously captured real-world environment with the first real-world object.
1 . A method comprising:
selecting, by an application, a virtual experience that represents a previously captured real-world environment at a first location;
accessing an image representing a new real-world environment at a second location, the image depicting a plurality of real-world objects;
identifying a subset of the plurality of real-world objects depicted in the image that excludes a selected first real-world object; and
overlaying the virtual experience on top of the subset of the plurality of real-world objects that excludes the selected first real-world object.
2 . The method of claim 1 , comprising:
receiving input that selects the first real-world object from the plurality of real-world objects depicted in the image; and
modifying the image, accessed at the second location, based on the virtual experience, to depict the previously captured real-world environment with the first real-world object.
3 . The method of claim 1 , further comprising:
removing the subset of the real-world objects from the image to keep only the first real-world object in the image.
4 . The method of claim 1 , further comprising:
capturing the image as part of a real-time feed received from a camera of a device;
detecting movement of the camera of the device; and
adjusting a relative position of the first real-world object within a view of the previously captured real-world environment as the movement of the camera is detected.
5 . The method of claim 4 , wherein adjusting the relative position comprises rotating the view of the previously captured real-world environment while the first real-world object remains stationary.
6 . The method of claim 1 , further comprising:
capturing one or more images at the first location using a camera of a device;
generating a three-dimensional (3D) model of the real-world environment at the first location using the one or more images; and
generating the virtual experience based on the 3D model of the real-world environment at the first location.
7 . The method of claim 6 , further comprising:
receiving data collected by at least one of a lidar sensor or Neural radiance fields (NeRF) to generate the 3D model.
8 . The method of claim 6 , further comprising applying a neural network to generate the 3D model.
9 . The method of claim 6 , wherein the virtual experience comprises a first virtual experience, further comprising:
capturing one or more images at a third location using the camera of the device;
generating a 3D model of a real-world environment at the third location using the one or more images captured at the third location; and
generating a second virtual experience based on the 3D model of the real-world environment at the third location.
10 . The method of claim 9 , further comprising:
storing a list of previously captured real-world environments comprising the first and second virtual experiences; and
displaying, by the application, the list of previously captured real-world environments comprising the first and second virtual experiences.
11 . The method of claim 10 , wherein selecting, by the application, the virtual experience comprises receiving input that selects the first virtual experience from the displayed list of previously captured real-world environments.
12 . The method of claim 1 , further comprising:
modifying one or more virtual items included as part of the virtual experience.
13 . The method of claim 12 , wherein modifying the one or more virtual items comprises at least one of moving, removing, or adding the virtual items in the virtual experience.
14 . The method of claim 1 , wherein the virtual experience comprises a virtual reality (VR) experience, further comprising:
generating a virtual representation of the first real-world object; and
adding the virtual representation of the first real-world object to the VR experience.
15 . The method of claim 1 , further comprising training a neural network to segment real-world items by performing operations comprising:
receiving training data comprising a plurality of training images and ground truth real-world item segmentations for each of the plurality of training images, each of the plurality of training images depicting a different type of real-world environment;
applying the neural network to a first training image of the plurality of training images to estimate segmentations of items depicted in the first training image;
computing a deviation between the estimated segmentations and the ground truth real-world item segmentations associated with the first training image; and
updating parameters of the neural network based on the computed deviation.
16 . The method of claim 1 , further comprising:
detecting a second user within a threshold proximity to a first user, the second user being a friend of the first user on the application; and
providing an option to a second device of the second user to join the first user in the virtual experience.
17 . The method of claim 16 , further comprising:
receiving input from the second device that modifies one or more virtual elements; and
applying one or more further modifications to the image based on the input received from the second device.
18 . The method of claim 17 , wherein the input comprises virtually drawing on walls depicted in the image.
19 . A system comprising:
at least one processor configured to perform operations comprising:
selecting, by an application, a virtual experience that represents a previously captured real-world environment at a first location;
accessing an image representing a new real-world environment at a second location, the image depicting a plurality of real-world objects;
identifying a subset of the plurality of real-world objects depicted in the image that excludes a selected first real-world object; and
overlaying the virtual experience on top of the subset of the plurality of real-world objects that excludes the selected first real-world object.
20 . A non-transitory machine-readable storage medium that includes instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:
selecting, by an application, a virtual experience that represents a previously captured real-world environment at a first location;
accessing an image representing a new real-world environment at a second location, the image depicting a plurality of real-world objects;
identifying a subset of the plurality of real-world objects depicted in the image that excludes a selected first real-world object; and
overlaying the virtual experience on top of the subset of the plurality of real-world objects that excludes the selected first real-world object.