Neural extension of 3D content in augmented reality environments
Generating augmented reality content includes inputting a first layout of a physical space and a first set of anchor content into a machine learning model; generating, via execution of the machine learning model, a first augmented reality view that includes (i) a first portion of the physical space and (ii) an extension of the first set of anchor content across a second portion of the physical space; and causing the first augmented reality view to be outputted in a computing device.
1. A computer-implemented method for generating augmented reality content, the method comprising:
inputting a first layout of a physical space and a first set of anchor content into a machine learning model, wherein the first set of anchor content is represented within the physical space;
generating, via execution of the machine learning model, a first three-dimensional (3D) volume that includes (i) a first subset of the physical space including the first set of anchor content and (ii) a placement of one or more 3D representations of the first set of anchor content in a second subset of the physical space, wherein:
the placement of the one or more 3D representations is located at a different position within the physical space relative to a position of the first set of anchor content within the first subset of physical space, and
generating the first 3D volume comprises:
applying a first set of neural network layers included in the machine learning model to the first set of anchor content to generate a semantic segmentation of the first set of anchor content, and
applying a second set of neural network layers included in the machine learning model to the first layout, the first set of anchor content, and the semantic segmentation to generate the first 3D volume; and
causing one or more views of the first 3D volume to be outputted within an augmented reality environment provided by in a computing device.
2. The computer-implemented method of claim 1 , wherein generating the first 3D volume comprises:
applying a first set of neural network layers included in the machine learning model to the first set of anchor content to generate the one or more 3D representations; and
applying a second set of neural network layers included in the machine learning model to the one or more 3D representations and the first layout to determine the placement of the one or more 3D representations in the second subset of the physical space.
3. The computer-implemented method of claim 1 , further comprising generating, via execution of the machine learning model, the first layout as a semantic segmentation of sensor data associated with the physical space.
4. The computer-implemented method of claim 3 , wherein the sensor data comprises at least one of an image of the physical space, a point cloud, a mesh, or a depth map.
5. The computer-implemented method of claim 1 , further comprising training the machine learning model based on a set of training layouts, a set of training anchor images, and one or more losses associated with the first 3D volume.
6. The computer-implemented method of claim 5 , wherein the one or more losses comprise a layout loss that is computed based on a representation of the first subset of the physical space in the first 3D volume and a corresponding subset of the physical space.
7. The computer-implemented method of claim 5 , wherein the one or more losses comprise a layout loss that is computed based on the first layout and the placement of the one or more 3D representations of the first set of anchor content within the first 3D volume.
8. The computer-implemented method of claim 1 , wherein the first 3D volume comprises a neural radiance field.
9. The computer-implemented method of claim 1 , wherein the first set of anchor content comprises at least one of an image, a video, or a 3D object.
10. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:
inputting a first layout of a physical space and a first set of anchor content into a machine learning model, wherein the first set of anchor content is represented within the physical space;
generating, via execution of the machine learning model, a first three-dimensional (3D) volume that includes (i) a first subset of the physical space including the first set of anchor content and (ii) a placement of one or more 3D representations of the first set of anchor content in a second subset of the physical space, wherein:
the placement of the one or more 3D representations is located at a different position within the physical space relative to a position of the first set of anchor content within the first subset of physical space; and
generating the first 3D volume comprises:
applying a first set of neural network layers included in the machine learning model to the first set of anchor content to generate a semantic segmentation of the first set of anchor content, and
applying a second set of neural network layers included in the machine learning model to the first layout, the first set of anchor content, and the semantic segmentation to generate the first 3D volume; and
causing one or more views of the first 3D volume to be outputted within an augmented reality environment provided by in a computing device.
11. The one or more non-transitory computer-readable media of claim 10 , wherein the instructions further cause the one or more processors to perform the step of applying a set of neural network layers included in the machine learning model to sensor data associated with the physical space to generate the first layout, wherein the first layout includes predictions of objects for regions of the sensor data.
12. The one or more non-transitory computer-readable media of claim 10 , wherein the instructions further cause the one or more processors to perform the steps of:
generating, via execution of the machine learning model, a second 3D volume that includes (i) a third subset of the physical space and (ii) a placement of one or more 3D representations of a second set of anchor content in a fourth subset of the physical space; and
causing one or more views of the second 3D volume to be outputted in the computing device.
13. The one or more non-transitory computer-readable media of claim 12 , wherein the first set of anchor content and the second set of anchor content comprise at least one of different video frames included in a video, depictions of two different scenes, or different sets of 3D objects.
14. The one or more non-transitory computer-readable media of claim 10 , wherein the instructions further cause the one or more processors to perform the step of training the first machine learning model based on one or more losses associated with the first 3D volume.
15. The one or more non-transitory computer-readable media of claim 14 , wherein the one or more losses comprise a similarity loss that is computed based on the first set of anchor content and a rendering of the second subset of the physical space within the 3D volume.
16. The one or more non-transitory computer-readable media of claim 14 , wherein the one or more losses comprise a reconstruction loss that is computed based on the one or more 3D representations of the first set of anchor content generated by the machine learning model and one or more 3D objects.
17. The one or more non-transitory computer-readable media of claim 14 , wherein the one or more losses comprise a segmentation loss that is computed based on a semantic segmentation of the first set of anchor content and a ground truth segmentation associated with the first set of anchor content.
18. The one or more non-transitory computer-readable media of claim 10 , wherein causing the one or more views of the first 3D volume to be outputted in the computing device comprises:
rendering the first 3D volume from the one or more views; and
outputting the one or more views within an augmented reality environment provided by the computing device.
19. A system, comprising:
one or more memories that store instructions, and
one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to perform the steps of:
inputting a first layout of a physical space and a first set of anchor content into a machine learning model, wherein the first set of anchor content is represented within the physical space;
generating, via execution of the machine learning model, a first three-dimensional (3D) volume that includes (i) a first subset of the physical space including the first set of anchor content and (ii) a placement of one or more 3D representations of the first set of anchor content in a second subset of the physical space, wherein:
the placement of the one or more 3D representations is located at a different position within the physical space relative to a position of the first set of anchor content within the first subset of physical space; and
generating the first 3D volume comprises:
applying a first set of neural network layers included in the machine learning model to the first set of anchor content to generate a semantic segmentation of the first set of anchor content, and
applying a second set of neural network layers included in the machine learning model to the first layout, the first set of anchor content, and the semantic segmentation to generate the first 3D volume; and
causing one or more views of the first 3D volume to be outputted within an augmented reality environment provided by in a computing device.