Semi-supervised layout estimation of interior spaces from panorama images
Introduced here computer programs and associated computer-implemented techniques for modeling interior spaces based on an analysis of digital images of those interior spaces. These computer programs can be trained to accomplish this without using extensive sets of labeled data. Instead, these computer programs are able to effectively supplement labeled data with unlabeled data to accurately model interior spaces in settings where limited labeled data is available. Such an approach allows these computer programs to establish the layouts of interior spaces without extensive knowledge of those interior spaces and without users, who are responsible for generating the digital images, interacting with the surroundings in a meaningful way.
1. A system for generating a layout of an interior space, the system comprising:
a processor;
a memory with instructions stored therein that, when executed by the processor, cause a computing device to:
receive, from an image sensor of the computing device:
one or more labeled panorama images comprising layout annotations, wherein the layout annotations represent boundary junctures associated with the interior space; and
one or more unlabeled panorama images without layout annotations, wherein the one or more labeled panorama images and the one or more unlabeled panorama images are representative of different portions of the interior space;
input, to a trained neural network model, the one or more labeled panorama images and the one or more unlabeled panorama images, wherein the trained neural network model is trained in a semi-supervised manner using labeled and unlabeled training data;
output, by the trained neural network model:
a 2-dimensional (2D) or 3-dimensional (3D) reconstruction layout of a least a portion of the interior space; and
coordinates indicating locations where one or more junctures are located within the 2D or 3D reconstruction layout.
2. The system of claim 1 , wherein the instructions further cause the computing device to receive, from a sensor suite of the computing device, location data corresponding to one or more juncture positions within the one or more labeled panorama images relative to a coordinate system associated with the interior space.
3. The system of claim 1 , wherein the one or more labeled panorama images and the one or more unlabeled panorama images are representative of a same portion of the interior space.
4. The system of claim 1 , wherein the trained neural network model comprises a stochastic neural network that is trained by introducing a stochastic weight to each node of the stochastic neural network during training.
5. The system of claim 1 , wherein the trained neural network model is selected based on a complexity or a type of room associated with the one or more labeled or unlabeled panorama images.
6. The system of claim 1 , wherein the one or more junctures represent locations relative to a coordinate system associated with the interior space where a pair of surfaces join, intersect, merge, or converge with one another within the interior space.
7. The system of claim 1 , wherein the trained neural network model is configured to output pixel-level coordinates indicating locations where one or more junctures are located within the reconstruction layout.
8. A method comprising:
receiving, from an image sensor of a computing device:
one or more labeled panorama images comprising layout annotations, wherein the layout annotations represent boundary junctures associated with an interior space; and
one or more unlabeled panorama images without layout annotations, wherein the one or more labeled panorama images and the one or more unlabeled panorama images are representative of different portions of the interior space;
inputting, to a trained neural network model, the one or more labeled panorama images and the one or more unlabeled panorama images, wherein the trained neural network model is trained in a semi-supervised manner using labeled and unlabeled training data;
outputting, by the trained neural network model:
a 2-dimensional (2D) or 3-dimensional (3D) reconstruction layout of a least a portion of the interior space; and
coordinates indicating locations where one or more junctures are located within the 2D or 3D reconstruction layout.
9. The method of claim 8 , further comprising receiving, from a sensor suite of the computing device, location data corresponding to one or more juncture positions within the one or more labeled panorama images relative to a coordinate system associated with the interior space.
10. The method of claim 8 , wherein the one or more labeled panorama images and the one or more unlabeled panorama images are representative of a same portion of the interior space.
11. The method of claim 8 , wherein the trained neural network model comprises a stochastic neural network that is trained by introducing a stochastic weight to each node of the stochastic neural network during training.
12. The method of claim 8 , wherein the trained neural network model is selected based on a complexity of the one or more labeled or unlabeled panorama images.
13. The method of claim 8 , wherein the trained neural network model is selected based on a type of room associated with the one or more labeled or unlabeled panorama images.
14. The method of claim 8 , wherein the trained neural network model is configured to output pixel-level coordinates indicating locations where one or more junctures are located within the reconstruction layout.
15. The method of claim 8 , further comprising outputting dimensions of the interior space.
16. A non-transitory medium with instructions stored thereon that, when executed by a processor of a computing device, cause the computing device to perform operations comprising:
receiving, from an image sensor of the computing device:
one or more labeled panorama images comprising layout annotations, wherein the layout annotations represent boundary junctures associated with an interior space; and
one or more unlabeled panorama images without layout annotations, wherein the one or more labeled panorama images and the one or more unlabeled panorama images are representative of different portions of the interior space;
inputting, to a trained neural network model, the one or more labeled panorama images and the one or more unlabeled panorama images, wherein the trained neural network model is trained in a semi-supervised manner using labeled and unlabeled training data;
outputting, by the trained neural network model:
a 2-dimensional (2D) or 3-dimensional (3D) reconstruction layout of a least a portion of the interior space; and
coordinates indicating locations where one or more junctures are located within the 2D or 3D reconstruction layout.
17. The non-transitory medium of claim 16 , further comprising receiving, from a sensor suite of the computing device, location data corresponding to one or more juncture positions within the one or more labeled panorama images relative to a coordinate system associated with the interior space.
18. The non-transitory medium of claim 16 , wherein the one or more labeled panorama images and the one or more unlabeled panorama images are representative of a same portion of the interior space.
19. The non-transitory medium of claim 16 , wherein the trained neural network model comprises a stochastic neural network that is trained by introducing a stochastic weight to each node of the stochastic neural network during training.
20. The non-transitory medium of claim 16 , wherein the trained neural network model is selected based on a complexity of the one or more labeled or unlabeled panorama images.