Semi-supervised 3D indoor layout estimation from a single 360 degree panorama
Introduced here computer programs and associated computer-implemented techniques for modeling interior spaces based on an analysis of digital images of those interior spaces. These computer programs can be trained to accomplish this without using extensive sets of labeled data. Instead, these computer programs are able to effectively supplement labeled data with unlabeled data to accurately model interior spaces in settings where limited labeled data is available. Such an approach allows these computer programs to establish the layouts of interior spaces without extensive knowledge of those interior spaces and without users, who are responsible for generating the digital images, interacting with the surroundings in a meaningful way.
1. A method comprising:
obtaining a set of labeled inputs, wherein each labeled input is representative of an annotated panorama image that includes at least one label indicating where a juncture at which a pair of surfaces are joined is located in the annotated panorama image;
obtaining a set of unlabeled inputs, wherein each unlabeled input is representative of a non-annotated panorama image that does not include any labels;
providing (i) the set of labeled inputs and (ii) the set of unlabeled inputs to a model as training data, so as to produce a trained model that is able to identify junctures of an interior space that is internal to a building upon being applied to a panorama image of the interior space,
wherein the trained model is representative of a stochastic neural network that is characterized by (i) supervised loss and (ii) an unsupervised loss that are combined as a weighted sum,
wherein the supervised loss is determined based on the set of labeled inputs, and
wherein the unsupervised loss is determined based on the set of labeled inputs and the set of unlabeled inputs; and
storing the trained model in a data structure.
2. The method of claim 1 , wherein the trained model is one of multiple trained models stored in the data structure.
3. The method of claim 2 , wherein each trained model of the multiple trained models is trained using different sets of labeled and unlabeled inputs.
4. The method of claim 1 , wherein the set of labeled inputs is associated with a first set of interior spaces, and wherein the set of unlabeled inputs is associated with a second set of interior spaces.
5. The method of claim 1 , wherein the set of labeled inputs is obtained from a first source, and wherein the set of unlabeled inputs is obtained from a second source.
6. The method of claim 1 , wherein the set of labeled inputs and the set of unlabeled inputs are sampled from a data distribution, and wherein the set of labeled inputs is representative of a labeled subset of the set of unlabeled inputs.
7. The method of claim 1 , wherein the set of labeled inputs and the set of unlabeled inputs are sampled from different data distributions.
8. The method of claim 1 , wherein the trained model is configured to encode, at each column of the panorama image, positions of the junctures of the interior space.
9. A non-transitory medium with instructions stored thereon that, when executed by a processor of a computing device, cause the computing device to perform operations comprising:
receiving first input that specifies a selection of a set of non-annotated panorama images,
wherein the set of non-annotated panorama images is representative of a set of unlabeled inputs;
receiving second input that specifies labels for a subset of the set of non- annotated panorama images, so as to produce a set of annotated panorama images,
wherein each of the labels indicates a location of a juncture at which a pair of surfaces are joined, and
wherein the set of annotated panorama images is representative of a set of labeled inputs;
providing (i) the set of unlabeled inputs and (ii) the set of labeled inputs to a model as training data, so as to produce a trained model that is able to identify junctures of an interior space that is internal to a building upon being applied to a panorama image of the interior space; and
storing the trained model in a data structure.
10. The non-transitory medium of claim 9 , wherein the operations further comprise:
indicating, in the data structure, a characteristic of the interior space for which the trained model is trained to identify junctures.
11. The non-transitory medium of claim 10 , wherein the characteristic is a type of room.
12. The non-transitory medium of claim 10 , wherein the characteristic is a number of corners.
13. The non-transitory medium of claim 10 , wherein said indicating includes programmatically associating a term with the trained model that serves as a title.
14. The non-transitory medium of claim 10 , wherein said indicating includes programmatically associating an alphanumeric identifier with the trained model.
15. The non-transitory medium of claim 10 , wherein the trained model is configured to encode, at each column of the panorama image, positions of the junctures of the interior space.
16. The non-transitory medium of claim 9 , wherein the trained model is a stochastic neural network that is trained by introducing a stochastic weight to each node during training.
17. A computing device comprising:
a processor; and
a memory with instructions stored therein that, when executed by the processor, cause the computing device to:
obtain a set of labeled inputs, wherein each labeled input is representative of an annotated panorama image that includes at least one label indicating where a juncture at which a pair of surfaces are joined is located in the annotated panorama image,
obtain a set of unlabeled inputs, wherein each unlabeled input is representative of a non-annotated panorama image that does not include any labels,
provide (i) the set of labeled inputs and (ii) the set of unlabeled inputs to a model as training data, so as to produce a trained model that is able to identify junctures of an interior space that is internal to a building upon being applied to a panorama image of the interior space, and
store the trained model in a data structure.
18. The computing device of claim 17 , wherein the set of unlabeled inputs is obtained responsive to receiving input indicative of a selection of a source.
19. The computing device of claim 17 , wherein the instructions further cause the computing device to:
select the model from amongst a series of untrained models based on an environment in which the model is expected to be deployed after training.