Automated classification based on photo-realistic image/model mappings
Techniques are provided for increasing the accuracy of automated classifications produced by a machine learning engine. Specifically, the classification produced by a machine learning engine for one photo-realistic image is adjusted based on the classifications produced by the machine learning engine for other photo-realistic images that correspond to the same portion of a 3D model that has been generated based on the photo-realistic images. Techniques are also provided for using the classifications of the photo-realistic images that were used to create a 3D model to automatically classify portions of the 3D model. The classifications assigned to the various portions of the 3D model in this manner may also be used as a factor for automatically segmenting the 3D model.
1. A method comprising:
generating a 3D-model of a real-world space based on a collection of photo-realistic images of the real-world space that were captured in the real world, each image of the collection of photo-realistic images including spatial metadata;
comparing the spatial metadata associated with each of the collection of photo-realistic images with spatial metadata associated with the 3D-model;
determining a source-set for a target portion of the 3D-model based on the comparison, the source-set including a subset of the collection of photo-realistic images, the subset of the collection of photo-realistic images including a target region to which the target portion of the 3D-model maps;
for each of the photo-realistic images of the source-set:
identifying a source-region within a particular photo-realistic image, the source-region being a region within the particular photo-realistic image corresponding to the target portion of the 3D-model; and
assigning a source-region classification to the particular photo-realistic image;
determining an aggregate classification of the target region based on a plurality of source-region classifications; and
assigning the aggregate classification to a target region classification of the target region.
2. The method of claim 1 , wherein the spatial metadata includes capture location orientation data and depth data of the photo-realistic images.
3. The method of claim 1 , further comprising providing to one or more computing devices, a confidence score associated with the target region classification.
4. The method of claim 3 , wherein the confidence score is calculated based on an aggregate of confidence scores of the plurality of source-regions within the particular source-set.
5. The method of claim 4 , wherein the aggregate of confidence scores is a weighted average of the plurality of source-regions within the particular source-set.
6. The method of claim 1 , wherein at least one source-region of the plurality of source-regions is a set of pixels within a photo-realistic image from the collection of photo-realistic images.
7. The method of claim 1 , wherein the real-world space is a building, and the target portion of the 3D-model is the portion of the 3D-model that represents a particular room of a house.
8. The method of claim 7 , wherein:
the source-region classifications assigned to the plurality of source-regions are room-type classifications; and
the aggregate classification assigned to the target region is a room-type classification.
9. The method of claim 1 , further comprising determining the one of the subset of the collection of photo-realistic images based on a view volume of each of the subset of the collection of photo-realistic images and calculating a variance of colors of neighboring pixels of each of the subset of the collection of photo-realistic images.
10. One or more non-transitory computer-readable media storing instructions which, when executed by one or more computing devices, cause:
generating a 3D-model of a real-world space based on a collection of photo-realistic images of the real-world space, that were captured in the real world, each image of the collection of photo-realistic images including spatial metadata;
selecting a target region of at least a subset of the collection of photo-realistic images;
comparing the spatial metadata associated with each of the collection of photo-realistic images with spatial metadata associated with the 3D-model;
determining a source-set for a target portion of the 3D-model based on the comparison, the source-set including a subset of the collection of photo-realistic images, the subset of the collection of photo-realistic images including a target region to which the target portion of the 3D-model maps;
for each of the photo-realistic images of the source-set:
identifying a source-region within a particular photo-realistic image, the source-region being a region within the particular photo-realistic image corresponding to the target portion of the 3D-model; and
assigning a source-region classification to the particular photo-realistic image;
determining an aggregate classification of the target region based on a plurality of source-region classifications; and
assigning the aggregate classification to a target region classification of the target region.
11. The one or more non-transitory computer-readable media of claim 10 , wherein the spatial metadata includes capture location orientation data and depth data of the photo-realistic images.
12. The one or more non-transitory computer-readable media of claim 10 , wherein the instructions further cause providing to one or more computing devices, a confidence score associated with the target region classification.
13. The one or more non-transitory computer-readable media of claim 12 , wherein the confidence score is calculated based on an aggregate of confidence scores of the plurality of source-regions within the particular source-set.
14. The one or more non-transitory computer-readable media of claim 13 , wherein the aggregate of confidence scores is a weighted average of the plurality of source-regions within the particular source-set.
15. The one or more non-transitory computer-readable media of claim 10 , wherein the real-world space is a building, and the target portion of the 3D-model is the portion of the 3D-model that represents a particular room of a house.
16. The one or more non-transitory computer-readable media of claim 15 , wherein:
the source-region classifications assigned to the plurality of source-regions are room-type classifications; and
the aggregate classification assigned to the target region is a room-type classification.
17. A hardware server, comprising:
a memory storing computer-executable components; and
a processor configured to execute the computer-executable components to perform:
generating a 3D-model of a real-world space based on a collection of photo-realistic images of the real-world space, that were captured in the real world, each image of the collection of photo-realistic images including spatial metadata;
selecting a target region of at least a subset of the collection of photo-realistic images;
comparing the spatial metadata associated with each of the collection of photo-realistic images with spatial metadata associated with the 3D-model;
determining a source-set for a target portion of the 3D-model based on the comparison, the source-set including a subset of the collection of photo-realistic images, the subset of the collection of photo-realistic images including a target region to which the target portion of the 3D-model maps;
for each of the photo-realistic images of the source-set:
identifying a source-region within a particular photo-realistic image, the source-region being a region within the particular photo-realistic image corresponding to the target portion of the 3D-model; and
assigning a source-region classification to the particular photo-realistic image;
determining an aggregate classification of the target region based on a plurality of source-region classifications; and
assigning the aggregate classification to a target region classification of the target region.
18. The hardware server of claim 17 , wherein the spatial metadata includes capture location orientation data and depth data of the photo-realistic images.
19. The hardware server of claim 17 , wherein the processor is further configured to perform providing to one or more computing devices, a confidence score associated with the target region classification.
20. The hardware server of claim 19 , wherein the confidence score is calculated based on an aggregate of confidence scores of the plurality of source-regions within the particular source-set.