Automatic measurements based on object classification
Various implementations disclosed herein include devices, systems, and methods that obtain a three-dimensional (3D) representation of a physical environment that was generated based on depth data and light intensity image data, generate a 3D bounding box corresponding to an object in the physical environment based on the 3D representation, classify the object based on the 3D bounding box and the 3D semantic data, and display a measurement of the object, where the measurement of the object is determined using one of a plurality of class-specific neural networks selected based on the classifying of the object.
1. A method comprising:
at an electronic device having a processor:
obtaining a three-dimensional (3D) representation of a physical environment that was generated based on depth data and light intensity image data, wherein the 3D representation is associated with 3D semantic data;
generating a 3D bounding box corresponding to an object in the physical environment based on the 3D representation, wherein the 3D bounding box is a refined 3D bounding box, wherein generating a refined 3D bounding box comprises:
generating a proposed 3D bounding box for an object using a first neural network; and
generating the refined 3D bounding box by:
inflating the proposed 3D bounding box based on a bounding box inflation scale,
identifying features of the object of the inflated proposed 3D bounding box using a second neural network, and
refining the proposed 3D bounding box based on the identified features;
classifying the object based on the 3D bounding box and the 3D semantic data; and
displaying a measurement of the object, the measurement of the object determined using one of a plurality of class-specific neural networks selected based on the classifying of the object.
2. The method of claim 1 , wherein classifying the object based on the 3D bounding box and the 3D semantic data comprises:
determining, based on the 3D semantic data, a class of the 3D bounding box using an object classification neural network; and
classifying the object corresponding to the 3D bounding box based on the classification of the 3D bounding box.
3. The method of claim 1 , wherein a first class-specific neural network is trained to determine specific points on a first classification of objects that are used for measurements of objects in the first classification.
4. The method of claim 3 , wherein a second class-specific neural network is trained to determine specific points on a second classification of objects that are used for measurements of objects in the second classification, wherein the second classification of objects is different than the first classification of objects.
5. The method of claim 4 , wherein the measurements of objects in the second classification of objects are different than the measurements of objects in the first classification of objects.
6. The method of claim 1 , wherein the first neural network generates the proposed 3D bounding box based on the 3D semantic data associated with the object.
7. The method of claim 1 , wherein the second neural network identifies the features of the object based on the 3D semantic data associated with the object.
8. The method of claim 1 , wherein a third neural network is trained to:
refine the accuracy of the identified features from the second neural network based on the 3D semantic data associated with the object and the light intensity image data; and
output a further refined 3D bounding box based on the refined accuracy of the identified features from the second neural network.
9. The method of claim 1 , wherein the 3D bounding box provides a location, an orientation, and a shape of an identified object.
10. The method of claim 1 , wherein the 3D representation comprises a 3D point cloud and the associated 3D semantic data includes semantic labels associated with at least a portion of 3D points within the 3D point cloud.
11. The method of claim 10 , wherein the semantic labels identify walls, wall attributes, objects, and classifications of the objects of the physical environment.
12. A device comprising:
a non-transitory computer-readable storage medium; and
one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the device to perform operations comprising:
obtaining a three-dimensional (3D) representation of a physical environment that was generated based on depth data and light intensity image data, wherein the 3D representation is associated with 3D semantic data;
generating a 3D bounding box corresponding to an object in the physical environment based on the 3D representation, wherein the 3D bounding box is a refined 3D bounding box, wherein generating a refined 3D bounding box comprises:
generating a proposed 3D bounding box for an object using a first neural network; and
generating the refined 3D bounding box by:
inflating the proposed 3D bounding box based on a bounding box inflation scale,
identifying features of the object of the inflated proposed 3D bounding box using a second neural network, and
refining the proposed 3D bounding box based on the identified features;
classifying the object based on the 3D bounding box and on the 3D semantic data; and
displaying a measurement of the object, the measurement of the object determined using one of a plurality of class-specific neural networks selected based on the classifying of the object.
13. The device of claim 12 , wherein classifying the object based on the 3D bounding box and the 3D semantic data comprises:
determining, based on the 3D semantic data, a class of the 3D bounding box using an object classification neural network; and
classifying the object corresponding to the 3D bounding box based on the classification of the 3D bounding box.
14. The device of claim 12 , wherein a first class-specific neural network is trained to determine specific points on a first classification of objects that are used for measurements of objects in the first classification.
15. The device of claim 14 , wherein a second class-specific neural network is trained to determine specific points on a second classification of objects that are used for measurements of objects in the second classification, wherein the second classification of objects is different than the first classification of objects.
16. The device of claim 15 , wherein the measurements of objects in the second classification of objects are different than the measurements of objects in the first classification of objects.
17. The device of claim 12 , wherein the first neural network generates the proposed 3D bounding box based on the 3D semantic data associated with the object.
18. A non-transitory computer-readable storage medium, storing program instructions computer-executable on a computer to perform operations comprising:
obtaining a three-dimensional (3D) representation of a physical environment that was generated based on depth data and light intensity image data, wherein the 3D representation is associated with 3D semantic data;
generating a 3D bounding box corresponding to an object in the physical environment based on the 3D representation, wherein the 3D bounding box is a refined 3D bounding box, wherein generating a refined 3D bounding box comprises:
generating a proposed 3D bounding box for an object using a first neural network; and
generating the refined 3D bounding box by:
inflating the proposed 3D bounding box based on a bounding box inflation scale,
identifying features of the object of the inflated proposed 3D bounding box using a second neural network, and
refining the proposed 3D bounding box based on the identified features;
classifying the object based on the 3D bounding box and on the 3D semantic data; and
displaying a measurement of the object, the measurement of the object determined using one of a plurality of class-specific neural networks selected based on the classifying of the object.