Semantic fusion
In one embodiment, a computing system accesses a plurality of images captured by one or more cameras from a plurality of camera poses. The computing system generates, using the plurality of images, a plurality of semantic segmentations comprising semantic information of one or more objects captured in the plurality of images. The computing system accesses a three-dimensional (3D) model of the one or more objects. The computing system determines, using the plurality of camera poses, a corresponding plurality of virtual camera poses relative to the 3D model of the one or more objects. The computing system generates a semantic 3D model by projecting the semantic information of the plurality of semantic segmentations towards the 3D model using the plurality of virtual camera poses.
1. A method comprising, by a computing system:
accessing a plurality of images captured by one or more cameras from a plurality of camera poses;
generating, using the plurality of images, a plurality of semantic segmentations comprising semantic information of one or more objects captured in the plurality of images;
accessing a three-dimensional (3D) model of the one or more objects;
determining, using the plurality of camera poses, a corresponding plurality of virtual camera poses relative to the 3D model of the one or more objects;
generating a semantic 3D model by projecting the semantic information of the plurality of semantic segmentations towards the 3D model using the plurality of virtual camera poses, wherein the semantic information from two or more of the plurality of semantic segmentations are combined to apply to a first object of the one or more objects, and wherein each of the plurality of semantic segmentations corresponds to each of the plurality of virtual camera poses.
2. The method of claim 1 , further comprising generating, using the plurality of images, a plurality of geometry-based segmentations comprising geometric information of the one or more objects captured in the plurality of images.
3. The method of claim 2 , wherein generating the semantic 3D model by projecting the semantic information of the plurality of semantic segmentations further comprises using the geometric information of the one or more objects to project the semantic information corresponding to the one or more objects captured in the plurality of images.
4. The method of claim 1 , further comprising generating, using the plurality of images, a plurality of instance segmentations comprising object identification of the one or more objects captured in the plurality of images.
5. The method of claim 4 , further comprising generating an instance 3D model by projecting the object identification of the plurality of instance segmentations towards the 3D model using the plurality of virtual camera poses.
6. The method of claim 5 , wherein generating the instance 3D model comprises combining object identification from each of the plurality of instance segmentations to apply to one of the one or more objects.
7. The method of claim 1 , further comprising generating a semantic tree based on the semantic information of the one or more objects captured in the plurality of images, wherein the semantic tree comprises a plurality of semantic levels, and wherein each semantic level corresponds to a threshold amount of semantic information.
8. The method of claim 7 , wherein the plurality of semantic levels comprises one or more of:
a first semantic level that comprises information corresponding to individual objects of the one or more objects,
a second semantic level that comprises information corresponding to object classes of the one or more objects, and
a third semantic level that comprises information corresponding to a semantic set of the one or more objects.
9. The method of claim 7 , wherein projecting the semantic information further comprises projecting the semantic information of a particular semantic level towards the 3D model.
10. The method of claim 1 , wherein the 3D model of the one or more objects is generated based on the plurality of images.
11. The method of claim 1 , further comprising calculating one or more confidence values that each semantic information of the one or more objects from the plurality of semantic segmentations corresponds to the respective object.
12. The method of claim 11 , wherein objects are associated with a plurality of environments, and wherein the one or more confidence values is based on whether the semantic information of the one or more objects corresponds to objects within a particular environment.
13. The method of claim 11 , wherein projecting the semantic information of the plurality of semantic segmentations towards the 3D model further comprises:
determining the confidence value of semantic information for one object exceeds a threshold confidence value; and
projecting the semantic information towards the 3D model of the respective object in response to the confidence value exceeding the threshold confidence value.
14. The method of claim 11 , wherein the semantic information associated with a confidence value less than a threshold confidence value is discarded.
15. The method of claim 1 , wherein generating the semantic 3D model comprises combining semantic information from each of the plurality of semantic segmentations to apply to one of the one or more objects.
16. The method of claim 1 , wherein projecting the semantic information towards the 3D model comprises adding a label that corresponds to the respective object in the 3D model.
17. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
access a plurality of images captured by one or more cameras from a plurality of camera poses;
generate, using the plurality of images, a plurality of semantic segmentations comprising semantic information of one or more objects captured in the plurality of images;
access a three-dimensional (3D) model of the one or more objects;
determine, using the plurality of camera poses, a corresponding plurality of virtual camera poses relative to the 3D model of the one or more objects;
generate a semantic 3D model by projecting the semantic information of the plurality of semantic segmentations towards the 3D model using the plurality of virtual camera poses, wherein the semantic information from two or more of the plurality of semantic segmentations are combined to apply to a first object of the one or more objects, and wherein each of the plurality of semantic segmentations corresponds to each of the plurality of virtual camera poses.
18. A system comprising: one or more processors; and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:
access a plurality of images captured by one or more cameras from a plurality of camera poses;
generate, using the plurality of images, a plurality of semantic segmentations comprising semantic information of one or more objects captured in the plurality of images;
access a three-dimensional (3D) model of the one or more objects;
determine, using the plurality of camera poses, a corresponding plurality of virtual camera poses relative to the 3D model of the one or more objects;
generate a semantic 3D model by projecting the semantic information of the plurality of semantic segmentations towards the 3D model using the plurality of virtual camera poses, wherein the semantic information from two or more of the plurality of semantic segmentations are combined to apply to a first object of the one or more objects, and wherein each of the plurality of semantic segmentations corresponds to each of the plurality of virtual camera poses.