Systems and methods for labeling 3D models using virtual reality and augmented reality
A virtual reality (VR) labeling computer system configured to receive a 3D model, process the 3D model using object recognition, identify at least one environmental feature within the 3D model, generate a processed 3D model including the at least one environmental feature, display a VR environment based upon the processed 3D model; receive user input including labeling data associated with the environmental feature; generate a labeled 3D model by embedding the labeling data into the processed 3D model; and generate training data based upon the labeled 3D model.
1. A computer-implemented method for labeling a three dimensional (3D) model using virtual reality (VR) techniques, the method implemented by a computer system including at least one processor, the method comprising:
receiving, by the processor, the 3D model;
processing, by the processor, the 3D model using object recognition;
identifying at least one environmental feature within the 3D model, wherein the at least one environmental feature is unidentified and unlabeled;
generating, by the processor, a processed 3D model including the at least one environmental feature;
displaying, through a VR device, a VR environment to a user based upon the processed 3D model, wherein the VR environment includes the at least one environmental feature;
prompting, through the VR environment, the user to input (i) labeling data for the environmental feature through user interaction with the VR device and (ii) boundary data by creating a bounding frame within the VR environment, wherein the labeling data identifies the environmental feature, wherein the bounding frame delineates a 3D region within the 3D model, and wherein the 3D region includes data representing the environmental feature;
processing, by the VR device, (i) the data within the 3D region using object recognition and (ii) the user input associated with the environmental feature;
identifying an environmental feature corresponding to the environmental feature represented by data within the 3D region;
generating a labeled 3D model by embedding the labeling data and the boundary data for the environmental feature into the processed 3D model;
extracting the environmental feature and associated labeling data and boundary data from the labeled 3D model; and
generating, by the processor, training data based upon the extracted environmental feature of the labeled 3D model.
2. The computer-implemented method of claim 1 , the method further comprising:
training a machine learning model using the training data.
3. The computer-implemented method of claim 1 , the method further comprising:
training a machine learning model using the labeled 3D model.
4. The computer-implemented method of claim 1 , wherein the 3D model is a point cloud.
5. The computer-implemented method of claim 4 , wherein the point cloud is generated based upon aerial photos of a real-world location.
6. The computer-implemented method of claim 1 , wherein processing the 3D model using object recognition includes processing the 3D model using a segmentation technique.
7. The computer-implemented method of claim 6 , wherein the segmentation technique is a semantic segmentation technique.
8. The computer-implemented method of claim 1 , wherein generating the processed 3D model further comprises updating meta-data of data points representing the environmental feature.
9. The computer-implemented method of claim 1 , wherein generating the processed 3D model further comprises:
generating data points representing the environmental feature; and
embedding the data points representing the environmental feature into the 3D model.
10. The computer-implemented method of claim 9 , wherein the data points representing the environmental feature are a surface mesh.
11. The computer-implemented method of claim 10 , wherein displaying the VR environment further comprises altering the appearance of the surface mesh.
12. The computer-implemented method of claim 11 , wherein altering the appearance of the environmental feature includes shading the environmental feature.
13. The computer-implemented method of claim 11 , wherein altering the appearance of the environmental feature includes outlining the environmental feature.
14. The computer-implemented method of claim 10 , wherein generating a labeled 3D model by embedding the labeling data into the processed 3D model further comprises updating meta-data associated with the surface mesh.
15. The computer-implemented method of claim 1 , wherein displaying the VR environment further comprises altering the appearance of the environmental feature.
16. The computer-implemented method of claim 1 , wherein the user input comprises hand gestures made within the VR environment.
17. The computer-implemented method of claim 1 , wherein the user input comprises eye movement detected by the processor.
18. The computer-implemented method of claim 1 , wherein the user input comprises spoken commands.
19. The computer-implemented method of claim 1 , wherein generating a labeled 3D model by embedding the labeling data into the processed 3D model further comprises updating meta-data associated with the 3D model.
20. The computer-implemented method of claim 1 , wherein generating a labeled 3D model by embedding the labeling data into the processed 3D model further comprises generating a surface mesh, embedding the surface mesh in the 3D model, and updating meta-data associated with the surface mesh.
21. The computer-implemented method of claim 1 , wherein generating the training data further comprises:
receiving a second labeled 3D model; and
aggregating the labeled 3D model and the second labeled 3D model as training data.
22. The computer-implemented method of claim 1 , wherein generating the training data includes translating the labeled 3D model from a first file format to a second file format.
23. The computer-implemented method of claim 1 , wherein generating the training data further comprises:
capturing a first and second element of the labeled 3D model; and
aggregating the first and second elements of the labeled 3D model as training data.
24. A virtual reality (VR) labeling computer system for labeling a three dimensional (3D) model using VR techniques, the VR labeling computer system including at least one processor in communication with at least one memory device, wherein the at least one processor is configured to:
receive the 3D model;
process the 3D model using object recognition;
identify at least one environmental feature within the 3D model, wherein the at least one environmental feature is unidentified and unlabeled;
generate a processed 3D model including the at least one environmental feature;
display, through a VR device, a VR environment to a user based upon the processed 3D model, wherein the VR environment includes the at least one environmental feature;
prompt, through the VR environment, the user to input (i) labeling data for the environmental feature through user interaction with the VR device and (ii) boundary data by creating a bounding frame within the VR environment, wherein the labeling data identifies the environmental feature, wherein the bounding frame delineates a 3D region within the 3D model, and wherein the 3D region includes data representing the environmental feature;
process, by the VR device, (i) the data within the 3D region using object recognition and (ii) the user input associated with the environmental feature;
identify an environmental feature corresponding to the environmental feature represented by data within the 3D region;
generate a labeled 3D model by embedding the labeling data and the boundary data for the environmental feature into the processed 3D model;
extract the environmental feature and associated labeling data and boundary data from the labeled 3D model; and
generate training data based upon the extracted environmental feature of the labeled 3D model.
25. At least one non-transitory computer-readable storage media having computer-executable instructions embodied thereon for labeling a three dimensional (3D) model using a virtual reality (VR) technique, wherein when executed by at least one processor, the computer-executable instructions cause the processor to:
receive the 3D model;
process the 3D model using object recognition;
identify at least one environmental feature within the 3D model, wherein the at least one environmental feature is unidentified and unlabeled;
generate a processed 3D model including the at least one environmental feature;
display, through a VR device, a VR environment to a user based upon the processed 3D model, wherein the VR environment includes the at least one environmental feature;
prompt, through the VR environment, the user to input (i) labeling data for the environmental feature through user interaction with the VR device and (ii) boundary data by creating a bounding frame within the VR environment, wherein the labeling data identifies the environmental feature, wherein the bounding frame delineates a 3D region within the 3D model, and wherein the 3D region includes data representing the environmental feature;
process, by the VR device, (i) the data within the 3D region using object recognition and (ii) user input associated with the environmental feature;
identify an environmental feature corresponding to the environmental feature represented by data within the 3D region;
generate a labeled 3D model by embedding the labeling data and the boundary data for the environmental feature into the processed 3D model;
extract the environmental feature and associated labeling data and boundary data from the labeled 3D model; and
generate training data based upon the extracted environmental feature of the labeled 3D model.