Natural language 3D data searching

1 . A method, comprising:

pre-processing three-dimensional (3D) data for a captured scene to generate a 3D point cloud;

extracting 3D samples from the 3D point cloud;

extracting, by projection, two-dimensional (2D) samples from the 3D point cloud, the 2D samples forming a representative summary of the 3D point cloud representing objects in the captured scene;

segmenting the 2D samples into 2D segments corresponding to the objects in the captured scene;

image encoding, with a pre-trained encoder, each of the 2D segments based on 3D points to which a respective 2D segment corresponds in order to form a portion of multi-modal embeddings for the captured scene;

masking the 3D samples using information relating to the 2D segments to produce 3D segments;

incorporating information from the multi-modal embeddings in an indexed point-cloud data structure for the 3D point cloud representing the objects in the captured scene with 3D spatial information for the captured scene;

performing at least one of querying or retrieving 3D point-cloud data from the indexed point-cloud data structure based on a user input including at least one of natural language or image references; and

enabling global scene understanding and local scene understanding using instance segmentation in conjunction with the multi-modal embeddings.

2 . The method of claim 1 , further comprising:

using the 2D samples and the 3D samples, producing a spatial data structure for the captured scene.

3 . The method of claim 2 , wherein producing the spatial data structure includes:

determining group tracking information and a group identifier for the 3D segments.

4 . The method of claim 3 , wherein enabling global scene understanding and local scene understanding using instance segmentation in conjunction with the multi-modal embeddings includes:

storing the multi-modal embeddings, the 3D points to which any of the 2D segments corresponds, the group tracking information, and the group identifier in the spatial data structure.

5 . The method of claim 3 , wherein producing the spatial data structure includes:

receiving camera pose information relating to the 2D samples and the 3D samples; and

employing the camera pose information in determining the group tracking information and the group identifier for the 3D segments by tracking parts of the 3D point cloud that have been sampled and a corresponding sampling resolution for each sampled part.

6 . The method of claim 5 , wherein employing the camera pose information includes:

assigning an embedding vector for an entire segment set to every 3D point contained in the entire segment set;

checking each frame for one of the entire segment sets for 3D points seen in a previously processed, different one of the entire segment sets;

assigning a new group identifier to 3D points not seen in any previously processed segment set;

assigning a previously assigned group identifier to 3D points seen in at least one previously processed segment set; and

averaging embeddings for overlapping images.

7 . The method of claim 3 , wherein enabling global scene understanding and local scene understanding using instance segmentation in conjunction with the multi-modal embeddings includes:

identifying and extracting subsets of the 3D point cloud using a 2D semantic segmentation model generating one of a discrete set of class labels or masks.

8 . The method of claim 3 , wherein performing at least one of querying or retrieving 3D-point-cloud data based on user input including at least one of the natural language or the image references includes:

performing a scan over the multi-modal embeddings in a spatial database based on the user input;

discarding entries in the spatial database that have a similarity score less than a defined threshold; and

selecting, from a set of point clouds corresponding to the 2D segments and the 3D segments remaining after discarding entries, a final group of point clouds based on one or more metrics selected from aggregate confidence, size, or proximity.

9 . An apparatus comprising:

at least one processing device configured to:

pre-process three-dimensional (3D) data for a captured scene to generate a 3D point cloud;

extract 3D samples from the 3D point cloud;

extract, by projection, two-dimensional (2D) samples from the 3D point cloud, the 2D samples forming a representative summary of the 3D point cloud representing objects in the captured scene;

segment 2D samples into 2D segments corresponding to the objects in the captured scene;

image encode, with a pre-trained encoder, each of the 2D segments based on 3D points to which a respective 2D segment corresponds in order to form a portion of multi-modal embeddings for the captured scene;

mask the 3D samples using information relating to the 2D segments to produce 3D segments;

incorporate information from the multi-modal embeddings in an indexed point-cloud data structure for the 3D point cloud representing the objects in the captured scene with 3D spatial information for the captured scene;

perform at least one of querying or retrieving 3D point-cloud data from the indexed point-cloud data structure based on a user input including at least one of natural language or image references; and

enable global scene understanding and local scene understanding using instance segmentation in conjunction with the multi-modal embeddings.

10 . The apparatus of claim 9 , wherein the at least one processing device is further configured, using the 2D samples and the 3D samples, to produce a spatial data structure for the captured scene.

11 . The apparatus of claim 10 , wherein, to produce the spatial data structure, the at least one processing device is configured to determine group tracking information and a group identifier for the 3D segments.

12 . The apparatus of claim 11 , wherein, to enable global scene understanding and local scene understanding using instance segmentation in conjunction with the multi-modal embeddings, the at least one processing device is configured to store the multi-modal embeddings, the 3D points to which any of the 2D segments corresponds, the group tracking information, and the group identifier in the spatial data structure.

13 . The apparatus of claim 11 , wherein, to produce the spatial data structure, the at least one processing device is configured to:

receive camera pose information relating to the 2D samples and the 3D samples; and

employ the camera pose information in determining the group tracking information and the group identifier for the 3D segments by tracking parts of the 3D point cloud that have been sampled and a corresponding sampling resolution for each sampled part.

14 . The apparatus of claim 13 , wherein, to employ the camera pose information, the at least one processing device is configured to:

assign an embedding vector for an entire segment set to every 3D point contained in the entire segment set;

check each frame for one of the entire segment sets for 3D points seen in a previously processed, different one of the entire segment sets;

assign a new group identifier to 3D points not seen in any previously processed segment set;

assign a previously assigned group identifier to 3D points seen in at least one previously processed segment set; and

average embeddings for overlapping images.

15 . The apparatus of claim 11 , wherein, to enable global scene understanding and local scene understanding using instance segmentation in conjunction with the multi-modal embeddings, the at least one processing device is configured to identify and extract subsets of the 3D point cloud using a 2D semantic segmentation model generating one of a discrete set of class labels or masks.

16 . The apparatus of claim 11 , wherein, to perform at least one of querying or retrieving 3D-point-cloud data based on user input, the at least one processing device is configured to:

perform a scan over the multi-modal embeddings in a spatial database based on the user input;

discard entries in the spatial database that have a similarity score less than a defined threshold; and

select, from a set of point clouds corresponding to the 2D segments and the 3D segments remaining after discarding entries, a final group of point clouds based on one or more metrics selected from aggregate confidence, size, or proximity.

17 . A non-transitory computer readable medium containing instructions that when executed cause at least one processor of an electronic device to:

pre-process three-dimensional (3D) data for a captured scene to generate a 3D point cloud;

extract 3D samples from the 3D point cloud;

extract, by projection, two-dimensional (2D) samples from the 3D point cloud, the 2D samples forming a representative summary of the 3D point cloud representing objects in the captured scene;

segment 2D samples into 2D segments corresponding to the objects in the captured scene;

image encode, with a pre-trained encoder, each of the 2D segments based on 3D points to which a respective 2D segment corresponds in order to form a portion of multi-modal embeddings for the captured scene;

mask the 3D samples using information relating to the 2D segments to produce 3D segments;

incorporate information from the multi-modal embeddings in an indexed point-cloud data structure for the 3D point cloud representing the objects in the captured scene with 3D spatial information for the captured scene;

perform at least one of querying or retrieving 3D point-cloud data from the indexed point-cloud data structure based on a user input including at least one of natural language or image references; and

enable global scene understanding and local scene understanding using instance segmentation in conjunction with the multi-modal embeddings.

18 . The non-transitory computer readable medium of claim 17 , further containing instructions that when executed cause the at least one processor to:

produce, using the 2D samples and the 3D samples, a spatial data structure for the captured scene.

19 . The non-transitory computer readable medium of claim 18 , wherein the instructions that when executed cause the at least one processor to produce the spatial data structure comprise instructions that when executed cause the at least one processor to:

determine group tracking information and a group identifier for the 3D segments.

20 . The non-transitory computer readable medium of claim 19 , wherein the instructions that when executed cause the at least one processor to enable global scene understanding and local scene understanding using instance segmentation in conjunction with the multi-modal embeddings comprise instructions that when executed cause the at least one processor to:

store the multi-modal embeddings, the 3D points to which any of the 2D segments corresponds, the group tracking information, and the group identifier in the spatial data structure.