High throughput point cloud processing
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing sensor data, e.g., laser sensor data, using neural networks. One of the methods includes obtaining a temporal sequence of multiple three-dimensional point clouds generated from sensor readings of an environment collected by one or more sensors within a given time period, each three-dimensional point cloud comprising a respective plurality of points in a first coordinate system; processing, using a feature extraction neural network, an input that comprises data derived from the temporal sequence of multiple three-dimensional point clouds to generate a feature embedding; receiving a query that specifies one time point within the given time period; and generating, from the feature embedding and conditioned on the query, one or more outputs that characterize one or more objects in the environment at the time point specified in the received query.
1 . A method comprising:
obtaining a temporal sequence of multiple three-dimensional point clouds generated from sensor readings of an environment collected by one or more sensors within a given time period, each three-dimensional point cloud comprising a respective plurality of points in a first coordinate system;
processing, using a feature extraction neural network, an input that comprises data derived from the temporal sequence of multiple three-dimensional point clouds to generate a feature embedding that corresponds to the given time period;
receiving a query that specifies a first time point and a second time point within the given time period;
generating a first identifier that uniquely identifies the first time point within the given time period, and a second identifier that uniquely identifies the second time point within the given time period;
generating, based on making a first forward pass through an output neural network using the first identifier and the feature embedding, one or more first outputs that characterize one or more objects in the environment at the first time point specified in the query; and
generating, based on making a second forward pass through the output neural network using the second identifier and the feature embedding, one or more second outputs that characterize the one or more objects in the environment at the second time point specified in the query.
2 . The method of claim 1 , wherein the query specifies a time point within the given time period at which a three-dimensional point cloud has not been generated because the one or more sensors have not completed a full sweep since a most recent full sweep.
3 . The method of claim 1 , wherein processing the input comprising data derived from the temporal sequence of multiple three-dimensional point clouds using the feature extraction neural network to generate the feature embedding comprises:
identifying, from each three-dimensional point cloud in the temporal sequence, (i) a first plurality of points that correspond to one or more dynamic objects in the environment that have moved during the given time period and (ii) a second plurality of points that correspond to one or more static objects in the environment that have not moved during the given time period;
generating multiple first sensor data segments that include the first plurality of points and a second sensor data segment that includes the second plurality of points;
processing an input comprising the multiple first sensor data segments and the second sensor data segment using the feature extraction neural network to generate the feature embedding.
4 . The method of claim 3 , wherein identifying the first plurality of points and the second plurality of points comprises:
generating a voxel representation that assigns each point of each three-dimensional point cloud in the temporal sequence to a respective voxel of a plurality of voxels.
5 . The method of claim 4 , wherein the plurality of voxels comprises one or more voxels having dynamic points the coordinate of which have changed during the given time period, and one or more voxels having static points the coordinate of which have not changed during the given time period.
6 . The method of claim 3 , wherein generating the multiple first sensor data segments that include the first plurality of points comprises adding time offsets to the multiple first sensor data segments.
7 . The method of claim 3 , wherein generating the multiple first sensor data segments that include the first plurality of points comprises:
generating one first sensor data segment corresponding to each three-dimensional point cloud included in the temporal sequence.
8 . The method of claim 1 , wherein the feature extraction neural network is configured to extract spatial temporal features of the plurality of points from the input.
9 . The method of claim 1 , wherein the one or more outputs that characterize the one or more objects comprises data defining a bounding box for each object.
10 . The method of claim 9 , wherein for each dynamic object in the environment, the output further comprises data defining a velocity of the dynamic object.
11 . The method of claim 1 , wherein the one or more outputs that characterize the one or more objects is an auto labeling output.
12 . The method of claim 1 , wherein generating the one or more first outputs comprises:
generating a first concatenation of based on concatenating the first identifier to the feature embedding; and
processing the concatenation using the output neural network to generate the one or more first outputs.
13 . The method of claim 1 , wherein the first identifier comprises a first sequence of numeric values, and wherein generating the first identifier comprises applying a sinusoidal temporal encoding scheme.
14 . A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
obtaining a temporal sequence of multiple three-dimensional point clouds generated from sensor readings of an environment collected by one or more sensors within a given time period, each three-dimensional point cloud comprising a respective plurality of points in a first coordinate system;
processing, using a feature extraction neural network, an input that comprises data derived from the temporal sequence of multiple three-dimensional point clouds to generate a feature embedding that corresponds to the given time period;
receiving a query that specifies a first time point and a second time point within the given time period;
generating a first identifier that uniquely identifies the first time point within the given time period, and a second identifier that uniquely identifies the second time point within the given time period;
generating, based on making a first forward pass through an output neural network using the first identifier and the feature embedding, one or more first outputs that characterize one or more objects in the environment at the first time point specified in the query; and
generating, based on making a second forward pass through the output neural network using the second identifier and the feature embedding, one or more second outputs that characterize the one or more objects in the environment at the second time point specified in the query.
15 . The system of claim 14 , wherein the query specifies a time point within the given time period at which a three-dimensional point cloud has not been generated because the one or more sensors have not completed a full sweep since a most recent full sweep.
16 . The system of claim 14 , wherein processing the input comprising data derived from the temporal sequence of multiple three-dimensional point clouds using the feature extraction neural network to generate the feature embedding comprises:
identifying, from each three-dimensional point cloud in the temporal sequence, (i) a first plurality of points that correspond to one or more dynamic objects in the environment that have moved during the given time period and (ii) a second plurality of points that correspond to one or more static objects in the environment that have not moved during the given time period;
generating multiple first sensor data segments that include the first plurality of points and a second sensor data segment that includes the second plurality of points;
processing an input comprising the multiple first sensor data segments and the second sensor data segment using the feature extraction neural network to generate the feature embedding.
17 . The system of claim 16 , wherein identifying the first plurality of points and the second plurality of points comprises:
generating a voxel representation that assigns each point of each three-dimensional point cloud in the temporal sequence to a respective voxel of a plurality of voxels.
18 . The system of claim 17 , wherein the plurality of voxels comprises one or more voxels having dynamic points the coordinate of which have changed during the given time period, and one or more voxels having static points the coordinate of which have not changed during the given time period.
19 . The system of claim 14 , wherein the first identifier comprises a first sequence of numeric values, and wherein generating the first identifier comprises applying a sinusoidal temporal encoding scheme.
20 . One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
obtaining a temporal sequence of multiple three-dimensional point clouds generated from sensor readings of an environment collected by one or more sensors within a given time period, each three-dimensional point cloud comprising a respective plurality of points in a first coordinate system;
processing, using a feature extraction neural network, an input that comprises data derived from the temporal sequence of multiple three-dimensional point clouds to generate a feature embedding that corresponds to the given time period;
receiving a query that specifies a first time point and a second time point within the given time period;
generating a first identifier that uniquely identifies the first time point within the given time period, and a second identifier that uniquely identifies the second time point within the given time period;
generating, based on making a first forward pass through an output neural network using the first identifier and the feature embedding, one or more first outputs that characterize one or more objects in the environment at the first time point specified in the query; and
generating, based on making a second forward pass through the output neural network using the second identifier and the feature embedding, one or more second outputs that characterize the one or more objects in the environment at the second time point specified in the query.