Spatio-temporal-interactive networks
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing point cloud data using spatio-temporal-interactive networks.
1. A method comprising:
obtaining a temporal sequence comprising respective point cloud inputs at each of a plurality of time steps during a time interval, each point cloud input comprising point cloud data generated from data captured by one or more sensors of a vehicle at the time step;
generating, from the temporal sequence, a respective feature representation for each of a plurality of time windows within the time interval, wherein each feature representation includes respective features for each of a plurality of spatial locations in the feature representation and each spatial location in each feature representation maps to a corresponding location in the environment;
processing the feature representations using a temporal region proposal neural network to generate a plurality of temporal region proposals, wherein:
each temporal region proposal corresponds to a possible agent in the environment,
each temporal region proposal identifies a respective spatial region in each of the feature representations that comprises a plurality of spatial locations in the feature representation, and
for each temporal region proposal, the respective spatial region identified by the temporal region proposal in any given feature representation is a prediction of where in the environment the corresponding possible agent was located during the time window corresponding to the given feature representation;
generating, for each temporal region proposal and from the feature representations, temporal region proposal features;
computing an interaction score for each pair of temporal region proposals in the plurality of temporal region proposals;
determining, for each temporal region proposal and from one or more of the computed interaction scores, an interaction embedding that represents interactions of the corresponding possible agent with the possible agents corresponding to the other temporal region proposals; and
for each temporal region proposal, processing the interaction embedding for the temporal feature proposal and the temporal region proposal features to generate a first output that predicts a future trajectory after the time interval of the possible agent corresponding to the temporal region proposal.
2. The method of claim 1 , wherein the possible agents are pedestrians.
3. The method of claim 1 , further comprising:
for each temporal region proposal, processing the temporal region proposal features to generate a second output that identifies a current location in the environment of the corresponding possible agent at the end of the time interval.
4. The method of claim 3 , wherein the second output also includes a final confidence score that represents a likelihood that the corresponding possible agent is an actual agent in the environment.
5. The method of claim 1 , wherein generating, for each temporal region proposal and from the feature representations, temporal region proposal features comprises:
generating local geometry features for the temporal region proposal by cropping each feature representation to include only the region identified by the temporal region proposal.
6. The method of claim 1 , wherein generating, for each temporal region proposal and from the feature representations, temporal region proposal features comprises:
generating local dynamics features for the temporal region proposal by:
identifying a region that includes all of the regions identified by the temporal region proposal in all of the feature representations; and
cropping each feature representation to include only the identified region.
7. The method of claim 6 , wherein the identified region is a smallest spatial region that includes all of the regions identified by the temporal region proposal in all of the feature representations.
8. The method of claim 1 , wherein generating, for each temporal region proposal and from the feature representations, temporal region proposal features comprises:
generating a history path feature for the temporal region proposal that identifies a location displacement of the region regions identified by the temporal region proposal across the feature representations.
9. The method of claim 8 , when also dependent on claims 6 and 7 , wherein generating temporal region proposal features comprises:
processing the local geometry features and the local dynamics features using a neural network to generate a combined feature; and
concatenating the combined feature with the history path feature.
10. The method of claim 1 , wherein each temporal region proposal corresponds to an anchor region, and wherein the temporal region proposal neural network generates an output that includes, for each temporal region proposal and for each feature representation, a regression vector that identifies a region in each feature representation relative to the anchor region corresponding to the temporal region proposal.
11. The method of claim 10 , wherein the anchor region is defined by coordinates of a box in a feature representation and a heading of the box, and wherein each regression vector includes a respective delta value for each of the coordinates and for the heading.
12. The method of claim 1 , wherein the temporal region proposal neural network generates a plurality of candidate temporal region proposals and a confidence score for each of the candidate temporal region proposals that indicates a predicted likelihood that the candidate temporal region proposal corresponds to an actual agent in the environment, and wherein generating the plurality of temporal region proposals comprises:
applying non-maximum suppression to the regions of the feature representation corresponding to the most recent time window identified by the candidate temporal region proposals based on the confidence scores to remove redundant candidate temporal region proposals.
13. A method performed by one or more computers, the method comprising:
obtaining a temporal sequence of point cloud data for a time interval;
generating, from the temporal sequence, respective feature representations for each of a plurality of time windows within the time interval;
processing, using a temporal region proposal neural network, the feature representations to generate a plurality of temporal region proposals, each temporal region proposal corresponding to a different possible agent;
generating, for each temporal region proposal and from the feature representations, temporal region proposal features;
computing an interaction score for each pair of temporal region proposals in the plurality of temporal region proposals;
determining, for each temporal region proposal and from one or more of the computed interaction scores, an interaction embedding that represents interactions of the corresponding possible agent with the possible agents corresponding to the other temporal region proposals; and
for each temporal region proposal, processing the interaction embedding for the temporal feature proposal and the temporal region proposal features to generate a first output that predicts a future trajectory after the time interval of the possible agent corresponding to the temporal region proposal.
14. The method of claim 13 , further comprising:
for each temporal region proposal, processing the temporal region proposal features to generate a second output that identifies a current location in the environment of the corresponding possible agent at the end of the time interval.
15. The method of claim 14 , wherein the second output also includes a final confidence score that represents a likelihood that the corresponding possible agent is an actual agent in the environment.
16. The method of claim 13 , wherein each temporal region proposal corresponds to an anchor region, and wherein the temporal region proposal neural network generates an output that includes, for each temporal region proposal and for each feature representation, a regression vector that identifies a region in each feature representation relative to the anchor region corresponding to the temporal region proposal.
17. The method of claim 16 , wherein generating, for each temporal region proposal and from the feature representations, temporal region proposal features comprises:
generating local geometry features for the temporal region proposal by cropping each feature representation to include only the region identified by the temporal region proposal.
18. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
obtaining a temporal sequence of point cloud data for a time interval;
generating, from the temporal sequence, respective feature representations for each of a plurality of time windows within the time interval;
processing, using a temporal region proposal neural network, the feature representations to generate a plurality of temporal region proposals, each temporal region proposal corresponding to a different possible agent;
generating, for each temporal region proposal and from the feature representations, temporal region proposal features;
computing an interaction score for each pair of temporal region proposals in the plurality of temporal region proposals;
determining, for each temporal region proposal and from one or more of the computed interaction scores, an interaction embedding that represents interactions of the corresponding possible agent with the possible agents corresponding to the other temporal region proposals; and
for each temporal region proposal, processing the interaction embedding for the temporal feature proposal and the temporal region proposal features to generate a first output that predicts a future trajectory after the time interval of the possible agent corresponding to the temporal region proposal.