Systems and Methods for Dynamic Object Removal from Three-Dimensional Data
Systems and methods for generating simulation data based on real-world environments are provided. A method includes obtaining multi-modal sensor data indicative of a dynamic object within an environment of a robotic platform. The multi-modal sensor data is associated with a plurality of timesteps including a first timestep and a second timestep. The method includes providing the multi-modal sensor data indicative of the dynamic object within the environment as an input to a machine-learned dynamic object removal model. And, the method includes receiving as an output of the machine-learned dynamic object removal model, in response to receipt of the multi-modal sensor data, a scene representation indicative of at least a portion of the environment including a reconstructed region based at least in part on removal of the dynamic object and multiple levels of granularity. The scene representation is used as a template for generating different simulations within the depicted environment.
1 . A computer-implemented method comprising:
obtaining sensor data indicative of a plurality of views of a dynamic object within an environment of an autonomous vehicle, the dynamic object occluding a region of the environment;
generating, by a machine-learned dynamic object removal model, a plurality of features corresponding to the plurality of views;
generating one or more attention weights based on the plurality of features;
generating an updated feature based on the plurality of features, the updated feature based on a weighted combination that is weighted based on the one or more attention weights; and
generating, by the machine-learned dynamic object removal model and based on the updated feature, a scene representation output indicative of at least a portion of the environment comprising a reconstructed region based on removal of the dynamic object, wherein the reconstructed region comprises inpainted data describing the region of the environment occluded by the dynamic object.
2 . The computer-implemented method of claim 1 , comprising:
generating the one or more attention weights is based on attending over patches within a frame.
3 . The computer-implemented method of claim 2 , wherein the patches correspond to the plurality of views.
4 . The computer-implemented method of claim 1 , wherein the inpainted data comprises inpainted pixel data.
5 . The computer-implemented method of claim 1 , wherein the inpainted data comprises inpainted depth data.
6 . The computer-implemented method of claim 1 , wherein the sensor data comprises a plurality of image frames respectively associated with a plurality of viewpoints based on orientations of corresponding image capturing devices.
7 . The computer-implemented method of claim 1 , wherein the sensor data comprises a plurality of image frames respectively associated with a plurality of timesteps.
8 . The computer-implemented method of claim 1 , comprising:
generating simulation data based at least in part on the scene representation output.
9 . The computer-implemented method of claim 8 , wherein the simulation data comprises:
a simulated environment that is based at least in part on the scene representation output; and
one or more simulated dynamic objects designed to move within the simulated environment.
10 . The computer-implemented method of claim 9 , comprising:
training a machine-learned model of an autonomous vehicle computing system using the simulation data to simulate one or more inputs to the machine-learned model.
11 . The computer-implemented method of claim 1 , wherein the sensor data comprises a plurality of modalities of sensor data, wherein at least one modality of the plurality of modalities comprises a three-dimensional representation of the dynamic object.
12 . A computing system, comprising:
one or more processors; and
one or more computer-readable media storing instructions executable by the one or more processors to cause the computing system to perform operations, the operations comprising:
obtaining sensor data indicative of a plurality of views of a dynamic object within an environment of an autonomous vehicle, the dynamic object occluding a region of the environment;
generating, by a machine-learned dynamic object removal model, a plurality of features corresponding to the plurality of views;
generating one or more attention weights based on the plurality of features;
generating an updated feature based on the plurality of features, the updated feature based on a weighted combination that is weighted based on the one or more attention weights; and
generating, by the machine-learned dynamic object removal model and based on the updated feature, a scene representation output indicative of at least a portion of the environment comprising a reconstructed region based on removal of the dynamic object, wherein the reconstructed region comprises inpainted data describing the region of the environment occluded by the dynamic object.
13 . The computing system of claim 12 , the operations comprising:
generating the one or more attention weights is based on attending over patches within a frame.
14 . The computing system of claim 13 , wherein the patches correspond to the plurality of views.
15 . The computing system of claim 12 , wherein the inpainted data comprises inpainted pixel data.
16 . The computing system of claim 12 , wherein the inpainted data comprises inpainted depth data.
17 . The computing system of claim 12 , wherein the sensor data comprises a plurality of image frames respectively associated with a plurality of viewpoints based on orientations of corresponding image capturing devices.
18 . The computing system of claim 12 , wherein the sensor data comprises a plurality of image frames respectively associated with a plurality of timesteps.
19 . The computing system of claim 12 , the operations comprising:
generating simulation data based at least in part on the scene representation output, wherein the simulation data comprises:
a simulated environment that is based at least in part on the scene representation output; and
one or more simulated dynamic objects designed to move within the simulated environment; and
training a machine-learned model of an autonomous vehicle computing system using the simulation data to simulate one or more inputs to the machine-learned model.
20 . One or more computer-readable media storing instructions executable by one or more processors to cause a computing system to perform operations, the operations comprising:
obtaining sensor data indicative of a plurality of views of a dynamic object within an environment of an autonomous vehicle, the dynamic object occluding a region of the environment;
generating, by a machine-learned dynamic object removal model, a plurality of features corresponding to the plurality of views;
generating one or more attention weights based on the plurality of features;
generating an updated feature based on the plurality of features, the updated feature based on a weighted combination that is weighted based on the one or more attention weights; and
generating, by the machine-learned dynamic object removal model and based on the updated feature, a scene representation output indicative of at least a portion of the environment comprising a reconstructed region based on removal of the dynamic object, wherein the reconstructed region comprises inpainted data describing the region of the environment occluded by the dynamic object.