Neural implicit scattering functions for inverse parameter estimation and dynamics modeling of multi-object interactions
A method for dynamic modeling and manipulation of multi-object scenes is described. The method includes using object-centric neural implicit scattering functions (OSFs) as object representations in a model-predictive control (MPC) framework for the multi-object scenes. The method also includes modeling a per-object light transport to enable compositional scene re-rendering under object rearrangement and varying lighting conditions. The method further includes applying inverse parameter estimation and graph neural network (GNN) dynamics models to estimate initial object poses and a light position in the multi-object scene. The method also includes manipulating an object perceived in the multi-object scene according to the applying of the inverse parameter estimation and the GNN dynamics models.
1 . A method for dynamic modeling and manipulation of multi-object scenes, the method comprising:
using object-centric neural implicit scattering functions (OSFs) as object representations in a model-predictive control (MPC) framework for the multi-object scenes;
modeling a per-object light transport to enable compositional scene re-rendering under object rearrangement and varying lighting conditions;
applying inverse parameter estimation and graph neural network (GNN) dynamics models to estimate initial object poses and a light position in the multi-object scene; and
manipulating an object perceived in the multi-object scene according to the applying of the inverse parameter estimation and the GNN dynamics models.
2 . The method of claim 1 , in which the multi-object scene exhibits multi-object interactions in extreme and/or harsh lighting conditions.
3 . The method of claim 1 , further comprising:
training a dynamics model using simulated data;
inputting 6D object poses of each object in the multi-object scene and a pusher's action; and
outputting future 6D poses for each object in the multi-object scene.
4 . The method of claim 3 , further comprising:
estimating the initial object poses in the multi-object scene and the light positions;
inferring the initial object poses in the multi-object scene and the light positions using the inverse parameter estimation using the OSFs;
performing model predictive control using the GNN dynamics model; and
updating object states at each step using the inverse parameter estimation.
5 . The method of claim 1 , further comprising training the GNN dynamics model using simulated data, including inputting 6D object poses of each object in the scene and a pusher's action, and to output future 6-D poses for each object.
6 . The method of claim 1 , further comprising using the inverse parameter estimation with OSF models to estimate the initial object poses and the light position at inference time.
7 . The method of claim 1 , further comprising using a learned, GNN dynamics model to perform model-predictive control while updating object states at each step using the inverse parameter estimation.
8 . The method of claim 1 , further comprising planning pushing and rearranging of an object by a robot according to predictions from video captured by the robot.
9 . A non-transitory computer-readable medium having program code recorded thereon for dynamic modeling and manipulation of multi-object scenes, the program code being executed by a processor and comprising:
program code to use object-centric neural implicit scattering functions (OSFs) as object representations in a model-predictive control (MPC) framework for the multi-object scenes;
program code to model a per-object light transport to enable compositional scene re-rendering under object rearrangement and varying lighting conditions;
program code to apply inverse parameter estimation and graph neural network (GNN) dynamics models to estimate initial object poses and a light position in the multi-object scene; and
program code to manipulate an object perceived in the multi-object scene according to the applying of the inverse parameter estimation and the GNN dynamics models.
10 . The non-transitory computer-readable medium of claim 9 , in which the multi-object scene exhibits multi-object interactions in extreme and/or harsh lighting conditions.
11 . The non-transitory computer-readable medium of claim 9 , further comprising:
program code to train a dynamics model using simulated data;
program code to input 6D object poses of each object in the multi-object scene and a pusher's action; and
program code to output future 6D poses for each object in the multi-object scene.
12 . The non-transitory computer-readable medium of claim 11 , further comprising:
program code to estimate the initial object poses in the multi-object scene and the light positions;
program code to infer the initial object poses in the multi-object scene and the light positions using the inverse parameter estimation using the OSFs;
program code to perform model predictive control using the GNN dynamics model; and
program code to update object states at each step using the inverse parameter estimation.
13 . The non-transitory computer-readable medium of claim 9 , further comprising program code to train the GNN dynamics model using simulated data, including inputting 6D object poses of each object in the scene and a pusher's action, and to output future 6-D poses for each object.
14 . The non-transitory computer-readable medium of claim 9 , further comprising program code to use inverse parameter estimation with OSF models to estimate the initial object poses and the light position at inference time.
15 . The non-transitory computer-readable medium of claim 9 , further comprising program code to use a learned, GNN dynamics model to perform model-predictive control while updating object states at each step using the inverse parameter estimation.
16 . The non-transitory computer-readable medium of claim 9 , further comprising planning pushing and rearranging of an object by a robot according to predictions from video captured by the robot.
17 . A system for dynamic modeling and manipulation of multi-object scenes, the system comprising:
a framework module to use object-centric neural implicit scattering functions (OSFs) as object representations in a model-predictive control (MPC) framework for the multi-object scenes;
a per object light transfer model to model a per-object light transport to enable compositional scene re-rendering under object rearrangement and varying lighting conditions;
a graph-based neural dynamics model to apply inverse parameter estimation and graph neural network (GNN) dynamics models to estimate initial object poses and a light position in the multi-object scene; and
an object manipulation module to manipulate an object perceived in the multi-object scene according to the applying of the inverse parameter estimation and the GNN dynamics models.
18 . The system of claim 17 , in which the multi-object scene exhibits multi-object interactions in extreme and/or harsh lighting conditions.
19 . The system of claim 17 , in which the graph-based neural dynamics model is further to use a learned, GNN dynamics model to perform model-predictive control while updating object states at each step using the inverse parameter estimation.
20 . The system of claim 17 , further comprising a planner module to push and/or rearrange an object by a robot according to predictions from video captured by the robot.