System and method of conditional neural floorplans for static-dynamic disentanglement

1 . A method of conditional neural ground planes for static-dynamic disentanglement, comprising:

extracting, using a convolutional neural network (CNN), CNN image features from an image to form a feature tensor;

resampling unprojected 2D features of the feature tensor to form feature pillars;

aggregating the feature pillars to form an entangled neural ground plane; decomposing the entangled neural ground plane using a 2D CNN to decompose a 2D grid of features represented by the entangled neural ground plane into a static neural ground plane and a dynamic neural ground plane; and

planning a trajectory of an ego vehicle according to 3D bounding boxes estimated in a scene surrounding the ego vehicle from a depth map of the static neural ground plane and the dynamic neural ground plane.

2 . The method of claim 1 , in which resampling comprises unprojecting the CNN image features along their respective rays as parameterized via an intrinsic parameter and an extrinsic parameter of an input camera to create a discrete volume in a world space.

3 . The method of claim 1 , in which aggregating comprises:

computing, using pillar aggregation, multilayer perception (MLP) softmax scores, in which α i =P(f i , x i ), where P(·) denotes the MLP and x i is a coordinate of an i-th point on the feature pillars f i ; and

aggregating the features as g=Σ i α i f i .

4 . The method of claim 1 , in which decomposing further compromises compositing, per-frame, of the dynamic neural ground plane into a time-invariant dynamic neural ground plane via differentiable volume rendering.

5 . The method of claim 1 , in which decomposing further comprises:

selecting context frames from video captured by an input camera;

extracting a compacted, static ground plane and a compacted dynamic ground plane per each of the selected context frames; and

pooling the compacted, static ground plane into a time-invariant static ground plane.

6 . The method of claim 1 , further comprising:

estimating the depth map from the static neural ground plane and the dynamic neural ground plane; and

estimating the 3D bounding boxes from the depth map of the static neural ground plane and the dynamic neural ground plane.

7 . A non-transitory computer-readable medium having program code recorded thereon of conditional neural ground planes for static-dynamic disentanglement, the program code being executed by a processor and comprising:

program code to extract, using a convolutional neural network (CNN), CNN image features from an image to form a feature tensor;

program code to resample unprojected 2D features of the feature tensor to form feature pillars;

program code to aggregate the feature pillars to form an entangled neural ground plane;

program code to decompose the entangled neural ground plane using a 2D CNN to decompose a 2D grid of features represented by the entangled neural ground plane into a static neural ground plane and a dynamic neural ground plane; and

program code to plan a trajectory of an ego vehicle according to 3D bounding boxes estimated in a scene surrounding the ego vehicle from a depth map of the static neural ground plane and the dynamic neural ground plane.

8 . The non-transitory computer-readable medium of claim 7 , in which the program code to resample comprises program code to unproject the CNN image features along their respective rays as parameterized via an intrinsic parameter and an extrinsic parameter of an input camera to create a discrete volume in a world space.

9 . The non-transitory computer-readable medium of claim 7 , in which the program code to aggregate comprises:

program code to compute, using pillar aggregation, multilayer perception (MLP) softmax scores, in which α i =P(f i , x i ), where P(·) denotes the MLP and x i is a coordinate of an i-th point on the feature pillars f i ; and

program code to aggregate the features as g=Σ i α i f i .

10 . The non-transitory computer-readable medium of claim 7 , in which the program code to decompose further comprises program code to composite, per-frame, of the dynamic neural ground plane into a time-invariant dynamic neural ground plane via differentiable volume rendering.

11 . The non-transitory computer-readable medium of claim 7 , in which the program code to decompose further comprises:

program code to select context frames from video captured by an input camera;

program code to extract a compacted, static ground plane and a compacted dynamic ground plane per each of the selected context frames; and

program code to pool the compacted, static ground plane into a time-invariant static ground plane.

12 . The non-transitory computer-readable medium of claim 7 , further comprising:

program code to estimate the depth map from the static neural ground plane and the dynamic neural ground plane; and

program code to estimate the 3D bounding boxes from the depth map of the static neural ground plane and the dynamic neural ground plane.

13 . A system of conditional neural ground planes for static-dynamic disentanglement, the system comprising:

a feature tensor formation model to extract, using a convolutional neural network (CNN), CNN image features from an image to form a feature tensor;

a feature pillar formation module to resample unprojected 2D features of the feature tensor to form feature pillars;

a neural ground plane formation model to aggregate the feature pillars to form an entangled neural ground plane;

a static-dynamic disentanglement model to decompose the entangled neural ground plane using a 2D CNN to decompose a 2D grid of features represented by the entangled neural ground plane into a static neural ground plane and a dynamic neural ground plane; and

a planner to plan a trajectory of an ego vehicle according to 3D bounding boxes estimated in a scene surrounding the ego vehicle from a depth map of the static neural ground plane and the dynamic neural ground plane.

14 . The system of claim 13 , in which the static-dynamic disentanglement model is further to composite, per-frame, of the dynamic neural ground plane into a time-invariant dynamic neural ground plane via differentiable volume rendering.

15 . The system of claim 13 , further comprising:

a perception module to estimate the depth map from the static neural ground plane and the dynamic neural ground plane and to estimate the 3D bounding boxes from the depth map of the static neural ground plane and the dynamic neural ground plane.