Neural radiance field for vehicle
A computer includes a processor and a memory, and the memory stores instructions executable by the processor to train a NeRF network to model a dynamic scene and, during the training, supervise the NeRF network with data from an event camera. The NeRF network is a neural radiance field modeling a geometry of the scene and a light intensity of the scene. The NeRF network includes a baseline network modeling the scene at an initial time and a deformation network modeling change to the scene since the initial time.
1 . A computer comprising a processor and a memory, the memory storing instructions executable by the processor to:
train a NeRF network to model a dynamic scene, the NeRF network being a neural radiance field modeling a geometry of the scene and a light intensity of the scene, the NeRF network including a baseline network modeling the scene at an initial time and a deformation network modeling change to the scene since the initial time;
during the training, supervise the NeRF network with data from an event camera, the data from the event camera including a plurality of events, each event being a change in the light intensity; and
update the NeRF network based on a loss function, the loss function including a nonevent loss based on a period between consecutive events at a pixel location;
wherein, in response to the event occurring over the period at a neighboring pixel location, the nonevent loss includes a difference between a predicted change in the light intensity at the pixel location over the period according to the NeRF network and a contrast threshold.
2 . The computer of claim 1 , wherein the instructions further include instructions to, after the training, actuate a component of a vehicle based on the NeRF network, the vehicle including the computer and the event camera.
3 . The computer of claim 1 , wherein each event includes a pixel location and a time of the change in the light intensity.
4 . The computer of claim 3 , wherein each event indicates that the change in the light intensity at the respective pixel location and the respective time is greater than the contrast threshold.
5 . The computer of claim 3 , wherein each event includes a polarity indicating a direction of the respective change in the light intensity.
6 . The computer of claim 1 , wherein the loss function includes an event loss based on the events at a pixel location.
7 . The computer of claim 6 , wherein the event loss includes a difference between the predicted change in the light intensity at the pixel location according to the NeRF network and a summation of the events at the pixel location.
8 . The computer of claim 1 , wherein the nonevent loss includes a difference between the predicted change in the light intensity at the pixel location over the period according to the NeRF network and a preset value.
9 . The computer of claim 1 , wherein, in response to a lack of an event occurring over the period at the neighboring pixel location, the nonevent loss enforces the predicted change in the light intensity at the pixel location over the period according to the NeRF network to be zero.
10 . The computer of claim 1 , wherein the baseline network receives a position and a direction as inputs, and the baseline network outputs a light intensity and a volume density, the light intensity and the volume density as seen in the direction from the position.
11 . The computer of claim 1 , wherein the deformation network receives a current position and a current time as inputs, and the deformation network outputs a spatial change from the initial time to the current time of a point that is at the current position at the current time.
12 . The computer of claim 11 , wherein the baseline network receives the current position adjusted by the spatial change as an input.
13 . The computer of claim 1 , wherein the baseline network and the deformation network are multilayer perceptrons.
14 . A method comprising:
training a NeRF network to model a dynamic scene, the NeRF network being a neural radiance field modeling a geometry of the scene and a light intensity of the scene, the NeRF network including a baseline network modeling the scene at an initial time and a deformation network modeling change to the scene since the initial time;
during the training, supervising the NeRF network with data from an event camera, the data from the event camera including a plurality of events, each event being a change in the light intensity; and
updating the NeRF network based on a loss function, the loss function including a nonevent loss based on a period between consecutive events at a pixel location;
wherein, in response to the event occurring over the period at a neighboring pixel location, the nonevent loss includes a difference between a predicted change in the light intensity at the pixel location over the period according to the NeRF network and a contrast threshold.
15 . The method of claim 14 , further comprising, after the training, actuating a component of a vehicle based on the NeRF network, the vehicle including the event camera.
16 . The method of claim 14 , wherein the deformation network receives a current position and a current time as inputs, the deformation network outputs a spatial change from the initial time to the current time of a point that is at the current position at the current time, and the baseline network receives the current position adjusted by the spatial change as an input.
17 . The computer of claim 1 , wherein:
in response to the event occurring over the period at the neighboring pixel location, the nonevent loss includes a first function of the predicted change in the light intensity at the pixel location over the period according to the NeRF network; and
in response to a lack of the event occurring over the period at the neighboring pixel location, the nonevent loss includes a second function of the predicted change in the light intensity at the pixel location over the period according to the NeRF network, the second function being different than the first function.
18 . The method of claim 14 , wherein:
in response to the event occurring over the period at the neighboring pixel location, the nonevent loss includes a first function of the predicted change in the light intensity at the pixel location over the period according to the NeRF network; and
in response to a lack of the event occurring over the period at the neighboring pixel location, the nonevent loss includes a second function of the predicted change in the light intensity at the pixel location over the period according to the NeRF network, the second function being different than the first function.