Method and apparatus for representing dynamic neural radiance fields from unsynchronized videos
The present disclosure relates to a method and apparatus for representing dynamic neural radiance fields from unsynchronized videos. A method of acquiring a video at an arbitrary viewpoint based on a dynamic neural radiance fields model according to an embodiment of the present disclosure may comprise: inputting one or more videos acquired from one or more views for one scene into the dynamic neural radiance fields model; inputting a time embedding for the one or more videos into the dynamic neural radiance fields model; and rendering the video at the arbitrary viewpoint based on color information and density information output by the dynamic neural radiance fields model. Herein, time synchronization related to the time embedding may be performed by applying an individual time offset learned for each view, for the one or more views.
1 . A method of acquiring a video at an arbitrary viewpoint based on a dynamic neural radiance fields model, the method comprising:
inputting one or more videos acquired from one or more views for one scene into the dynamic neural radiance fields model;
inputting a time embedding for the one or more videos into the dynamic neural radiance fields model; and
rendering the one or more videos at the arbitrary viewpoint based on color information and density information output by the dynamic neural radiance fields model,
wherein time synchronization related to the time embedding is performed by applying an individual time offset learned for each view, for the one or more views.
2 . The method of claim 1 ,
wherein the time offset corresponds to a learnable parameter.
3 . The method of claim 2 ,
wherein the time offset is repeatedly learned to reduce an error between the rendered one or more videos and ground-truth image.
4 . The method of claim 1 ,
wherein the time offset is applied based on a time of a reference video for the one scene.
5 . The method of claim 1 ,
wherein, in a case that the dynamic neural radiance fields model is a model with embeddings at discrete times, the time corrected using the time offset is used as an input to a spatial-time plane, for the time embedding.
6 . The method of claim 5 ,
wherein, the spatial-time plane is normalized to [−a, a] for the dynamic neural radiance fields model, and
wherein a is a value greater than 0 and less than 1.
7 . The method of claim 1 ,
wherein, in a case that the dynamic neural radiance fields model is a model with embeddings for each continuous time, the dynamic neural radiance fields model further includes a specific neural network that outputs a corrected time embedding based on the time offset.
8 . The method of claim 7 ,
wherein the specific neural network includes a fully connected neural network based on two layers.
9 . The method of claim 1 ,
wherein each of the one or more videos corresponds to a 3-dimensional video based on 3-dimensional coordinates.
10 . An apparatus of acquiring a video at an arbitrary viewpoint based on a dynamic neural radiance fields model, the apparatus comprising:
at least one processor and at least one memory,
wherein the at least one processor is configured to:
input one or more videos acquired from one or more views for one scene into the dynamic neural radiance fields model;
input a time embedding for the one or more videos into the dynamic neural radiance fields model; and
render the one or more videos at the arbitrary viewpoint based on color information and density information output by the dynamic neural radiance fields model,
wherein time synchronization related to the time embedding is performed by applying an individual time offset learned for each view, for the one or more views.
11 . The apparatus of claim 10 ,
wherein the time offset corresponds to a learnable parameter.
12 . The apparatus of claim 11 ,
wherein the time offset is repeatedly learned to reduce an error between the rendered one or more videos and ground-truth image.
13 . The apparatus of claim 10 ,
wherein the time offset is applied based on a time of a reference video for the one scene.
14 . The apparatus of claim 10 ,
wherein, in a case that the dynamic neural radiance fields model is a model with embeddings at discrete times, the time corrected using the time offset is used as the input to the spatial-time plane, for the time embedding.
15 . The apparatus of claim 14 ,
wherein, the spatial-time plane is normalized to [−a, a] for the dynamic neural radiance fields model, and
wherein a is a value greater than 0 and less than 1.
16 . The apparatus of claim 10 ,
wherein, in a case that the dynamic neural radiance fields model is a model with embeddings for each continuous time, the dynamic neural radiance fields model further includes a specific neural network that outputs a corrected time embedding based on the time offset.
17 . The apparatus of claim 16 ,
wherein the specific neural network includes a fully connected neural network based on two layers.
18 . The apparatus of claim 10 ,
wherein each of the one or more videos corresponds to a 3-dimensional video based on 3-dimensional coordinates.
19 . One or more non-transitory computer readable medium storing one or more instructions,
wherein the one or more instructions are executed by one or more processors and control an apparatus for acquiring a video at an arbitrary viewpoint based on a dynamic neural radiance fields model to:
input one or more videos acquired from one or more views for one scene into the dynamic neural radiance fields model;
input a time embedding for the one or more videos into the dynamic neural radiance fields model; and
render the one or more videos at the arbitrary viewpoint based on color information and density information output by the dynamic neural radiance fields model,
wherein time synchronization related to the time embedding is performed by applying an individual time offset learned for each view, for the one or more views.
20 . The computer readable medium of claim 19 ,
wherein the time offset corresponds to a learnable parameter.