Methods and processors for rendering a 3D object using multi-camera image inputs
View Patent ↗Methods and processors for rendering a 3D object are disclosed. The method includes acquiring multi-camera image input including first image frames of the 3D object generated by a first camera and second image frames of the 3D object generated by a second camera, acquiring an initial 3D Gaussian Splatting (3DGS) model having a plurality of initial parameters including an initial frame-wise GS parameter and an initial camera-wise GS parameter, generating an adjusted 3DGS model by adjusting, based on the multi-camera image input, at least one of: the initial frame-wise GS parameter, the initial camera-wise GS parameter, generating, by the adjusted 3DGS model, a 3DGS output and rendering a 2D image of the 3D object using the 3DGS output.
1 . A method rendering a three-dimensional (3D) object, the method executable by a processor, the method comprising:
acquiring multi-camera image input including first image frames of the 3D object generated by a first camera and second image frames of the 3D object generated by a second camera, the first camera being different from the second camera;
acquiring an initial 3D Gaussian Splatting (3DGS) model having a plurality of initial parameters, the plurality of initial parameters including an initial frame-wise Gaussian Splatting (GS) parameter and an initial camera-wise GS parameter;
generating an adjusted 3DGS model by adjusting, based on the multi-camera image input, at least one of:
(i) the initial frame-wise GS parameter to compensate for misalignment between the first image frames and the second image frames; and
(ii) the initial camera-wise GS parameter to compensate for cross-camera variation between the first camera and the second camera;
wherein the adjusting the initial frame-wise GS parameter is executed in accordance with:
μ
′
=
M
R
μ
+
M
T
wherein μ is an initial frame-wise GS position parameter, μ′ is an adjusted frame-wise GS position parameter, M_R is a rotation matrix, and M_T is a translation matrix;
generating, by the adjusted 3DGS model, a 3DGS output based on the multi-camera image input;
rendering a two-dimensional (2D) image of the 3D object using the 3DGS output.
2 . The method of claim 1 , wherein the generating the adjusted 3DGS model comprises adjusting both the initial frame-wise GS parameter and the initial camera-wise GS parameter.
3 . The method of claim 1 , wherein the initial frame-wise GS parameter is an initial GS position parameter, and the initial camera-wise GS parameter is an initial GS Spherical Harmonics (SH) parameter.
4 . The method of claim 1 , wherein the plurality of initial parameters further comprises a GS rotation parameter, a GS scale parameter, and a GS opacity parameter.
5 . A method rendering a three-dimensional (3D) object, the method executable by a processor, the method comprising:
acquiring multi-camera image input including first image frames of the 3D object generated by a first camera and second image frames of the 3D object generated by a second camera, the first camera being different from the second camera;
acquiring an initial 3D Gaussian Splatting (3DGS) model having a plurality of initial parameters, the plurality of initial parameters including an initial frame-wise Gaussian Splatting (GS) parameter and an initial camera-wise GS parameter;
generating an adjusted 3DGS model by adjusting, based on the multi-camera image input, at least one of:
(iii) the initial frame-wise GS parameter to compensate for misalignment between the first image frames and the second image frames; and
(iv) the initial camera-wise GS parameter to compensate for cross-camera variation between the first camera and the second camera;
wherein the adjusting the initial camera-wise GS parameter is executed in accordance with:
SH
d
′
=
β
d
SH
+
γ
d
,
SH
r
′
=
β
r
SH
+
γ
r
,
wherein β is a scaling factor, γ is a bias factor, SH is the initial camera-wise GS SH parameter, SH d ′ is a 0th band of an adjusted camera-wise GS SH parameter, SH r ′ is a higher band of an adjusted camera-wise GS SH parameter;
generating, by the adjusted 3DGS model, a 3DGS output based on the multi-camera image input;
rendering a two-dimensional (2D) image of the 3D object using the 3DGS output.
6 . The method of claim 1 , wherein the method further comprises:
using a Multi-layer Perceptron (MLP) model to optimize the rotation matrix and the translation matrix.
7 . The method of claim 5 , wherein the method further comprises:
using a Multi-layer Perceptron (MLP) model to optimize the scaling factor and the bias factor.
8 . The method of claim 1 , wherein the rendering comprises employing a differentiable renderer to project the 3DGS output onto a 2D plane.
9 . The method of claim 1 , wherein the method further comprises:
generating a camera memory bank using the multi-camera image data; and
generating a global camera index using the camera memory bank.
10 . A processor for rendering a three-dimensional (3D) object, the processing being configured to:
acquire multi-camera image input including first image frames of the 3D object generated by a first camera and second image frames of the 3D object generated by a second camera, the first camera being different from the second camera;
acquire an initial 3D Gaussian Splatting (3DGS) model having a plurality of initial parameters, the plurality of initial parameters including an initial frame-wise Gaussian Splatting (GS) parameter and an initial camera-wise GS parameter;
generate an adjusted 3DGS model by adjusting, based on the multi-camera image input, at least one of:
(v) the initial frame-wise GS parameter to compensate for misalignment between the first image frames and the second image frames; and
(vi) the initial camera-wise GS parameter to compensate for cross-camera variation between the first camera and the second camera;
wherein to adjust the initial frame-wise GS parameter is executed by the processor in accordance with:
μ
′
=
M
R
μ
+
M
T
wherein μ is an initial frame-wise GS position parameter, μ′ is an adjusted frame-wise GS position parameter, M R is a rotation matrix, and M T is a translation matrix;
generate, using the adjusted 3DGS model, a 3DGS output based on the multi-camera image input; and
render a two-dimensional (2D) image of the 3D object using the 3DGS output.
11 . The processor of claim 10 , wherein to generate the adjusted 3DGS model comprises the processor configured to adjust both the initial frame-wise GS parameter and the initial camera-wise GS parameter.
12 . The processor of claim 10 , wherein the initial frame-wise GS parameter is an initial GS position parameter, and the initial camera-wise GS parameter is an initial GS Spherical Harmonics (SH) parameter.
13 . The processor of claim 10 , wherein the plurality of initial parameters further comprises a GS rotation parameter, a GS scale parameter, and a GS opacity parameter.
14 . The processor of claim 10 , wherein to adjust the initial camera-wise GS parameter is executed in accordance with:
SH
d
′
=
β
d
SH
+
γ
d
,
SH
r
′
=
β
r
SH
+
γ
r
,
wherein β is a scaling factor, γ is a bias factor, SH is the initial camera-wise GS SH parameter, SH d ′ is a 0th band of an adjusted camera-wise GS SH parameter, SH r ′ is a higher band of an adjusted camera-wise GS SH parameter.
15 . The processor of claim 10 , wherein the processor is further configured to:
use a Multi-Layer Perceptron (MLP) model to optimize the rotation matrix and the translation matrix.
16 . The processor of claim 14 , wherein the processor is further configured to:
use a Multi-Layer Perceptron (MLP) model to optimize the scaling factor and the bias factor.
17 . The processor of claim 10 , wherein to render comprises the processor configured to employ a differentiable renderer to project the 3DGS output onto a 2D plane.
18 . The processor of claim 10 , wherein the processor is further configured to:
generate a camera memory bank using the multi-camera image data; and
generate a global camera index using the camera memory bank.