Systems and methods for accelerated radiance field training via selective primitive reuse
A radiance field generation system (RFGS) and associated methods perform accelerated training of variant three-dimensional (3D) assets through selective primitive reuse. The RFGS receives a base 3D asset and a variant 3D asset and generates a first radiance field representation of the base asset with splat primitives. Matching regions between the 3D assets are identified by comparing images of the 3D assets rendered from corresponding viewpoints. Splat primitives in the first radiance field representation that represent the matching regions are locked, marking them as immutable and excluding them from parameter adjustments during subsequent optimization. The RFGS generates a second radiance field representation of the variant 3D asset by reusing the locked splat primitives without modification while iteratively optimizing a second subset of splat primitives representing differing regions. This spatially constrained optimization significantly reduces training iterations and computational overhead by focusing optimization solely on detected areas of difference.
1 . A method comprising:
receiving a base three-dimensional (3D) asset and a variant 3D asset;
generating a first radiance field representation of the base 3D asset, the first radiance field representation comprising a plurality of radiance field primitives;
identifying matching regions between the base 3D asset and the variant 3D asset by comparing images capturing the base 3D asset and the variant 3D asset from corresponding viewpoints;
locking a first subset of primitives from the plurality of radiance field primitives that represent the matching regions in the first radiance field representation; and
generating a second radiance field representation of the variant 3D asset by iteratively optimizing a second subset of primitives corresponding to unlocked primitives that represent differing regions in the second radiance field representation while reusing locked primitives from the first subset of primitives without modification.
2 . The method of claim 1 further comprising:
initializing the second radiance field representation using the first subset of primitives.
3 . The method of claim 1 further comprising:
determining a common set of camera poses relative to each of the base 3D asset and the variant 3D asset; and
generating the images of the base 3D asset and the variant 3D asset from the common set of camera poses.
4 . The method of claim 3 further comprising:
calculating a similarity measurement between paired images from a same camera pose using at least one of a pixel-by-pixel difference analysis, a structural similarity comparison, or a perceptual similarity model.
5 . The method of claim 1 further comprising:
mapping image differences identified during said comparing back to radiance field primitives of the first radiance field representation.
6 . The method of claim 1 , wherein the base 3D asset and the variant 3D asset are originally provided in a first 3D format comprising one of a mesh model or a point cloud, and wherein the plurality of radiance field primitives represents the base 3D asset in a second 3D format comprising a splat representation.
7 . The method of claim 1 , wherein identifying the matching regions comprises:
rendering the base 3D asset and the variant 3D asset from the corresponding viewpoints to produce the images capturing the base 3D asset and the variant 3D asset.
8 . The method of claim 1 , wherein generating the first radiance field representation comprises:
generating a Gaussian splatting representation, and wherein each radiance field primitive comprises a splat defined by a position, a scale, an orientation, an opacity, and one or more appearance parameters.
9 . The method of claim 1 , wherein locking the first subset of primitives comprises:
setting a locking indicator for each primitive in the first subset of primitives that prevents parameter updates for the primitive during said iterative optimizing.
10 . The method of claim 1 , wherein iteratively optimizing the second subset of primitives comprises:
computing loss based on renderings of the second radiance field representation; and
adjusting the second subset of primitives based on the loss without adjusting the first subset of primitives.
11 . The method of claim 1 further comprising:
performing parameter-specific loss analysis for at least a third subset of primitives from the plurality of radiance field primitives associated with a differing region; and
determining that a difference is attributable to a first parameter of the third subset of primitives.
12 . The method of claim 11 further comprising:
locking at least a second parameter of the third subset of primitives prior to performing said iterative optimizing; and
optimizing the first parameter of the third subset of primitives during said iterative optimizing without adjusting the second parameter.
13 . A radiance field generation system comprising:
one or more hardware processors configured to:
receive a base three-dimensional (3D) asset and a variant 3D asset;
generate a first radiance field representation of the base 3D asset, the first radiance field representation comprising a plurality of radiance field primitives;
identify matching regions between the base 3D asset and the variant 3D asset by comparing images capturing the base 3D asset and the variant 3D asset from corresponding viewpoints;
lock a first subset of primitives from the plurality of radiance field primitives that represent the matching regions in the first radiance field representation; and
generate a second radiance field representation of the variant 3D asset by iteratively optimizing a second subset of primitives corresponding to unlocked primitives that represent differing regions in the second radiance field representation while reusing locked primitives from the first subset of primitives without modification.
14 . The radiance field generation system of claim 13 , wherein the one or more hardware processors are further configured to:
initialize the second radiance field representation using the first subset of primitives.
15 . The radiance field generation system of claim 13 , wherein the one or more hardware processors are further configured to:
determine a common set of camera poses relative to each of the base 3D asset and the variant 3D asset; and
generate the images of the base 3D asset and the variant 3D asset from the common set of camera poses.
16 . The radiance field generation system of claim 15 , wherein the one or more hardware processors are further configured to:
calculate a similarity measurement between paired images from a same camera pose using at least one of a pixel-by-pixel difference analysis, a structural similarity comparison, or a perceptual similarity model.
17 . The radiance field generation system of claim 13 , wherein the one or more hardware processors are further configured to:
map image differences identified during said comparing back to radiance field primitives of the first radiance field representation.
18 . The radiance field generation system of claim 13 , wherein the base 3D asset and the variant 3D asset are originally provided in a first 3D format comprising one of a mesh model or a point cloud, and wherein the plurality of radiance field primitives represents the base 3D asset in a second 3D format comprising a splat representation.
19 . The radiance field generation system of claim 13 , wherein identifying the matching regions comprises:
rendering the base 3D asset and the variant 3D asset from the corresponding viewpoints to produce the images capturing the base 3D asset and the variant 3D asset.
20 . A non-transitory computer-readable medium storing instructions that, when executed by one or more processors of a radiance field generation system, cause the radiance field generation system to perform operations comprising:
receiving a base three-dimensional (3D) asset and a variant 3D asset;
generating a first radiance field representation of the base 3D asset, the first radiance field representation comprising a plurality of radiance field primitives;
identifying matching regions between the base 3D asset and the variant 3D asset by comparing images capturing the base 3D asset and the variant 3D asset from corresponding viewpoints;
locking a first subset of primitives from the plurality of radiance field primitives that represent the matching regions in the first radiance field representation; and
generating a second radiance field representation of the variant 3D asset by iteratively optimizing a second subset of primitives corresponding to unlocked primitives that represent differing regions in the second radiance field representation while reusing locked primitives from the first subset of primitives without modification.