Reconstruction of dynamic scenes based on differences between collected view and synthesized view
A system for generating a 4D representation of a scene in motion given a sinogram collected from the scene while in motion. The system generates, based on scene parameters, an initial 3D representation of the scene indicating linear attenuation coefficients (LACs) of voxels of the scene. The system generates, based on motion parameters, a 4D motion field indicating motion of the scene. The system generates, based on the initial 3D representation and the 4D motion field, a 4D representation of the scene that is a sequence of 3D representations having LACs. The system generates a synthesized sinogram of the scene from the generated 4D representation. The system adjusts the scene parameters and the motion parameters based on differences between the collected sinogram and the synthesized sinogram. The processing is repeated until the differences satisfy a termination criterion.
1. A method performed by one or more computing systems for generating a four-dimensional (4D) representation of a three-dimensional (3D) scene that has motion, the 4D representation representing the motion of the scene, the method comprising:
accessing a collected view of the scene, the collected view representing attenuation of an electromagnetic signal transmitted through the scene at various angles; and
for each of a plurality of iterations,
applying a 3D representation generator to generate an initial 3D representation of the scene for the iteration, the 3D representation generator having scene weights, a 3D representation having voxels that each represent a portion of the scene;
applying a 4D motion generator to generate a 4D motion field as a sequence of 3D motion fields for the iteration, a 3D motion field indicating location of voxels of the initial 3D representation, the 4D motion generator having motion weights;
applying a 4D representation generator to generate a 4D representation having a sequence of 3D representations based on the initial 3D representation and the 4D motion field;
generating a synthesized view of the scene from the generated 4D representation; and
adjusting the scene weights and the motion weights based on differences between the collected view and the synthesized view.
2. The method of claim 1 wherein the electromagnetic signals are transmitted by an electromagnetic transmitter and collected by an electromagnetic collector.
3. The method of claim 1 wherein the 3D representation generator is a neural network.
4. The method of claim 1 wherein the 3D representation generator comprises a random sampling layer and a neural network, the random sampling layer having an input and generating an output based on a random sample of a distribution and a sharpness factor, the generated output being input to the neural network which outputs an initial 3D representation.
5. The method of claim 1 wherein the 4D motion generator generates the 4D motion field based on a polynomial equation with a motion weight for each coordinate of each voxel for each order of the polynomial equation.
6. The method of claim 1 wherein the 4D representation generator generates attenuation coefficients using differentiable image sampling.
7. The method of claim 1 wherein the adjusting of the scene weights and the motion weights applies a gradient descent to minimize an objective function.
8. A method performed by one or more computing systems for generating a four-dimensional (4D) representation of a scene based on a collected view of the scene, the collected view being collected while the scene is in motion, the method comprising:
generating a 4D representation of the scene based on an initial three-dimensional (3D) representation and a 4D motion field, the initial 3D representation generated based on scene parameters, the 4D motion field generated based on motion parameters;
generating a synthesized view of the scene from the generated 4D representation; and
adjusting the scene parameters and the motion parameters based on differences between the collected view and the synthesized view.
9. The method of claim 8 further comprising
generating the initial three-dimensional (3D) representation, a 3D representation having voxels that each represent a portion of the scene; and
generating the 4D motion field that includes the motion field indicating motion of voxels of the initial 3D representation.
10. The method of claim 9 further comprising repeating the generating of the initial 3D representation, the 4D motion field, the 4D representation, and the synthesized view and the adjusting of the scene parameters and motion parameters until a termination criterion is satisfied.
11. The method of claim 8 wherein the scene includes a body part.
12. The method of claim 8 wherein the scene includes an object that is being manufactured.
13. The method of claim 8 wherein the scene relates to a geological event.
14. One or more computing systems for generating a four-dimensional (4D) representation of an object in motion based on a collected view of the object that is collected when the object is in motion, the one or more computing systems comprising:
one or more non-transitory computer-readable storage mediums for storing computer-executable instructions for controlling the one or more computing systems to, until a termination condition is satisfied:
generate an initial three-dimensional (3D) representation of the object based on object parameters, a 3D representation having voxels representing portions of the object;
generate a 4D motion field based on motion parameters, the 4D motion field indicating location over time of voxels of the initial 3D representation; and
generate a 4D representation of the object based on the initial 3D representation and the 4D motion fields, the 4D representation being a sequence of 3D representations that each represents a different time, the 4D representation comprising the 3D representations;
generate a synthesized view of the object based on the 4D representation; and
adjust the object parameters and the motion parameters based on differences between the collected view and the synthesized view;
one or more processors for executing the computer-executable instructions stored in the one or more computer-readable storage mediums.
15. The one or more computing systems of claim 14 wherein a linear attenuation coefficient is associated with each voxel.
16. The one or more computing systems of claim 15 wherein the instructions that generate the 4D representation generate linear attenuation coefficients for voxels.
17. One or more non-transitory computer-readable storage mediums storing instructions comprising:
a three-dimensional (3D) representation generator that generates an initial 3D representation of a scene based on scene parameters, a 3D representation having voxels with values;
a four-dimensional (4D) motion generator that generates a 4D motion field based on motion parameters, the 4D motion field indicating motion of voxels of the scene;
a 4D representation generator that generates a 4D representation of the scene based on the initial 3D representation and the 4D motion field, the 4D representation including a sequence of 3D representations having voxels with voxel values;
a synthesized view generator that generates a synthesized view of the scene based on the 4D representation;
a loss function that calculates a difference between a collected view of the scene and the synthesized view of the scene; and
an optimizer that adjusts the scene parameters and the motion parameters based on the difference.
18. The one or more non-transitory computer-readable storage mediums of claim 17 wherein the instructions further comprise instructions that determine whether a termination criterion is satisfied based on the difference.
19. The one or more non-transitory computer-readable storage mediums of claim 17 wherein the collected view and synthesized view are sinograms.
20. The one or more non-transitory computer-readable storage mediums of claim 17 wherein the 3D representation generator includes a multi-level perceptron.