Three-dimensional (3D) reconstructions of dynamic scenes using a reconfigurable hybrid imaging system

1. A computer-implemented method for a three-dimensional (3D) reconstruction of a dynamic scene, comprising:

receiving, by one or more hardware processors, a plurality of color image sequences from a plurality of color imaging sensors, and at least one depth image sequence from at least one depth imaging sensor, wherein a color imaging sensor quantity is larger than a depth imaging sensor quantity;

generating, by the one or more hardware processors, a plurality of calibrated color image sequences and at least one calibrated depth image sequence based on the plurality of color imaging sequences and the at least one depth image sequence;

constructing, by the one or more hardware processors, a plurality of initial 3D patches using the plurality of calibrated color image sequences and the at least one calibrated depth image sequence; and

generating, by the one or more hardware processors, a 3D patch cloud by expanding the plurality of initial 3D patches.

2. The method of claim 1 , wherein each of the plurality of color imaging sequences and the at least one depth image sequence includes images that capture the dynamic scene over a period of time.

3. The method of claim 1 , wherein generating the plurality of calibrated color image sequences and the at least one calibrated depth image sequence comprises:

determining intrinsic parameters and extrinsic parameters for each of the plurality of color imaging sensors and the at least one depth imaging sensor; and

calibrating the plurality of color image sequences and the at least one depth image sequence based on the intrinsic parameters and the extrinsic parameters to generate the plurality of calibrated color image sequences and the at least one calibrated depth image sequence.

4. The method of claim 3 , wherein determining the extrinsic parameters for each of the plurality of color imaging sensors and the at least one depth imaging sensor includes geometrically calibrating the plurality of color imaging sensors and the at least one depth imaging sensor with a reference imaging sensor, wherein the reference imaging sensor is one of the plurality of color imaging sensors and the at least one depth imaging sensor.

5. The method of claim 1 , wherein constructing the plurality of initial 3D patches comprises:

determining a set of salient points from the plurality of calibrated color image sequences and the at least one calibrated depth image sequence; and

for each salient point:

determining a set of imaging points corresponding to the salient point in the plurality of calibrated color image sequences and the at least one calibrated depth image sequence; and

computing an initial 3D patch corresponding to the salient point based on the set of imaging points using a constrained triangulation.

6. The method of claim 5 , wherein the constrained triangulation includes minimizing a cost function comprising at least one of a photo-consistency metric for the plurality of calibrated color image sequences, or a depth-consistency metric for the at least one calibrated depth image sequence.

7. The method of claim 1 , wherein generating the 3D patch cloud by expanding the plurality of initial 3D patches comprises:

spatially dividing images of the plurality of calibrated color image sequences and the at least one calibrated depth image sequence into cells; and

expanding the plurality of initial 3D patches using a regularized optimization to generate the 3D patch cloud that covers the cells.

8. The method of claim 1 , further comprising computing a 3D mesh surface using the 3D patch cloud, with the 3D mesh comprising the 3D reconstruction of the dynamic scene.

9. A computer-implemented system, comprising:

a memory storage comprising instructions for a three-dimensional (3D) reconstruction of a dynamic scene; and

one or more hardware processors in communication with the memory storage, wherein the one or more hardware processors execute the instructions to:

receive a plurality of color image sequences from a plurality of color imaging sensors, and at least one depth image sequence from at least one depth imaging sensor, wherein a color imaging sensor quantity is larger than a depth imaging sensor quantity;

generate a plurality of calibrated color image sequences and at least one calibrated depth image sequence based on the plurality of color imaging sequences and the at least one depth image sequence;

construct a plurality of initial 3D patches using the plurality of calibrated color image sequences and the at least one calibrated depth image sequence; and

generate a 3D patch cloud by expanding the plurality of initial 3D patches.

10. The computer-implemented system of claim 9 , wherein each of the plurality of color imaging sequences and the at least one depth image sequence includes images that capture the dynamic scene over a period of time.

11. The computer-implemented system of claim 9 , wherein generating the plurality of calibrated color image sequences and the at least one calibrated depth image sequence comprises:

determining intrinsic parameters and extrinsic parameters for each of the plurality of color imaging sensors and the at least one depth imaging sensor; and

calibrating the plurality of color image sequences and the at least one depth image sequence based on the intrinsic parameters and the extrinsic parameters to generate the plurality of calibrated color image sequences and the at least one calibrated depth image sequence.

12. The computer-implemented system of claim 11 , wherein determining the extrinsic parameters for each of the plurality of color imaging sensors and the at least one depth imaging sensor includes geometrically calibrating the plurality of color imaging sensors and the at least one depth imaging sensor with a reference imaging sensor, wherein the reference imaging sensor is one of the plurality of color imaging sensors and the at least one depth imaging sensor.

13. The computer-implemented system of claim 9 , wherein constructing the plurality of initial 3D patches comprises:

determining a set of salient points from the plurality of calibrated color image sequences and the at least one calibrated depth image sequence; and

for each salient point:

determining a set of imaging points corresponding to the salient point in the plurality of calibrated color image sequences and the at least one calibrated depth image sequence; and

computing an initial 3D patch corresponding to the salient point based on the set of imaging points using a constrained triangulation.

14. The computer-implemented system of claim 13 , wherein the constrained triangulation includes minimizing a cost function comprising at least one of a photo-consistency metric for the plurality of calibrated color image sequences, or a depth-consistency metric for the at least one calibrated depth image sequence.

15. The computer-implemented system of claim 9 , wherein generating the 3D patch cloud by expanding the plurality of initial 3D patches comprises:

spatially dividing images of the plurality of calibrated color image sequences and the at least one calibrated depth image sequence into cells; and

expanding the plurality of initial 3D patches using a regularized optimization to generate the 3D patch cloud that covers the cells.

16. The computer-implemented system of claim 9 , wherein the one or more hardware processors further execute the instructions to compute a 3D mesh surface using the 3D patch cloud, with the 3D mesh comprising the 3D reconstruction of the dynamic scene.

17. A non-transitory computer-readable medium storing computer instructions for a three-dimensional (3D) reconstruction of a dynamic scene which, when executed by one or more hardware processors, cause the one or more hardware processors to perform operations comprising:

receiving a plurality of color image sequences from a plurality of color imaging sensors, and at least one depth image sequence from at least one depth imaging sensor, wherein a color imaging sensor quantity is larger than a depth imaging sensor quantity;

generating a plurality of calibrated color image sequences and at least one calibrated depth image sequence based on the plurality of color imaging sequences and the at least one depth image sequence;

constructing a plurality of initial 3D patches using the plurality of calibrated color image sequences and the at least one calibrated depth image sequence; and

generating a 3D patch cloud by expanding the plurality of initial 3D patches.

18. The non-transitory computer-readable medium of claim 17 , wherein generating the plurality of calibrated color image sequences and the at least one calibrated depth image sequence comprises:

determining intrinsic parameters and extrinsic parameters for each of the plurality of color imaging sensors and the at least one depth imaging sensor; and

calibrating the plurality of color image sequences and the at least one depth image sequence based on the intrinsic parameters and the extrinsic parameters to generate the plurality of calibrated color image sequences and the at least one calibrated depth image sequence.

19. The non-transitory computer-readable medium of claim 17 , wherein constructing the plurality of initial 3D patches comprises:

determining a set of salient points from the plurality of calibrated color image sequences and the at least one calibrated depth image sequence; and

for each salient point:

determining a set of imaging points corresponding to the salient point in the plurality of calibrated color image sequences and the at least one calibrated depth image sequence; and

computing an initial 3D patch corresponding to the salient point based on the set of imaging points using a constrained triangulation.

20. The non-transitory computer-readable medium of claim 17 , wherein generating the 3D patch cloud by expanding the plurality of initial 3D patches comprises:

spatially dividing images of the plurality of calibrated color image sequences and the at least one calibrated depth image sequence into cells; and

expanding the plurality of initial 3D patches using a regularized optimization to generate the 3D patch cloud that covers the cells.