Dynamically calibrating a depth sensor

1. A method comprising:

simultaneously capturing within a current scene:

a red-green-blue (RGB) image by a RGB sensor of a device; and

an infrared (IR) image by an IR sensor of the device;

concurrently capturing, via the IR sensor, an uncalibrated depth image within the current scene;

calculating calibration data from the RGB image and the IR image;

applying the calibration data to the uncalibrated depth image to generate a corrected depth image;

combining the RGB image and the corrected depth image to generate a RGB depth image; and

presenting the RGB depth image on a display of the device.

2. The method of claim 1 , wherein capturing the uncalibrated depth image further comprises:

projecting, by a structured light emitter of the device, a structured light pattern within the current scene;

wherein the uncalibrated depth image is captured while the current scene is illuminated with the structured light pattern.

3. The method of claim 1 , wherein applying the calibration data to the uncalibrated depth image further comprises rectifying the uncalibrated depth image based on the calibration data to generate the corrected depth image.

4. The method of claim 1 , wherein calculating the calibration data further comprises:

analyzing the RGB image to identify a first set of two-dimensional points;

analyzing the IR image to identify a second set of two-dimensional points that correspond to the first set of two-dimensional points;

determining, for each corresponding two-dimensional point, a two-dimensional difference in location between the RGB image and the IR image; and

calculating the calibration data based on the two-dimensional differences of each corresponding two-dimensional point between the RGB image and the IR image.

5. The method of claim 4 , wherein calculating the calibration data further comprises:

analyzing the uncalibrated depth image to identify a depth within the current scene for each point of the second set of two-dimensional points; and

performing at least one of a linear regression and a nonlinear regression using the first set of two-dimensional points, the second set of two-dimensional points, and the depth of the second set of two-dimensional points.

6. The method of claim 5 , further comprising:

determining, from a result of the at least one of the linear regression and the nonlinear regression, a set of camera intrinsics and a set of camera extrinsics;

wherein applying the calibration data to the uncalibrated depth image further comprises transforming the uncalibrated depth image based on the set of camera intrinsics and the set of camera extrinsics.

7. The method of claim 5 , further comprising:

determining lens distortion parameters from a result of the at least one of the linear regression and the nonlinear regression; and

wherein applying the calibration data to the uncalibrated depth image further comprises transforming the uncalibrated depth image based on the lens distortion parameters.

8. A device comprising:

a memory;

a red-green-blue (RGB) sensor that captures a RGB image within a current scene;

an infrared (IR) sensor that simultaneously captures an IR image within the current scene and that concurrently captures an uncalibrated depth image within the current scene;

at least one processor that:

calculates calibration data from the RGB image and the IR image;

applies the calibration data to the uncalibrated depth image to generate a corrected depth image; and

combines the RGB image and the corrected depth image to generate a RGB depth image; and

at least one display that presents the RGB depth image.

9. The device of claim 8 , further comprising:

a structured light emitter that projects a structured light pattern within the current scene;

wherein in capturing the uncalibrated depth image, the IR sensor captures the uncalibrated depth image while the current scene is illuminated with the structured light pattern.

10. The device of claim 8 , wherein in applying the calibration data to the uncalibrated depth image, the at least one processor rectifies the uncalibrated depth image based on the calibration data to generate the corrected depth image.

11. The device of claim 8 , wherein in calculating the calibration data, the at least one processor:

analyzes the RGB image to identify a first set of two-dimensional points;

analyzes the IR image to identify a second set of two-dimensional points that correspond to the first set of two-dimensional points;

determines, for each corresponding two-dimensional point, a two-dimensional difference in location between the RGB image and the IR image; and

calculates the calibration data based on the two-dimensional differences of each corresponding two-dimensional point between the RGB image and the IR image.

12. The device of claim 11 , wherein in calculating the calibration data, the at least one processor:

analyzes the uncalibrated depth image to identify a depth within the current scene for each point of the second set of two-dimensional points; and

performs at least one of a linear regression and a nonlinear regression using the first set of two-dimensional points, the second set of two-dimensional points, and the depth of the second set of two-dimensional points.

13. The device of claim 12 , wherein:

the at least one processor determines, from a result of the at least one of the linear regression and the nonlinear regression, a set of camera intrinsics and a set of camera extrinsics; and

in applying the calibration data to the uncalibrated depth image, the at least one processor transforms the uncalibrated depth image based on the set of camera intrinsics and the set of camera extrinsics.

14. The device of claim 12 , wherein:

the at least one processor determines lens distortion parameters from a result of the at least one of the linear regression and the nonlinear regression; and

in applying the calibration data to the uncalibrated depth image, the at least one processor transforms the uncalibrated depth image based on the lens distortion parameters.

15. A computer program product comprising:

a non-transitory computer readable storage device; and

program code on the computer readable storage device that, when executed by a processor associated with a device, enables the device to provide the functionality of:

simultaneously capturing within a current scene:

a red-green-blue (RGB) image by a RGB sensor of a device; and

an infrared (IR) image by an IR sensor of the device;

concurrently capturing, via the IR sensor, an uncalibrated depth image within the current scene;

calculating calibration data from the RGB image and the IR image;

applying the calibration data to the uncalibrated depth image to generate a corrected depth image;

combining the RGB image and the corrected depth image to generate a RGB depth image; and

presenting the RGB depth image on a display of the device.

16. The computer program product of claim 15 , the program code for calculating the calibration data further comprising code for:

analyzing the RGB image to identify a first set of two-dimensional points;

analyzing the IR image to identify a second set of two-dimensional points that correspond to the first set of two-dimensional points;

determining, for each corresponding two-dimensional point, a two-dimensional difference in location between the RGB image and the IR image; and

calculating the calibration data based on the two-dimensional differences of each corresponding two-dimensional point between the RGB image and the IR image.

17. The computer program product of claim 16 , the program code for calculating the calibration data further comprising code for:

analyzing the uncalibrated depth image to identify a depth within the current scene for each point of the second set of two-dimensional points; and

performing at least one of a linear regression and a nonlinear regression using the first set of two-dimensional points, the second set of two-dimensional points, and the depth of the second set of two-dimensional points.

18. The computer program product of claim 17 , the program code further comprising code for:

determining, from a result of the at least one of the linear regression and the nonlinear regression, a set of camera intrinsics and a set of camera extrinsics;

wherein applying the calibration data to the uncalibrated depth image further comprises transforming the uncalibrated depth image based on the set of camera intrinsics and the set of camera extrinsics.

19. The computer program product of claim 17 , the program code further comprising code for:

determining lens distortion parameters from a result of the at least one of the linear regression and the nonlinear regression; and

wherein applying the calibration data to the uncalibrated depth image further comprises transforming the uncalibrated depth image based on the lens distortion parameters.

20. The computer program product of claim 15 , wherein:

the program code for capturing the uncalibrated depth image further comprises code for projecting, by a structured light emitter of the device, a structured light pattern within the current scene, wherein the uncalibrated depth image is captured while the current scene is illuminated with the structured light pattern; and

the program code for applying the calibration data to the uncalibrated depth image further comprise code for rectifying the uncalibrated depth image based on the calibration data to generate the corrected depth image.