Automatic scene calibration
Described herein is a method of calibrating a three-dimensional imaging system. During calibration, a position and an orientation of the three-dimensional imaging system is determined with respect to a first parameter comprising a real world vertical direction (V w ) and to a second parameter comprising an origin of a three-dimensional scene captured by the imaging system. The first and second parameters are used to derive a calibration matrix (M C2w ) which is used to convert measurements from a virtual coordinate system (M c ) of the three-dimensional imaging system into a real coordinate system (M w ) related to the real world. The calibration matrix (M C2w ) is used to rectify measurements prior to signal processing. An inverse calibration matrix (M w2c ) is also determined. Continuous monitoring and adjustment of the setup of the three-dimensional imaging system is carried out and the calibration matrix (M c2w ) and its inverse (M w2c ) are adjusted accordingly.
1. A method of calibrating a three-dimensional time-of-flight imaging system in a three-dimensional environment, the method comprising the steps of:
a) determining a reference orthogonal virtual three-dimensional coordinate system for the three-dimensional imaging system, the reference orthogonal virtual three-dimensional coordinate system having horizontal, vertical and depth axes in which the horizontal axis and the vertical axis are respectively aligned with the horizontal and vertical axes of a sensor in the three-dimensional imaging system with the depth axis being orthonormal to the plane of the sensor defined by its horizontal and vertical axes;
b) obtaining a vertical direction of the real world in the virtual coordinate system;
c) determining with respect to the reference coordinate system a real world three-dimensional orthonormal coordinate system having horizontal, vertical and depth axes in which the vertical axis is rotated to align it with respect to the vertical direction;
d) determining a point in the scene as a new origin for the real world three-dimensional orthonormal coordinate system;
e) deriving a translation vector from the origin of the virtual three-dimensional coordinate system to the point defined as the new origin of the scene;
f) deriving a rotation matrix for transforming the virtual three-dimensional coordinate system into the real world three-dimensional coordinate system;
g) deriving a calibration matrix (M c2w ) for the three-dimensional imaging system as the rotation matrix translated by the translation vector;
h) aligning a plane defined by the vertical and depth axes of the real world three-dimensional coordinate system to be coplanar with a plane defined by the virtual vertical axis and the virtual depth axis of the virtual three-dimensional coordinate system; and
i) automatically refining the calibration matrix if changes are detected in at least one of:
the position and the orientation of the three-dimensional imaging system with respect to the three-dimensional environment.
2. A method according to claim 1 , wherein step g) further comprises deriving an inverse calibration matrix from the calibration matrix for transforming the real world three-dimensional coordinate system into the virtual three-dimensional coordinate system.
3. A method according to claim 1 , wherein step b) comprises deriving the vertical direction of the real world as the opposite vector of a gravity vector determined using a measurement device, the measurement device comprising at least one inertial measurement unit.
4. A method according to claim 1 , wherein step
b) comprises deriving the vertical direction of the real world from a normal to a plane in the scene, the plane in the scene being determined by the steps of:
i) capturing the scene with the three-dimensional imaging system in a first orientation;
ii) determining a plurality of planes within the scene using a plane fitting algorithm; and
iii) determining a reference plane within the scene as being the ground.
5. A method according to claim 4 , wherein step iii) comprises determining the reference plane as one best satisfying:
a statistical mode;
the largest surface; and
a minimum surface area.
6. A method according to claim 4 , wherein step iii) comprises determining the reference plane as a combination of: a statistical mode of the principal component analysis of the scene; the largest surface; and a minimum surface area.
7. A method according to claim 1 , wherein step b) comprises deriving the vertical direction of the real world from a specific user stance within the scene, the vertical direction being the direction that is aligned with the vertical direction of a user standing in a predefined calibration posture.
8. A method according to claim 7 , further comprising the steps of deriving a horizontal axis and a vertical axis from the predefined calibration posture, and aligning the real world three-dimensional coordinate system with the derived horizontal and vertical axes.
9. A method according to claim 1 , wherein step
b) comprises deriving the vertical direction of the real world from edges detected within the scene.
10. A method according to claim 1 , wherein step b) comprises refining the vertical direction by combining two or more of the steps of:
deriving the vertical direction of the real world as the opposite vector of a gravity vector determined using a measurement device;
deriving the vertical direction of the real world from a normal to a plane in the scene;
deriving the vertical direction of the real world from a specific user stance within the scene, the vertical direction being the direction that is aligned with the vertical direction of a user standing in a predefined calibration posture; and
deriving the vertical direction of the real world from edges detected within the scene.
11. A method according to claim 1 , wherein step d) comprises defining the point for the new origin using one of:
a predefined point in space;
the lowest location of the points defining a user;
a point on a detected plane; and
the location of a predetermined object within the scene.
12. A method according to claim 1 , further comprising the step of aligning at least the vertical axis of the virtual coordinate system with the Y-Z plane of the real world coordinate system by controlling a motorised system supporting the three-dimensional imaging system.
13. A depth sensing imaging system having inertial measurement unit, the depth sensing imaging system operating in accordance with a method according to claim 1 comprising the steps of:
a) determining a reference orthogonal virtual three-dimensional coordinate system for the three-dimensional imaging system, the reference orthogonal virtual three-dimensional coordinate system having horizontal, vertical and depth axes in which the horizontal axis and the vertical axis are respectively aligned with the horizontal and vertical axes of a sensor in the three-dimensional imaging system with the depth axis being orthonormal to the plane of the sensor defined by its horizontal and vertical axes;
b) obtaining a vertical direction of the real world in the virtual coordinate system:
c) determining with respect to the reference coordinate system a real world three-dimensional orthonormal coordinate system having horizontal, vertical and depth axes in which the vertical axis is rotated to align it with respect to the vertical direction;
d) determining a point in the scene as a new origin for the real world three-dimensional orthonormal coordinate system;
e) deriving a translation vector from the origin of the virtual three-dimensional coordinate system to the point defined as the new origin of the scene
f) deriving a rotation matrix for transforming the virtual three-dimensional coordinate system into the real world three-dimensional coordinate system;
g) deriving a calibration matrix for the three-dimensional imaging system as the rotation matrix translated by the translation vector;
h) aligning a plane defined by the vertical and depth axes of the real world three-dimensional coordinate system to be coplanar with a plane defined by the virtual vertical axis and the virtual depth axis of the virtual three-dimensional coordinate system; and
i) automatically refining the calibration matrix if changes are detected in at least one of: the position and the orientation of the three-dimensional imaging system with respect to the three-dimensional environment.
14. A depth sensing imaging system having motorised means for adjusting at least one of: position and orientation of the depth sensing imaging system in accordance with operation of a method comprising the steps of:
a) determining a reference orthogonal virtual three-dimensional coordinate system for the three-dimensional imaging system, the reference orthogonal virtual three-dimensional coordinate system having horizontal, vertical and depth axes in which the horizontal axis and the vertical axis are respectively aligned with the horizontal and vertical axes of a sensor in the three-dimensional imaging system with the depth axis being orthonormal to the plane of the sensor defined by its horizontal and vertical axes;
b) obtaining a vertical direction of the real world in the virtual coordinate system:
c) determining with respect to the reference coordinate system a real world three-dimensional orthonormal coordinate system having horizontal, vertical and depth axes in which the vertical axis is rotated to align it with respect to the vertical direction;
d) determining a point in the scene as a new origin for the real world three-dimensional orthonormal coordinate system;
e) deriving a translation vector from the origin of the virtual three-dimensional coordinate system to the point defined as the new origin of the scene;
f) deriving a rotation matrix for transforming the virtual three-dimensional coordinate system into the real world three-dimensional coordinate system;
g) deriving a calibration matrix for the three-dimensional imaging system as the rotation matrix translated by the translation vector;
h) aligning a plane defined by the vertical and depth axes of the real world three-dimensional coordinate system to be coplanar with a plane defined by the virtual vertical axis and the virtual depth axis of the virtual three-dimensional coordinate system; and
i) automatically refining the calibration matrix if changes are detected in at least one of: the position and the orientation of the three-dimensional imaging system with respect to the three-dimensional environment.
15. A three-dimensional time of flight imaging system having calibration means operating in accordance with a method comprising the steps of:
a) determining a reference orthogonal virtual three-dimensional coordinate system for the three-dimensional imaging system, the reference orthogonal virtual three-dimensional coordinate system having horizontal, vertical and depth axes in which the horizontal axis and the vertical axis are respectively aligned with the horizontal and vertical axes of a sensor in the three-dimensional imaging system with the depth axis being orthonormal to the plane of the sensor defined by its horizontal and vertical axes;
b) obtaining a vertical direction of the real world in the virtual coordinate system:
c) determining with respect to the reference coordinate system a real world three-dimensional orthonormal coordinate system having horizontal, vertical and depth axes in which the vertical axis is rotated to align it with respect to the vertical direction;
d) determining a point in the scene as a new origin for the real world three-dimensional orthonormal coordinate system;
e) deriving a translation vector from the origin of the virtual three-dimensional coordinate system to the point defined as the new origin of the scene;
f) deriving a rotation matrix for transforming the virtual three-dimensional coordinate system into the real world three-dimensional coordinate system;
g) deriving a calibration matrix for the three-dimensional imaging system as the rotation matrix translated by the translation vector;
h) aligning a plane defined by the vertical and depth axes of the real world three-dimensional coordinate system to be coplanar with a plane defined by the virtual vertical axis and the virtual depth axis of the virtual three-dimensional coordinate system; and
i) automatically refining the calibration matrix if changes are detected in at least one of: the position and the orientation of the three-dimensional imaging system with respect to the three-dimensional environment.