Method and system for measuring coplanar point distances using an RGB-D camera
Systems, methods, and software are described which enable robust measurement of distances between 3D points, such as for pupillary distance (PD) of a wearer of corrective lenses that are known to be coplanar from a color-and-depth camera, e.g., a red-green-blue-depth (RGB-D) camera. Examples can provide a late fusion pipeline that first detects and tracks 2D landmarks in color imagery, projects those detections (detected 2D landmarks) into the depth frame, and discards measurements that appear noncoplanar and nonparallel. A user can be directed to change poses to precisely correct for nonparallel camera poses. Temporal filters can be used for the resulting distance measurements.
1 . A system for measuring coplanar point distances of facial features using an RGB-D camera, the system comprising:
an RGB-D camera having an RGB camera and a depth (D) camera, wherein the RGB camera includes an RGB sensor and is configured to produce output signals on an RGB channel corresponding to one or more RGB images, and wherein the depth camera includes a depth sensor and is configured to produce output signals on a depth channel corresponding to one or more depth images;
a memory including computer-executable instructions; and
a processor coupled to the memory and operative to execute the computer-executable instructions, the computer-executable instructions causing the processor to perform operations including:
(i) using a landmark detector to detect one or more pairs of coplanar points as symmetric landmark pairs in one or more color images of a user's face captured by the RGB-D camera, wherein each symmetric landmark pair corresponds to symmetrical facial features of the user, respectively;
(ii) fusing color and depth channels corresponding to the RGB camera and the depth camera of the RGB-D camera, respectively, by using corresponding camera projection matrices and extrinsic parameters to project the detected symmetric landmark pairs from the one or more color images into corresponding one or more depth images, wherein the camera projection matrices define camera lens and sensor intrinsic parameters, and wherein the extrinsic parameters describe a transformation between the color and depth sensors;
(iii) enforcing coplanarity and parallelity of the detected symmetric landmark pairs;
(iv) flattening each symmetric landmark pair by assigning both landmarks the lesser of the pair's two depth values to mitigate potentially noisy depth sensor measurements, thereby reducing asymmetric sensor bias and noise of the RGB-D camera;
(v) projecting each landmark into world coordinates using the intrinsic parameters of the depth sensor, wherein the intrinsic parameters include the optical center and focal length of the depth camera; and
(vi) calculating a point distance between the points of one symmetric landmark pair in the world coordinates of the depth sensor.
2 . The system of claim 1 , wherein the calculated point distance comprises a pupillary distance.
3 . The system of claim 1 , wherein the processor is further configured to use a coarse object pose estimator to constrain a rotation angle of the camera relative to the user.
4 . The system of claim 1 , wherein the one or more images comprise a plurality of frames of video from the RGB-D camera.
5 . The system of claim 1 , wherein fusing the color and depth channels further comprises synchronizing the color and depth channels.
6 . The system of claim 1 , wherein using the landmark detector comprises using a convolutional neural network.
7 . The system of claim 1 , wherein using the landmark detector comprises using a deep landmark detection network.
8 . The system of claim 1 , wherein using the landmark detector comprises edge detection.
9 . The system of claim 1 , wherein using the landmark detector comprises using an optical flow algorithm.
10 . The system of claim 1 , wherein using the landmark detector comprises using a facial landmark detection model.
11 . The system of claim 1 , wherein using the landmark detector comprises using SIFT or SURF as a feature-based methodology.
12 . The system of claim 1 , wherein the processor is configured to calculate the distance between the points using the Cartesian distance formula.
13 . The system of claim 1 , wherein the processor is further configured to apply a temporal filter to successive values of the calculated point distance between the points.
14 . The system of claim 1 , wherein the processor is further configured to incorporate a vergence adjustment into the calculated point distance, wherein the two points correspond to pupils of a user, and wherein the calculated distance comprises an interpupillary distance (IPD).
15 . A method of using an RGB-D camera for measuring coplanar point distances of facial features, wherein the RGB-D camera includes an RGB camera and a depth (D) camera, and wherein the RGB-D camera is connected to memory and a processor, the method comprising:
(i) using a landmark detector to detect one or more pairs of coplanar points as symmetric landmark pairs in one or more color images of a user's face captured by the RGB-D camera, wherein each symmetric landmark pair corresponds to symmetrical facial features of the user, respectively;
(ii) fusing color and depth channels corresponding to the RGB camera and the depth camera of the RGB-D camera, respectively, by using corresponding camera projection matrices and extrinsic parameters to project the detected symmetric landmark pairs from the one or more color images into corresponding one or more depth images, wherein the camera projection matrices define camera lens and sensor intrinsic parameters, and wherein the extrinsic parameters describe a transformation between the color and depth sensors;
(iii) enforcing coplanarity and parallelity of the detected symmetric landmark pairs;
(iv) flattening each symmetric landmark pair by assigning both landmarks the lesser of the pair's two depth values to mitigate potentially noisy depth sensor measurements, thereby reducing metric sensor bias and noise of the RGB-D camera;
(v) projecting each landmark into world coordinates using the intrinsic parameters of the depth sensor, wherein the intrinsic parameters include the optical center and focal length of the depth camera; and
(vi) calculating a point distance between the points of one symmetric landmark pair in the world coordinates of the depth sensor.
16 . The method of claim 15 , wherein the calculated point distance comprises a pupillary distance.
17 . The method of claim 15 , wherein the processor is further configured to use a coarse object pose estimator to constrain a rotation angle of the camera relative to the user.
18 . The method of claim 15 , wherein the one or more images comprise a plurality of frames of video from the RGB-D camera.
19 . The method of claim 15 , wherein fusing the color and depth channels further comprises synchronizing the color and depth channels.
20 . The method of claim 15 , wherein using the landmark detector comprises using a convolutional neural network.
21 . The method of claim 15 , wherein using the landmark detector comprises using a deep landmark detection network.
22 . The method of claim 15 , wherein using the landmark detector comprises edge detection.
23 . The method of claim 15 , wherein using the landmark detector comprises using an optical flow algorithm.
24 . The method of claim 15 , wherein using the landmark detector comprises using a facial landmark detection model.
25 . The method of claim 15 , wherein using the landmark detector comprises using SIFT or SURF as a feature-based methodology.
26 . The method of claim 15 , wherein the processor is configured to calculate the distance between the points using the Cartesian distance formula.
27 . The method of claim 15 , wherein the processor is further configured to apply a temporal filter to successive values of the calculated point distance between the points.
28 . The method of claim 15 , wherein the processor is further configured to incorporate a vergence adjustment into the calculated point distance, wherein the two points correspond to pupils of a user, and wherein the calculated distance comprises an interpupillary distance (IPD).
29 . A non-transitory computer readable storage medium including computer executable instructions for measuring coplanar point distances of facial features using an RGB-D camera, which when read by a processor cause the processor to perform operations including:
(i) using a landmark detector to detect one or more pairs of coplanar points as symmetric landmark pairs in one or more color images of a user's face captured by the RGB-D camera, wherein each symmetric landmark pair corresponds to symmetrical facial features of the user, respectively;
(ii) fusing color and depth channels corresponding to the RGB camera and the depth camera of the RGB-D camera, respectively, by using corresponding camera projection matrices and extrinsic parameters to project the detected symmetric landmark pairs from the one or more color images into corresponding one or more depth images, wherein the camera projection matrices define camera lens and sensor intrinsic parameters, and wherein the extrinsic parameters describe a transformation between the color and depth sensors;
(iii) enforcing coplanarity and parallelity of the detected symmetric landmark pairs;
(iv) flattening each symmetric landmark pair by assigning both landmarks the lesser of the pair's two depth values to mitigate potentially noisy depth sensor measurements, thereby reducing asymmetric sensor bias and noise of the RGB-D camera;
(v) projecting each landmark into world coordinates using the intrinsic parameters of the depth sensor, wherein the intrinsic parameters include the optical center and focal length of the depth camera; and
(vi) calculating a point distance between the points of one symmetric landmark pair in the world coordinates of the depth sensor.
30 . The non-transitory storage medium of claim 29 , wherein the calculated point distance comprises a pupillary distance.
31 . The non-transitory storage medium of claim 29 , wherein the processor is further configured to use a coarse object pose estimator to constrain a rotation angle of the camera relative to the user.
32 . The non-transitory storage medium of claim 29 , wherein the one or more images comprise a plurality of frames of video from the RGB-D camera.
33 . The non-transitory storage medium of claim 29 , wherein fusing the color and depth channels further comprises synchronizing the color and depth channels.
34 . The non-transitory storage medium of claim 29 , wherein using the landmark detector comprises using a convolutional neural network.
35 . The non-transitory storage medium of claim 29 , wherein using the landmark detector comprises using a deep landmark detection network.
36 . The non-transitory storage medium of claim 29 , wherein using the landmark detector comprises edge detection.
37 . The non-transitory storage medium of claim 29 , wherein using the landmark detector comprises using an optical flow algorithm.
38 . The non-transitory storage medium of claim 29 , wherein using the landmark detector comprises using a facial landmark detection model.
39 . The non-transitory storage medium of claim 29 , wherein using the landmark detector comprises using SIFT or SURF as a feature-based methodology.
40 . The non-transitory storage medium of claim 29 , wherein the processor is configured to calculate the distance between the points using the Cartesian distance formula.
41 . The non-transitory storage medium of claim 29 , wherein the processor is further configured to apply a temporal filter to successive values of the calculated point distance between the points.
42 . The non-transitory storage medium of claim 29 , wherein the processor is further configured to incorporate a vergence adjustment into the calculated point distance, wherein the two points correspond to pupils of a user, and wherein the calculated distance comprises an interpupillary distance (IPD).