Visual tracking eye glasses in visual head and eye tracking systems
The invention relates to the application area of camera-based head and eye tracking systems. The performance of such systems typically suffers when eye glasses are worn, as the frames of the glasses interfere with the tracking of the facial features utilized by the system. This invention describes how the appearance of the glasses can be utilized by such a tracking system, not only eliminating the interference of the glasses with the tracking but also aiding the tracking of the facial features. The invention utilizes a shape model of the glasses which can be tracked by a specialized tracker to derive 3D pose information.
1. In a tracking system configured to track a face/head, a method of tracking facial features of the face/head, the method including the step of:
(a) determining if glasses rims, frames or arms are present on the face/head; and
(b) where said rims, frames or arms are present on the face/head, determining appearance characteristics of the rims, frames or arms thereby to derive 2D image position or 3D pose measurements of the glasses or the face/head,
wherein deriving the 2D image position or 3D pose measurements includes performing analysis thereby to progressively optimise the fit of a three dimensional generative glasses model to a progressive set of sequential input images containing the face/head, wherein the fit is defined in terms of glasses shape parameters and glasses pose parameters.
2. A method as claimed in claim 1 wherein said method utilizes the 2D image position or 3D pose measurements derived from the determined appearance characteristics of frames, rim or arms of the glasses to track a 2D image position or 3D pose of other facial features of the face, or to track a 2D image position or 3D pose of the head as a whole.
3. A method as claimed in claim 1 including, on the basis of the 2D image position or 3D pose measurements, removing, excluding or weighting one or more image regions covered by the frames, rims, arms or lenses of the glasses for further processing an image containing the face/head.
4. A method as claimed in claim 1 wherein one or more edges in the image generated by the rims, frames or arms of the glasses are used to track a position of the 2D image position or 3D pose of the glasses.
5. A method as claimed in claim 1 wherein a 2D or 3D shape model of the rim or frame of the glasses is utilized to derive the 2D image position or 3D pose measurements.
6. A method as claimed in claim 1 wherein a 2D or 3D generative model of the rim or frame of the glasses is utilized to derive the 2D image position or 3D measurements.
7. A face or facial feature tracking system wherein appearance characteristics of frames, rims or arms of glasses worn on a face/head are used as part of a face or facial feature tracking system, including performing analysis thereby to progressively optimise the fit of a three dimensional generative glasses model to a progressive set of sequential input images containing the face/head, wherein the fit is defined in terms of glasses shape parameters and glasses pose parameters.
8. A face or facial feature tracking system which determines appearance characteristics of the frames, rim or arms a pair of glasses worn on the face/head to track the 2D image position or 3D pose of the pair of glasses or the face/head, including performing analysis thereby to progressively optimise the fit of a three dimensional generative glasses model to a progressive set of sequential input images containing the face/head, wherein the fit is defined in terms of glasses shape parameters and glasses pose parameters.
9. A face or facial feature tracking system which utilizes 2D image position or 3D pose measurements derived from appearance characteristics of frames, rim or arms of glasses worn on a face/head to track a 2D image position or 3D pose of other facial features of the face/head or a 2D image position or 3D pose of the head, wherein deriving the 2D image position or 3D pose measurements includes performing analysis thereby to progressively optimise the fit of a three dimensional generative glasses model to a progressive set of sequential input images containing the face/head, wherein the fit is defined in terms of glasses shape parameters and glasses pose parameters.
10. A method of facial image processing including:
(a) receiving data indicative of a progressive set of sequential input facial images; and
(b) tracking, in the facial images, 2D image position or 3D pose of a pair of glasses worn by a face/head, the glasses having a frame, rim, lenses and arms, thereby to remove, exclude or weight one or more image regions covered by the frame, rim, arms or lenses of the glasses for further processing of the facial images, wherein tracking the 2D image position or 3D pose includes performing analysis thereby to progressively optimise the fit of a three dimensional generative glasses model to the input images, wherein the fit is defined in terms of glasses shape parameters and glasses pose parameters.
11. A method of tracking a glasses frame on a face in an input image sequence, the method comprising the steps of:
utilising a training database of images to derive first example structural information of glasses on a subject;
utilising principle component analysis of said first example structural information to derive a three-dimensional generative model of the structural information, said generative model including a series of parameter values; and
for an initial input image in the input image sequence, deriving an initial set of parameter values and a corresponding initial generative model generated thereby, comparing the initial generative model with the initial input image in the input image sequence to derive a difference measure thereof, and altering the parameter values so as to minimise the difference measure;
for a subsequent input image in the input image sequence, deriving a subsequent set of parameter values, using the altered parameter values, and a corresponding subsequent generative model generated thereby, comparing the initial generative model with the initial input image in the input image sequence to derive a difference measure thereof, and altering the parameter values so as to minimise the difference measure.
12. A method as claimed in claim 11 wherein the first example structural information includes 3-Dimensional contour information of the glasses.
13. A method as claimed in claim 12 wherein said structural information includes a series of control points for said contour information.
14. A method as claimed in claim 13 wherein the principle component analysis is done on said control points.
15. A method as claimed in claim 11 wherein said comparison is done between a projection of the initial model on a 2-dimensional space and a edge detection filtered version of the current input image.
16. A method as claimed in claim 11 wherein said minimisation includes Levenberg-Marquardt type minimisation.
17. A method as claimed in claim 11 further comprising incorporating the tracking of the glasses frame into a face tracker.