Image processing method and apparatus
View Patent ↗An image processing technique includes acquiring a main image of a scene and determining one or more facial regions in the main image. The facial regions are analysed to determine if any of the facial regions includes a defect. A sequence of relatively low resolution images nominally of the same scene is also acquired. One or more sets of low resolution facial regions in the sequence of low resolution images are determined and analysed for defects. Defect free facial regions of a set are combined to provide a high quality defect free facial region. At least a portion of any defective facial regions of the main image are corrected with image information from a corresponding high quality defect free facial region.
1. An image processing method comprising:
a) acquiring a main image of a scene;
b) determining one or more facial regions in said main image;
c) analysing said one or more facial regions to determine if any one of said facial regions includes a defect;
d) acquiring a sequence of relatively low resolution images nominally of said scene for a time period not including a time for acquiring said main image;
e) determining one or more sets of low resolution facial regions in said sequence of low resolution images;
f) analysing said sets of facial regions to determine if any facial regions of a set corresponding to a defective facial region of said main image include a defect;
g) combining defect free facial regions of said corresponding set to provide a high quality defect free facial region; and
h) correcting at least a portion of any defective facial regions of said main image with image information from a corresponding high quality defect free facial region,
wherein said step f) of analysing further comprises: prior to said combining, removing facial regions including faces exceeding an average size of faces in a set of facial regions by a threshold amount from said set of facial regions.
2. A method according to claim 1 wherein said time period includes one or more of a time period preceding or a time period following said time for acquiring said main image.
3. A method according to claim 1 wherein said correcting step comprises:
applying a model comprising a plurality of vertices defining a periphery of a facial region to each high quality defect free facial region and a corresponding defective facial region; and
mapping pixels of said high quality defect free facial region to said defective facial region according to the correspondence of vertices for said respective regions.
4. A method according to claim 3 wherein said model comprises an Active Appearance Model (AAM).
5. A method according to claim 1 wherein said main image is acquired at an exposure level different to the exposure level of said low resolution images and wherein said correcting comprises mapping luminance levels of said high quality defect free facial region to luminance levels of said defective facial region.
6. A method according to claim 1 comprising storing sets of low resolution facial regions from said sequence of low resolution images in an image header file of said main image.
7. A method according to claim 1 comprising displaying said main image; and responsive to user interaction, performing steps c) and steps f) to h).
8. A method according to claim 1 wherein step e) is performed at a time preceding performing step a).
9. A method according to claim 1 comprising: displaying said corrected image; and responsive to user interaction, saving said corrected image.
10. A method according to claim 1 wherein said step f) of analysing comprises:
applying an Active Appearance Model (AAM) to each face of a set of facial regions;
analysing AAM parameters for each face of the set of facial regions to provide an indication of facial expression; and
prior to said combining, removing faces having a defective expression from said set of facial regions.
11. A method according to claim 1 wherein said step f) of analysing comprises:
applying an Active Appearance Model (AAM) to each face of a set of facial regions;
analysing AAM parameters for each face of the set of facial regions to provide an indication of facial orientation; and
prior to said combining, removing faces having an undesirable orientation from said set of facial regions.
12. A method according to claim 1 wherein said step c) of analysing comprises:
applying an Active Appearance Model (AAM) to each facial region; and
analysing AAM parameters for each facial region to provide an indication of facial expression.
13. A method according to claim 1 wherein said step c) of analysing comprises:
analysing each facial region for one or more of: contrast, sharpness, texture, luminance levels or skin color.
14. A method according to claim 1 wherein said step c) of analysing comprises:
analysing each facial region to determine if any eyes of said facial region are closed.
15. A method according to claim 1 wherein said step c) of analysing comprises:
analysing each facial region to determine if any mouth of said facial region is open.
16. A method according to claim 1 wherein said step c) of analysing comprises:
analysing each facial region to determine if any mouth of said facial region is smiling.
17. A method according to claim 1 wherein steps g) and h) are performed only when said set of facial regions exceeds a given number.
18. A method according to claim 1 comprising: prior to said combining, resizing and aligning faces of said set of facial regions.
19. A method according to claim 18 , wherein said aligning is performed according to cardinal points of faces of said set of facial regions.
20. A method according to claim 1 wherein said correcting further comprises one or more of: blending; or infilling a corrected region of said main image with the remainder of said main image.
21. A method according to claim 1 comprising: displaying said corrected image; and responsive to user interaction, adjusting the correction of said corrected image.
22. An image processing method comprising:
a) acquiring a main image of a scene;
b) determining one or more facial regions in said main image;
c) analysing said one or more main image facial regions for defects and determining that one or more are defective;
d) acquiring a sequence of relatively low resolution images nominally of said scene;
e) determining one or more sets of low resolution facial regions in said sequence of low resolution images;
f) analysing said one or more sets of low resolution facial regions to determine one or more that correspond to a defective main image facial region;
g) correcting at least a portion of a defective main image facial region with image information from one or more corresponding low resolution facial regions not including a same defect as said portion of said defective main image facial region,
wherein said step f) of analysing further comprises: prior to said combining, removing facial regions including faces exceeding an average size of faces in a set of facial regions by a threshold amount from said set of facial regions.
23. A method according to claim 22 wherein said correcting comprises:
applying a model comprising a plurality of vertices defining a periphery of a facial region to one or more low resolution facial regions not including said same defect and a corresponding defective main image facial region; and
mapping pixels of said one or more low resolution facial regions to said corresponding defective main image facial region according to the correspondence of vertices for respective regions.
24. A method according to claim 23 wherein said model comprises an Active Appearance Model (AAM).
25. A method according to claim 22 wherein said main image is acquired at an exposure level different than the exposure level of said low resolution images and wherein said correcting comprises mapping luminance levels of one or more low resolution facial regions not have said same defect to luminance levels of said defective main image facial region.
26. A method according to claim 22 comprising storing sets of low resolution facial regions from said sequence of low resolution images in an image header file of said main image.
27. A method according to claim 22 , further comprising displaying said main image; and responsive to user interaction, selectively performing said analyzing said one or more facial regions, said analyzing said one of more sets or said correcting, or combinations thereof.
28. A method according to claim 22 , further comprising displaying a corrected image based on said correcting, and responsive to user interaction, saving said corrected image.
29. A method according to claim 22 , wherein said analysing said one or more sets comprises:
applying an Active Appearance Model (AAM) to each face of a set of facial regions;
analysing AAM parameters for each face of the set of facial regions to provide an indication of facial expression; and
prior to said combining, removing faces having a defective expression from said set of facial regions.
30. A method according to claim 22 , wherein said analysing said one or more sets comprises:
applying an Active Appearance Model (AAM) to each face of a set of facial regions;
analysing AAM parameters for each face of the set of facial regions to provide an indication of facial orientation; and
prior to said combining, removing faces having an undesirable orientation from said set of facial regions.
31. A method according to claim 22 wherein said analysing one or more facial regions comprises:
applying an Active Appearance Model (AAM) to each facial region; and
analysing AAM parameters for each facial region to provide an indication of facial expression.
32. A method according to claim 22 , wherein said analysing one or more facial regions comprises analysing at least one facial region to determine if an eye of said facial region is closed.
33. A method according to claim 22 , wherein said analysing one or more facial regions comprises analysing at least one facial region to determine if a mouth of said facial region is open.
34. A method according to claim 22 , wherein said analysing one or more facial regions comprises analysing at least one facial region to determine if a mouth of said facial region is frowning.
35. A method according to claim 22 further comprising displaying a corrected image based on said correcting and responsive to user interaction, adjusting the correcting of said corrected image.
36. A method according to claim 22 , wherein said sequence of low resolution images is acquired for a time period not including a time for acquiring said main image.
37. A method according to claim 22 , further comprising combining defect-free low resolution facial regions into a combined image, and correcting at least said portion of said defective main image facial region with image information from the combined image.
38. An image processing apparatus comprising:
a) means for acquiring a main image of a scene;
b) means for determining one or more facial regions in said main image;
c) analysing said one or more facial regions to determine if any one of said facial regions includes a defect;
d) means for acquiring a sequence of relatively low resolution images nominally of said scene for a time period not including a time for acquiring said main image;
e) means for determining one or more sets of low resolution facial regions in said sequence of low resolution images;
f) means for analysing said sets of facial regions to determine if any facial regions of a set corresponding to a defective facial region of said main image include a defect;
g) means for combining defect free facial regions of said corresponding set to provide a high quality defect free facial region; and
h) means for correcting at least a portion of any defective facial regions of said main image with image information from a corresponding high quality defect free facial region,
wherein said element f) means for analysing further comprises: means for removing facial regions, prior to said combining, including faces exceeding an average size of faces in a set of facial regions by a threshold amount from said set of facial regions.
39. An image processing apparatus according to claim 38 comprising a digital image acquisition device or general purpose computer or both.
40. An apparatus for providing images with enhanced facial regions in digital images, including a processor and one or more processor-readable media for programming the processor to control the apparatus to perform an image processing method that comprises:
a) acquiring a main image of a scene;
b) determining one or more facial regions in said main image;
c) analysing said one or more main image facial regions for defects and determining that one or more are defective;
d) acquiring a sequence of relatively low resolution images nominally of said scene;
e) determining one or more sets of low resolution facial regions in said sequence of low resolution images;
f) analysing said one or more sets of low resolution facial regions to determine one or more that correspond to a defective main image facial region; and
g) correcting at least a portion of a defective main image facial region with image information from one or more corresponding low resolution facial regions not including a same defect as said portion of said defective main image facial region,
wherein said step f) of analysing further comprises: prior to said combining, removing facial regions including faces exceeding an average size of faces in a set of facial regions by a threshold amount from said set of facial regions.
41. An apparatus according to claim 40 , wherein said correcting comprises:
applying a model comprising a plurality of vertices defining a periphery of a facial region to one or more low resolution facial regions not including said same defect and a corresponding defective main image facial region; and
mapping pixels of said one or more low resolution facial regions to said corresponding defective main image facial region according to the correspondence of vertices for respective regions.
42. An apparatus according to claim 41 wherein said model comprises an Active Appearance Model (AAM).
43. An apparatus according to claim 40 wherein said main image is acquired at an exposure level different than the exposure level of said low resolution images and wherein said correcting comprises mapping luminance levels of one or more low resolution facial regions not have said same defect to luminance levels of said defective main image facial region.
44. An apparatus according to claim 40 wherein the method further comprises storing sets of low resolution facial regions from said sequence of low resolution images in an image header file of said main image.
45. An apparatus according to claim 40 , wherein the method further comprises displaying said main image; and responsive to user interaction, selectively performing said analyzing said one or more facial regions, said analyzing said one of more sets or said correcting, or combinations thereof.
46. An apparatus according to claim 40 , wherein the method further comprises displaying a corrected image based on said correcting, and responsive to user interaction, saving said corrected image.
47. An apparatus according to claim 40 , wherein said analysing said one or more sets comprises:
applying an Active Appearance Model (AAM) to each face of a set of facial regions;
analysing AAM parameters for each face of the set of facial regions to provide an indication of facial expression; and
prior to said combining, removing faces having a defective expression from said set of facial regions.
48. An apparatus according to claim 40 , wherein said analysing said one or more sets comprises:
applying an Active Appearance Model (AAM) to each face of a set of facial regions;
analysing AAM parameters for each face of the set of facial regions to provide an indication of facial orientation; and
prior to said combining, removing faces having an undesirable orientation from said set of facial regions.
49. An apparatus according to claim 40 wherein said analysing one or more facial regions comprises:
applying an Active Appearance Model (AAM) to each facial region; and
analysing AAM parameters for each facial region to provide an indication of facial expression.
50. An apparatus according to claim 40 , wherein said analysing one or more facial regions comprises analysing at least one facial region to determine if an eye of said facial region is closed.
51. An apparatus according to claim 40 , wherein said analysing one or more facial regions comprises analysing at least one facial region to determine if a mouth of said facial region is open.
52. An apparatus according to claim 40 , wherein said analysing one or more facial regions comprises analysing at least one facial region to determine if a mouth of said facial region is frowning.
53. An apparatus according to claim 40 wherein the method further comprises displaying a corrected image based on said correcting and responsive to user interaction, adjusting the correcting of said corrected image.
54. An apparatus according to claim 40 , wherein said sequence of low resolution images is acquired for a time period not including a time for acquiring said main image.
55. An apparatus according to claim 40 , wherein the method further comprises combining defect-free low resolution facial regions into a combined image, and correcting at least said portion of said defective main image facial region with image information from the combined image.
56. An image processing method comprising:
a) acquiring a main image of a scene;
b) determining one or more facial regions in said main image;
c) analysing said one or more facial regions to determine if any one of said facial regions includes a defect;
d) acquiring a sequence of relatively low resolution images nominally of said scene for a time period not including a time for acquiring said main image;
e) determining one or more sets of low resolution facial regions in said sequence of low resolution images;
f) analysing said sets of facial regions to determine if any facial regions of a set corresponding to a defective facial region of said main image include a defect;
g) combining defect free facial regions of said corresponding set to provide a high quality defect free facial region; and
h) correcting at least a portion of any defective facial regions of said main image with image information from a corresponding high quality defect free facial region,
wherein said step f) of analysing further comprises: prior to said combining, removing facial regions including faces with an orientation outside an average orientation of faces in a set of facial regions by a threshold amount from said set of facial regions.
57. A method according to claim 56 wherein said time period includes one or more of a time period preceding or a time period following said time for acquiring said main image.
58. A method according to claim 56 wherein said correcting step comprises:
applying a model comprising a plurality of vertices defining a periphery of a facial region to each high quality defect free facial region and a corresponding defective facial region; and
mapping pixels of said high quality defect free facial region to said defective facial region according to the correspondence of vertices for said respective regions.
59. A method according to claim 58 wherein said model comprises an Active Appearance Model (AAM).
60. A method according to claim 56 wherein said main image is acquired at an exposure level different to the exposure level of said low resolution images and wherein said correcting comprises mapping luminance levels of said high quality defect free facial region to luminance levels of said defective facial region.
61. A method according to claim 56 comprising storing sets of low resolution facial regions from said sequence of low resolution images in an image header file of said main image.
62. A method according to claim 56 comprising displaying said main image; and responsive to user interaction, performing steps c) and steps f) to h).
63. A method according to claim 56 wherein step e) is performed at a time preceding performing step a).
64. A method according to claim 56 comprising: displaying said corrected image; and responsive to user interaction, saving said corrected image.
65. A method according to claim 56 wherein said step f) of analysing comprises:
applying an Active Appearance Model (AAM) to each face of a set of facial regions;
analysing AAM parameters for each face of the set of facial regions to provide an indication of facial expression; and
prior to said combining, removing faces having a defective expression from said set of facial regions.
66. A method according to claim 56 wherein said step f) of analysing comprises:
applying an Active Appearance Model (AAM) to each face of a set of facial regions;
analysing AAM parameters for each face of the set of facial regions to provide an indication of facial orientation; and
prior to said combining, removing faces having an undesirable orientation from said set of facial regions.
67. A method according to claim 56 wherein said step c) of analysing comprises:
applying an Active Appearance Model (AAM) to each facial region; and
analysing AAM parameters for each facial region to provide an indication of facial expression.
68. A method according to claim 56 wherein said step c) of analysing comprises:
analysing each facial region for one or more of: contrast, sharpness, texture, luminance levels or skin color.
69. A method according to claim 56 wherein said step c) of analysing comprises:
analysing each facial region to determine if any eyes of said facial region are closed.
70. A method according to claim 56 wherein said step c) of analysing comprises:
analysing each facial region to determine if any mouth of said facial region is open.
71. A method according to claim 56 wherein said step c) of analysing comprises:
analysing each facial region to determine if any mouth of said facial region is smiling.
72. A method according to claim 56 wherein steps g) and h) are performed only when said set of facial regions exceeds a given number.
73. A method according to claim 56 comprising: prior to said combining, resizing and aligning faces of said set of facial regions.
74. A method according to claim 73 , wherein said aligning is performed according to cardinal points of faces of said set of facial regions.
75. A method according to claim 56 wherein said correcting further comprises one or more of: blending; or infilling a corrected region of said main image with the remainder of said main image.
76. A method according to claim 56 comprising: displaying said corrected image; and responsive to user interaction, adjusting the correction of said corrected image.
77. An image processing method comprising:
a) acquiring a main image of a scene;
b) determining one or more facial regions in said main image;
c) analysing said one or more main image facial regions for defects and determining that one or more are defective;
d) acquiring a sequence of relatively low resolution images nominally of said scene;
e) determining one or more sets of low resolution facial regions in said sequence of low resolution images;
f) analysing said one or more sets of low resolution facial regions to determine one or more that correspond to a defective main image facial region;
g) correcting at least a portion of a defective main image facial region with image information from one or more corresponding low resolution facial regions not including a same defect as said portion of said defective main image facial region,
wherein said step f) of analysing further comprises: prior to said combining, removing facial regions including faces with an orientation outside an average orientation of faces in a set of facial regions by a threshold amount from said set of facial regions.
78. A method according to claim 77 wherein said correcting comprises:
applying a model comprising a plurality of vertices defining a periphery of a facial region to one or more low resolution facial regions not including said same defect and a corresponding defective main image facial region; and
mapping pixels of said one or more low resolution facial regions to said corresponding defective main image facial region according to the correspondence of vertices for respective regions.
79. A method according to claim 78 wherein said model comprises an Active Appearance Model (AAM).
80. A method according to claim 77 wherein said main image is acquired at an exposure level different than the exposure level of said low resolution images and wherein said correcting comprises mapping luminance levels of one or more low resolution facial regions not have said same defect to luminance levels of said defective main image facial region.
81. A method according to claim 77 comprising storing sets of low resolution facial regions from said sequence of low resolution images in an image header file of said main image.
82. A method according to claim 77 , further comprising displaying said main image; and responsive to user interaction, selectively performing said analyzing said one or more facial regions, said analyzing said one of more sets or said correcting, or combinations thereof.
83. A method according to claim 77 , further comprising displaying a corrected image based on said correcting, and responsive to user interaction, saving said corrected image.
84. A method according to claim 77 , wherein said analysing said one or more sets comprises:
applying an Active Appearance Model (AAM) to each face of a set of facial regions;
analysing AAM parameters for each face of the set of facial regions to provide an indication of facial expression; and
prior to said combining, removing faces having a defective expression from said set of facial regions.
85. A method according to claim 77 , wherein said analysing said one or more sets comprises:
applying an Active Appearance Model (AAM) to each face of a set of facial regions;
analysing AAM parameters for each face of the set of facial regions to provide an indication of facial orientation; and
prior to said combining, removing faces having an undesirable orientation from said set of facial regions.
86. A method according to claim 77 wherein said analysing one or more facial regions comprises:
applying an Active Appearance Model (AAM) to each facial region; and
analysing AAM parameters for each facial region to provide an indication of facial expression.
87. A method according to claim 77 , wherein said analysing one or more facial regions comprises analysing at least one facial region to determine if an eye of said facial region is closed.
88. A method according to claim 77 , wherein said analysing one or more facial regions comprises analysing at least one facial region to determine if a mouth of said facial region is open.
89. A method according to claim 77 , wherein said analysing one or more facial regions comprises analysing at least one facial region to determine if a mouth of said facial region is frowning.
90. A method according to claim 77 further comprising displaying a corrected image based on said correcting and responsive to user interaction, adjusting the correcting of said corrected image.
91. A method according to claim 77 , wherein said sequence of low resolution images is acquired for a time period not including a time for acquiring said main image.
92. A method according to claim 77 , further comprising combining defect-free low resolution facial regions into a combined image, and correcting at least said portion of said defective main image facial region with image information from the combined image.
93. An image processing apparatus comprising:
a) means for acquiring a main image of a scene;
b) means for determining one or more facial regions in said main image;
c) analysing said one or more facial regions to determine if any one of said facial regions includes a defect;
d) means for acquiring a sequence of relatively low resolution images nominally of said scene for a time period not including a time for acquiring said main image;
e) means for determining one or more sets of low resolution facial regions in said sequence of low resolution images;
f) means for analysing said sets of facial regions to determine if any facial regions of a set corresponding to a defective facial region of said main image include a defect;
g) means for combining defect free facial regions of said corresponding set to provide a high quality defect free facial region; and
h) means for correcting at least a portion of any defective facial regions of said main image with image information from a corresponding high quality defect free facial region,
wherein said element f) means for analysing further comprises: means for removing facial regions, prior to said combining, including faces with an orientation outside an average orientation of faces in a set of facial regions by a threshold amount from said set of facial regions.
94. An image processing apparatus according to claim 93 comprising a digital image acquisition device or general purpose computer or both.
95. An apparatus for providing images with enhanced facial regions in digital images, including a processor and one or more processor-readable media for programming the processor to control the apparatus to perform an image processing method that comprises:
a) acquiring a main image of a scene;
b) determining one or more facial regions in said main image;
c) analysing said one or more main image facial regions for defects and determining that one or more are defective;
d) acquiring a sequence of relatively low resolution images nominally of said scene;
e) determining one or more sets of low resolution facial regions in said sequence of low resolution images;
f) analysing said one or more sets of low resolution facial regions to determine one or more that correspond to a defective main image facial region; and
g) correcting at least a portion of a defective main image facial region with image information from one or more corresponding low resolution facial regions not including a same defect as said portion of said defective main image facial region,
wherein said step f) of analysing further comprises: prior to said combining, removing facial regions including faces with an orientation outside an average orientation of faces in a set of facial regions by a threshold amount from said set of facial regions.
96. An apparatus according to claim 95 , wherein said correcting comprises:
applying a model comprising a plurality of vertices defining a periphery of a facial region to one or more low resolution facial regions not including said same defect and a corresponding defective main image facial region; and
mapping pixels of said one or more low resolution facial regions to said corresponding defective main image facial region according to the correspondence of vertices for respective regions.
97. An apparatus according to claim 96 wherein said model comprises an Active Appearance Model (AAM).
98. An apparatus according to claim 95 wherein said main image is acquired at an exposure level different than the exposure level of said low resolution images and wherein said correcting comprises mapping luminance levels of one or more low resolution facial regions not have said same defect to luminance levels of said defective main image facial region.
99. An apparatus according to claim 95 wherein the method further comprises storing sets of low resolution facial regions from said sequence of low resolution images in an image header file of said main image.
100. An apparatus according to claim 95 , wherein the method further comprises displaying said main image; and responsive to user interaction, selectively performing said analyzing said one or more facial regions, said analyzing said one of more sets or said correcting, or combinations thereof.
101. An apparatus according to claim 95 , wherein the method further comprises displaying a corrected image based on said correcting, and responsive to user interaction, saving said corrected image.
102. An apparatus according to claim 95 , wherein said analysing said one or more sets comprises:
applying an Active Appearance Model (AAM) to each face of a set of facial regions;
analysing AAM parameters for each face of the set of facial regions to provide an indication of facial expression; and
prior to said combining, removing faces having a defective expression from said set of facial regions.
103. An apparatus according to claim 95 , wherein said analysing said one or more sets comprises:
applying an Active Appearance Model (AAM) to each face of a set of facial regions;
analysing AAM parameters for each face of the set of facial regions to provide an indication of facial orientation; and
prior to said combining, removing faces having an undesirable orientation from said set of facial regions.
104. An apparatus according to claim 95 wherein said analysing one or more facial regions comprises:
applying an Active Appearance Model (AAM) to each facial region; and
analysing AAM parameters for each facial region to provide an indication of facial expression.
105. An apparatus according to claim 95 , wherein said analysing one or more facial regions comprises analysing at least one facial region to determine if an eye of said facial region is closed.
106. An apparatus according to claim 95 , wherein said analysing one or more facial regions comprises analysing at least one facial region to determine if a mouth of said facial region is open.
107. An apparatus according to claim 95 , wherein said analysing one or more facial regions comprises analysing at least one facial region to determine if a mouth of said facial region is frowning.
108. An apparatus according to claim 95 wherein the method further comprises displaying a corrected image based on said correcting and responsive to user interaction, adjusting the correcting of said corrected image.
109. An apparatus according to claim 95 , wherein said sequence of low resolution images is acquired for a time period not including a time for acquiring said main image.
110. An apparatus according to claim 95 , wherein the method further comprises combining defect-free low resolution facial regions into a combined image, and correcting at least said portion of said defective main image facial region with image information from the combined image.