Correction of subject area detection information, and image combining apparatus and method using the correction

1. An image combining method for a display apparatus for combining a real image obtained by image sensing real space with a computer-generated image, and for displaying a combined image, comprising:

using a display apparatus to perform the steps of:

a real image acquisition step of acquiring a first real image including an object to be a subject of a mask area, and of acquiring a second real image not including the object;

a determination step of determining colors which exist in the first real image and do not exist in the second real image;

a definition step of defining a closed area in a color space by connecting with an outside line a group of points indicating the determined colors in the color space and interpolating colors inside the connected line;

an extraction step of extracting an area, whose color is included in the determined and interpolated colors in the closed area defined at said definition step, as a mask area from said real image; and

a combining step of combining said real image with said computer-generated image by using said mask area extracted at said extraction step.

2. The image combining method according to claim 1 , wherein the color determined at said determination step is represented with color coordinate information in color space constructed with luminance coordinate information and color coordinate information.

3. The image combining method according to claim 1 , wherein at said extraction step, regarding each mask area extracted from said real image based on the color determined at said determination step, it is determined whether or not the mask area is noise, and an area determined as noise is changed to an area other than the mask area.

4. The image combining method according to claim 3 , wherein at said extraction step, if the area of extracted mask area is smaller than a predetermined value, the mask area is determined as noise.

5. The image combining method according to claim 3 , wherein at said extraction step, extracted mask areas are sorted in area size order, and a mask area in a predetermined place from the largest mask area and subsequent smaller mask areas are determined as noise.

6. The image combining method according to claim 1 , wherein at said extraction step, regarding each area other than the mask area extracted from said real image based on the mask area, it is determined whether or not it is noise, and an area determined as noise is changed to a mask area.

7. The image combining method according to claim 6 , wherein at said extraction step, if the area other than the extracted mask area is smaller than a predetermined value, the area is determined as noise.

8. The image combining method according to claim 6 , wherein at said extraction step, extracted areas other than the mask area are sorted in area size order, and an area in a predetermined place from the largest area and subsequent smaller areas are determined as noise.

9. The image combining method according to claim 1 , further comprising:

a presentation step of generating a subject area image where the mask area extracted at said extraction step and an area other than the mask area are represented in visually different manners, and presenting a correction image obtained by superposing the subject area image, as a translucent image, over said real image, via a user interface; and

an update step of generating color information of mask area to be added/deleted to/from color information of an area of said real image corresponding to an area of said correction image designated via said user interface, and updating the currently-registered mask area color information.

10. The image combining method according to claim 9 , wherein in said subject area image, said predetermined subject area and said other area than said subject area have different colors.

11. An image combining apparatus which combines a real image obtained by image sensing real space with a computer-generated image, and which displays the combined image, comprising:

a real image acquisition unit configured to acquire a first real image including an object to be a subject of a mask area, and to acquire a second real image not including the object;

a determination unit configured to determine colors which exists in the first real image and do not exist in the second image;

a definition unit to define a closed area in a color space by connecting with an outside line a group of points indicating the determined colors in the color space and interpolating colors inside the connected line;

an extraction unit configured to extract an area, whose color is included in the determined and interpolated colors in the closed area defined by said definition unit, as a mask area from said real image; and

a combining unit configured to combine said real image with said computer-generated image by using said mask area extracted by said extraction unit.

12. A computer program stored on a computer readable medium for use by a computer to perform the respective steps of the image combining method according to claim 1 .

13. A computer-readable memory storing a computer program of instructions which, when executed by a computer processing system, causes the system to perform the image combining method according to claim 1 .