Image processing method, recording medium, and image processing system
An image processing method to be executed by an image processing system includes: estimating a structure of a space inside a construction from a background image in which the space is imaged in all directions; estimating a region in which a virtual object is allowed to be arranged in the space based on the estimated structure; and combining the virtual object with the background image in the estimated region.
1 . An image processing method, comprising:
estimating a structure of a space inside a construction from a background image in which the space is imaged in all directions;
detecting a subject appearing in the background image and a three-dimensional (3D) position of an imaging device which captured the background image, the subject including a fixed structural component of the construction;
estimating a 3D position of the subject in the space;
estimating, based on the structure of the space, the 3D position of the subject, information of the subject and information of a virtual object to be placed:
an arrangement allowable region, in which the virtual object is allowed to be arranged in the space, by generating a 3D space model based on the estimated structure of the space, calculating distances between potential placement positions and detected subject positions in the 3D space model, and identifying positions that satisfy predetermined distance thresholds from the 3D position of the subject and from other objects as the arrangement allowable region;
an arrangement prohibited region, in which the virtual object is prohibited from being arranged in the space, by defining a 3D coordinate region in the 3D space model that includes the 3D position of the imaging device, expanding the 3D coordinate region to include a predetermined radius around the imaging device position, and designating the expanded coordinate region as unavailable for virtual object placement; and
combining the virtual object with the background image in the arrangement allowable region, and not in the arrangement prohibited region, to generate a staged image including the virtual object in the space.
2 . The image processing method according to claim 1 , wherein
the detecting detects a type of the subject, and
the information of the subject includes the detected type of the subject.
3 . The image processing method according to claim 1 , wherein
the detecting detects a support supporting an imager configured to image the space, and
the combining combines a predetermined image with the background image to generate the staged image, and the staged image hides the detected support.
4 . The image processing method according to claim 1 , wherein
when the subject includes a particular subject serving as a light source in the background image, the detecting detects the particular subject,
the estimating the 3D position estimates a position of the detected particular subject in the space, and
the combining combines an image, indicative of a virtual light source, at the 3D position in the space of the particular subject estimated in the estimating the 3D position.
5 . The image processing method according to claim 1 , wherein the estimating the structure estimates a size of the space.
6 . The image processing method according to claim 1 , wherein
the estimating the structure estimates a purpose of the space,
the image processing method further comprises determining the virtual object based on the purpose, and
the combining combines the virtual object with the background image to generate the staged image.
7 . The image processing method according to claim 1 , further comprising:
receiving purpose information, indicative of a purpose of the space, from an external device; and
determining the virtual object based on the received purpose information, wherein
the combining combines the determined virtual object with the background image to generate the staged image.
8 . The image processing method according to claim 6 , wherein the determining selects the virtual object corresponding to the purpose of the space from condition information indicative of an arrangement condition of a virtual object stored in a memory of an image processing system.
9 . The image processing method according to claim 1 , further comprising:
arranging the virtual object in the arrangement allowable region, wherein
the combining combines the arranged virtual object with the background image in the arrangement allowable region.
10 . A non-transitory computer readable recording medium to cause a computer to execute the image processing method comprising:
estimating a structure of a space inside a construction from a background image in which the space is imaged in all directions;
detecting a subject appearing in the background image and a three-dimensional (3D) position of an imaging device which captured the background image, the subject including a fixed structural component of the construction;
estimating a 3D position of the subject in the space;
estimating, based on the structure of the space, the 3D position of the subject, information of the subject and information of a virtual object to be placed:
an arrangement allowable region, in which the virtual object is allowed to be arranged in the space, by generating a 3D space model based on the estimated structure of the space, calculating distances between potential placement positions and detected subject positions in the 3D space model, and identifying positions that satisfy predetermined distance thresholds from the 3D position of the subject and from other objects as the arrangement allowable region;
an arrangement prohibited region, in which the virtual object is prohibited from being arranged in the space, by defining a 3D coordinate region in the 3D space model that includes the 3D position of the imaging device, expanding the 3D coordinate region to include a predetermined radius around the imaging device position, and designating the expanded coordinate region as unavailable for virtual object placement; and
combining the virtual object with the background image in the arrangement allowable region, and not in the arrangement prohibited region, to generate a staged image including the virtual object in the space.
11 . An image processing system, comprising:
circuitry configured to
estimate a structure of a space inside a construction from a background image in which the space is imaged in all directions;
detect a subject appearing in the background image and a three-dimensional (3D) position of an imaging device which captured the background image, the subject including a fixed structural component of the construction;
estimate a 3D position of the subject in the space;
estimate, based on the structure of the space, the 3D position of the subject, information of the subject and information of a virtual object to be placed:
an arrangement allowable region, in which the virtual object is allowed to be arranged in the space, by generating a 3D space model based on the estimated structure of the space, calculating distances between potential placement positions and detected subject positions in the 3D space model, and identifying positions that satisfy predetermined distance thresholds from the 3D position of the subject and from other objects as the arrangement allowable region;
an arrangement prohibited region, in which the virtual object is prohibited from being arranged in the space, by defining a 3D coordinate region in the 3D space model that includes the 3D position of the imaging device, expanding the 3D coordinate region to include a predetermined radius around the imaging device position, and designating the expanded coordinate region as unavailable for virtual object placement; and
combine the virtual object with the background image in the arrangement allowable region, and not in the arrangement prohibited region, to generate a staged image including the virtual object in the space.
12 . The image processing system according to claim 11 , wherein the circuitry is further configured to control a display to display the staged image.
13 . The image processing system according to claim 12 , wherein the circuitry controls the display to switch display between the background image and the staged image.
14 . The image processing system according to claim 12 , wherein the circuitry is further configured to
calculate a center position of the virtual object in the staged image, and
control the display to display additional information in a superimposed manner at the calculated center position on the staged image.
15 . The image processing system according to claim 14 , wherein the additional information is one of an icon corresponding to the virtual object and a link to a Web site.
16 . The image processing system according to claim 12 , wherein the circuitry is further configured to control the display to display an image which is one of a color image of the virtual object appearing in the staged image and an edge image in which an edge of the virtual object appearing in the staged image is enhanced.
17 . The image processing method according to claim 1 , wherein
the structure of the space indicates dimensions of the space,
the 3D position of the subject indicates a coordinate position of the subject within the dimensions of the space,
the information of the subject indicates dimensions of the subject, and
the information of the virtual object indicates virtual dimensions of the object to be placed.
18 . The image processing method according to claim 17 , wherein
in a case that the virtual object is furniture, the arrangement allowable region is estimated to be a first threshold distance away from the subject and a second threshold distance away from other objects already within the space, and
in a case that the virtual object is a rug, the arrangement allowable region is estimated to overlap with other objects already within the space.