Image processing device, image processing method, and image processing system
In a current image processing environment, providing feedback for a user operation is slow, and a user cannot confirm the result of the operation in real time. Thus, the user interrupts the operation every time the user confirms the feedback, and a time period for performing image processing is long. An image processing device includes a guidance information generator. The guidance information generator acquires trajectory information to be used to segment an image object on an image input from a user, generates guidance information indicating a segmentation region desired by the user based on the trajectory information, and presents the segmentation region based on a user's intention in real-time. The guidance information generator provides a function of smoothly continuously segmenting an image object.
1. An image processing device for segmenting an image object, comprising:
a guidance information generator that generates a segmentation region desired by a user from trajectory information to be used to segment the image object input by the user,
wherein the guidance information generator further comprises a deletion detector configured to compare a first segmentation region corresponding to the trajectory information input by the user associated with a first time period and a first direction and including a first region on an image as a region of a first object with a second segmentation region corresponding to the trajectory information input by the user associated with a second time period, that is after the first time period and is equal to or shorter than a predetermined time period, and a second direction, and
when it is determined that the second segmentation region reaches a predetermined reverse criterion, defined at least as the second direction being opposite of the first direction, a guidance information calculator generates a third segmentation region from which the first region has been removed from the region of the first object.
2. The image processing device according to claim 1 , wherein
the guidance information generator is configured to
detect a stroke input on the image object on the image, and
extract, as the trajectory information, position information and direction information that correspond to a stroke segment that has been input for a certain time period and is a part of the stroke.
3. The image processing device according to claim 2 , wherein
the guidance information generator includes a trajectory transformer and a recurrent neural network (“RNN”) characteristic extractor,
the trajectory transformer is configured to
calculate, from the position information indicated in the trajectory information, a position map indicating a route through which the stroke segment has passed on the image, and
calculate, from the direction information indicated in the trajectory information, a direction map indicating a direction in which the route through which the stroke segment has passed on the image extends, and
the RNN characteristic extractor is configured to
generate an intention map indicating, for each of pixels within the image, a probability with which the stroke passes through the pixel, based on the image, the direction map, and the position map.
4. The image processing device according to claim 3 , wherein
the guidance information generator further includes a refining unit,
the guidance information calculator is configured to
calculate the segmentation region based on the image and the intention map, and
the refining unit is configured to
generate a final image indicating the segmentation region and present the generated final image to the user.
5. The image processing device according to claim 1 , further comprising:
a user interface, wherein
the guidance information generator includes a trajectory transformer, a recurrent neural network (“RNN”) characteristic extractor, and a refining unit,
the guidance information generator is configured to
detect a stroke input on the image object on the image via the user interface, and
detect, as the trajectory information, position information and direction information that correspond to a stroke segment that has been input for a certain time period and is a part of the stroke,
the trajectory transformer is configured to
calculate, from the position information indicated in the trajectory information, a position map indicating a route through which the stroke segment has passed on the image, and
calculate, from the direction information indicated in the trajectory information, a direction map indicating a direction in which the route through which the stroke segment has passed on the image extends,
the RNN characteristic extractor is configured to
generate an intention map indicating, for each of pixels within the image, a probability with which the stroke passes through the pixel, based on the image, the direction map, and the position map,
the guidance information calculator is configured to
calculate the segmentation region based on the image and the intention map, and
the refining unit is configured to
generate a final image indicating the image and the segmentation region and present the final image to the user via the user interface.
6. An image processing method to be executed by an image processing device for segmenting an image object, wherein
the image processing device includes a user interface and a guidance information generator, and
the image processing method comprises:
detecting an input stroke on an image displayed on the user interface;
extracting trajectory information corresponding to a stroke segment that has been input for a certain time period and is a part of the stroke;
generating a final image indicating a segmentation region based on the trajectory information;
presenting the final image on the user interface; and
comparing a first segmentation region corresponding to the trajectory information associated with a first time period and a first direction and including a first region on the image as a region of a first object with a second segmentation region corresponding to the trajectory information associated with a second time period, that is after the first time period and is equal to or shorter than a predetermined time period, and a second direction, and
when it is determined that the second segmentation region reaches a predetermined reverse criterion, defined at least as the second direction being opposite of the first direction, a guidance information calculator generates a third segmentation region from which the first region has been removed from the region of the first object.
7. The image processing method according to claim 6 , wherein
the image processing device further includes a guidance information generator,
the guidance information generator includes a trajectory transformer and a recurrent neural network (“RNN”) characteristic extractor,
the trajectory information includes position information and direction information, and
the generating the final image includes
causing the trajectory transformer to calculate, from the position information included in the trajectory information, a position map indicating a route through which the stroke segment has passed on the image,
causing the trajectory transformer to calculate, from the direction information included in the trajectory information, a direction map indicating a direction in which the route through which the stroke segment has passed on the image extends, and
causing the RNN characteristic extractor to generate an intention map indicating, for each of pixels within the image, a probability with which the stroke passes through the pixel, based on the image, the direction map, and the position map.
8. An image processing system for segmenting an image object, comprising:
an image processing device; and
a client terminal, wherein
the image processing device and the client terminal are connected to each other via a communication network,
the image processing device includes a guidance information generator wherein the guidance information generator further comprises a guidance information calculator and a deletion detector, and
the guidance information generator is configured to
receive, from the client terminal, an image and stroke information on a stroke input on the image object on the image,
calculate trajectory information indicating a trajectory of the stroke based on the stroke information, and
generate a segmentation region desired by a user from the trajectory information and transmit the generated segmentation region to the client terminal, and
the deletion detector is configured to
compare a first segmentation region corresponding to the trajectory information a first time period and a first direction and including a first region on the image as a region of a first object with a second segmentation region corresponding to the trajectory information a second time period, that is after the first time period and is equal to or shorter than a predetermined time period, and a second direction, and
when it is determined that the second segmentation region reaches a predetermined reverse criterion, defined at least as the second direction being opposite of the first direction, the guidance information calculator generates a third segmentation region from which the first region has been removed from the region of the first object.
9. The image processing system according to claim 8 , wherein
the guidance information generator is configured to
calculate the trajectory information by extracting position information and direction information that correspond to a stroke segment that has been input for a certain time period and is a part of the stroke corresponding to the stroke information.
10. The image processing system according to claim 9 , wherein
the guidance information generator includes a trajectory transformer and a recurrent neural network (“RNN”) characteristic extractor,
the trajectory transformer is configured to
calculate, from the position information indicated in the trajectory information, a position map indicating a route through which the stroke segment has passed on the image, and
calculate, from the direction information indicated in the trajectory information, a direction map indicating a direction in which the route through which the stroke segment has passed on the image extends, and
the RNN characteristic extractor is configured to
generate an intention map indicating, for each of pixels within the image, a probability with which the stroke passes the pixel, based on the image, the direction map, and the position map.