Method and apparatus for processing image interaction

1. A method of processing an interaction with an object in an input image, the method comprising:

extracting, using an encoder, an input feature from the input image;

identifying the interaction for an application to the input image from a user input;

converting the input feature to a second feature based on the interaction for the application to the input image; and

generating, using a decoder, a result image from the second feature,

wherein the interaction comprising any one or any combination of a rotation, a movement, and a transformation of the object in the input image,

wherein the converting of the input feature comprises

combining the input feature with a first feature map corresponding to a first interaction and a second feature map corresponding to a second interaction, and

converting the input feature combined with the first feature map and the second feature map to the second feature that is suitable for an input of the decoder, and

wherein the input feature has a same spatial size as the second feature.

2. The method of claim 1 , wherein the converting of the input feature comprises:

combining a feature map corresponding to the interaction with the input feature; and

converting, using a conversion network, the input feature combined with the feature map to the second feature that is suitable for an input of the decoder.

3. The method of claim 2 , wherein the feature map comprises a value corresponding to the interaction.

4. The method of claim 2 , further comprising:

identifying an interaction for application to the input image from the user input,

wherein the converting of the input feature comprises generating a feature map for combination with the input feature using a value corresponding to the identified interaction.

5. The method of claim 1 , wherein the converting of the input feature comprises:

selecting a conversion network corresponding to the interaction from conversion networks defined for each interaction; and

converting the input feature to the second feature using the selected conversion network.

6. The method of claim 5 , further comprising:

identifying an interaction for an application to the input image from the user input,

wherein the converting of the input feature further comprises identifying a conversion network corresponding to the identified interaction from the conversion networks.

7. The method of claim 1 , wherein

the input image and the result image are single-channel two-dimensional (2D) images,

the input feature comprises feature maps that have spatial sizes less than a spatial size of the input image, and

the second feature comprises feature maps that have spatial sizes less than a spatial size of the result image.

8. The method of claim 1 , wherein a spatial size of the input image is equal to a spatial size of the result image.

9. The method of claim 1 , wherein

the encoder comprises a convolutional neural network (CNN) configured to generate a multi-channel feature from a single-channel 2D image, and

the decoder comprises a CNN configured to generate a single-channel 2D image from a multi-channel feature.

10. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1 .

11. A training method to process an image interaction with an object in a training input image, the training method comprising:

extracting, using an encoder, a training input feature from the training input image;

converting, using a conversion network, the training input feature to a second feature based on the interaction for an application to the training input image;

generating, using a decoder, a training result image from the second feature; and

training the conversion network so that the training result image is substantially identical to an image generated by applying the interaction to the training input image,

wherein the interaction comprising any one or any combination of a rotation, a movement and a transformation of the object in the input image,

wherein the converting of the training input feature comprises

combining the training input feature with a first feature map corresponding to a first interaction and a second feature map corresponding to a second interaction; and

converting, using the conversion network, the training input feature combined with the first feature map and the second feature map to the second feature that is suitable for an input of the decoder, and

wherein the input feature has a same spatial size as the second feature.

12. The training method of claim 11 , wherein the converting of the training input feature comprises:

combining a feature map corresponding to the interaction with the training input feature; and

converting, using the conversion network, the training input feature combined with the feature map to the second feature that is suitable for an input of the decoder.

13. The training method of claim 11 , wherein the converting of the training input feature comprises:

applying a weight to a first feature map, in response to a feature map corresponding to the interaction being absent; and

converting, using the conversion network, the training input feature combined with the first feature map to the second feature that is suitable for an input of the decoder.

14. The training method of claim 11 , wherein the converting of the training input feature comprises:

combining a first feature map with a second feature map to obtain a combined feature map, in response to a feature map corresponding to the interaction being absent; and

converting, using the conversion network, the training input feature combined with the combined feature map to the second feature that is suitable for an input of the decoder.

15. The training method of claim 11 , wherein the converting of the training input feature comprises:

selecting a conversion network corresponding to the interaction from conversion networks defined for each interaction; and

converting the training input feature to the second feature using the selected conversion network.

16. The training method of claim 11 , further comprising:

training the encoder and the decoder so that the training result image is substantially identical to the image generated by applying the interaction to the training input image.

17. The training method of claim 11 , wherein

training input features are extracted by the encoder from training input images,

training result images are generated by the decoder from the training input features, and

the encoder and the decoder are pre-trained so that the training input images are substantially identical to the training result images.

18. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 11 .

19. An apparatus for processing an image interaction with an object in an input image, the apparatus comprising:

a processor comprising an encoder and a decoder, the processor being configured to:

extract, using the encoder, an input feature from the input image;

identify the interaction for an application to the input image from a user input;

convert the input feature to a second feature based on the interaction for the application to the input image; and

generate, using the decoder, a result image from the second feature,

wherein the interaction comprising any one or any combination of a rotation, a movement and a transformation of the object in the input image,

wherein the processor is further configured to

combine the input feature with a first feature map corresponding to a first interaction and a second feature map corresponding to a second interaction; and

convert the input feature combined with the first feature map and the second feature map to the second feature that is suitable for an input of the decoder,

wherein the input feature has a same spatial size as the second feature.

20. The apparatus of claim 19 , wherein the processor is further configured to:

combine a feature map corresponding to the interaction with the input feature; and

convert, using a conversion network, the input feature combined with the feature map to the second feature that is suitable for an input of the decoder.

21. The apparatus of claim 19 , wherein the processor is further configured to:

select a conversion network corresponding to the interaction from conversion networks defined for each interaction; and

convert the input feature to the second feature using the selected conversion network.

22. The apparatus of claim 19 , wherein

the input image and the result image are single-channel two-dimensional (2D) images;

the input feature comprises feature maps that have spatial sizes less than a spatial size of the input image; and

the second feature comprises feature maps that have spatial sizes less than a spatial size of the result image.

23. The apparatus of claim 19 , wherein

the encoder comprises a convolutional neural network (CNN) configured to generate a multi-channel feature from a single-channel 2D image; and

the decoder comprises a CNN configured to generate a single-channel 2D image from a multi-channel feature.

24. The apparatus of claim 19 , wherein the interaction comprises any one or any combination of a rotation, a movement and a transformation of an object in the input image.

25. A training apparatus to process an image interaction with an object in a training input image, the training apparatus comprising:

a processor comprising an encoder and a decoder, the processor being configured to:

extract, using the encoder, a training input feature from the training input image;

convert, using a conversion network, the training input feature to a second feature based on the interaction for an application to the training input image;

generate, using the decoder, a training result image from the second feature; and

train the conversion network so that the training result image is substantially identical to an image generated by applying the interaction to the training input image,

wherein the interaction comprising any one or any combination of a rotation, a movement and a transformation of an object in the input image,

wherein the processor is further configured to:

combine the training input feature with a first feature map corresponding to a first interaction and a second feature map corresponding to a second interaction; and

convert, using the conversion network, the training input feature combined with the first feature map and the second feature map to the second feature that is suitable for an input of the decoder,

wherein the input feature has a same spatial size as the second feature.

26. The training apparatus of claim 25 , wherein the processor is further configured to:

combine a feature map corresponding to the interaction with the training input feature; and

convert, using the conversion network, the training input feature combined with the feature map to the second feature that is suitable for an input of the decoder.

27. The training apparatus of claim 25 , wherein the processor is further configured to

select a conversion network corresponding to the interaction from conversion networks defined for each interaction; and

convert the training input feature to the second feature using the selected conversion network.