Image processing method and apparatus, electronic device, and storage medium

1 . An image processing method performed by an electronic device, the image processing method comprising:

acquiring a target image comprising a first object for motion transfer and a plurality of source images corresponding to the target image, wherein each of the plurality of source images includes a same object, which is a second object different from the first object in the target image;

for each source image of the plurality of source images:

performing multi-dimensional feature extraction on the source image and the target image to acquire keypoint feature information of corresponding keypoints in the source image and the target image, and appearance feature information corresponding to the source image, the keypoint feature information comprising keypoint perspective information;

performing perspective transformation on the keypoints according to the keypoint perspective information to acquire optic flow information of the keypoints, the optic flow information indicating transformation information of the keypoints from the source image to the target image;

determining motion information corresponding to the keypoints based on the optic flow information and the keypoint feature information, wherein the motion information includes attitude information and occlusion information of the source image;

fusing the attitude information, the occlusion information and the appearance feature information to acquire initial image information corresponding to the source image; and

fusing the initial image information corresponding to the plurality of source images to acquire a processed image based on a weighted combination using attention weights associated with the plurality of source images respectively, the processed image being an image acquired after transferring motion of the first object in the target image to the second object in the plurality of source images.

2 . The image processing method according to claim 1 , wherein the keypoint feature information further comprises keypoint position information and keypoint deformation information, and the performing perspective transformation on the keypoints according to the keypoint perspective information to acquire optic flow information of the keypoints comprises:

fusing the keypoint perspective information and the keypoint deformation information to acquire a perspective transformation parameter; and

performing perspective transformation on the keypoints according to the perspective transformation parameter and the keypoint position information to acquire the optic flow information of the keypoints.

3 . The image processing method according to claim 2 , wherein the fusing the keypoint perspective information and the keypoint deformation information to acquire a perspective transformation parameter comprises:

respectively constructing a first initial perspective transformation parameter of the source image and a second initial perspective transformation parameter of the target image based on the keypoint perspective information and the keypoint deformation information; and

fusing the first initial perspective transformation parameter and the second initial perspective transformation parameter to acquire the perspective transformation parameter.

4 . The image processing method according to claim 2 , wherein the performing perspective transformation on the keypoints according to the perspective transformation parameter and the keypoint position information to acquire the optic flow information of the keypoint comprises:

acquiring unit optic flow information of the keypoints, the unit optic flow information being optic flow information corresponding to a warping operation between the source image and the target image;

respectively screening out first position information of the keypoints in the source image and second position information of the corresponding keypoints in the target image from the keypoint position information;

performing the perspective transformation on the keypoints according to the unit optic flow information, the second position information and the perspective transformation parameter to acquire initial optic flow information of the keypoints; and

fusing the first position information and the initial optic flow information to acquire the optic flow information of the keypoints.

5 . The image processing method according to claim 4 , wherein the determining motion information corresponding to the keypoints based on the optic flow information and the keypoint feature information comprises:

performing the warping operation on the source image according to the optic flow information to acquire at least one warped source image;

fusing the unit optic flow information, the first position information, the second position information and the warped source image to acquire fused source image information; and

recognizing the motion information corresponding to the keypoints from the fused source image information.

6 . The image processing method according to claim 5 , wherein the fusing the unit optic flow information, the first position information, the second position information and the warped source image to acquire fused source image information comprises:

determining a first position difference value of the first position information and the unit optic flow information;

determining a second position difference value of the second position information and the unit optic flow information;

determining transformation position information of the keypoints according to the first position difference value and the second position difference value; and

stitching the transformation position information and the warped source image to acquire the fused source image information.

7 . The image processing method according to claim 5 , further comprising:

acquiring a source image sample and a target image sample;

performing a motion transfer from an object in the target image sample to an object in in the source image sample using a preset processing model to acquire a target motion image,

converging the preset processing model according to the target motion image and the target image sample to acquire a trained processing model based on a target loss function generated using:

a feature loss function generated based on input value differences between the target motion image and the target image sample at a plurality of zooming levels of an original resolution,

a keypoint position loss function generated by performing space transformation on the target image sample to acquire a space transformed image and performing keypoint feature extraction on the target image sample and the space transformed image, and

a keypoint deformation loss function generated based on a first keypoint deformation information obtained from the keypoint feature extraction performed on the target image sample and a second keypoint deformation information obtained from the keypoint feature extraction performed on the space transformed image;

wherein the recognizing motion information corresponding to the keypoints from the fused source image information comprises:

performing multi-dimensional feature extraction on the fused source image information by using the trained processing model to acquire local motion feature information of the keypoints, the occlusion information of the source image and an attention weight;

weighting the local motion feature information to acquire weighted motion feature information of the keypoints; and

fusing the weighted motion feature information to acquire the attitude information of the first object in the target image, and using the attitude information, the occlusion information and the attention weight as the motion information corresponding to the keypoints.

8 . An electronic device, comprising a processor and a memory, the memory storing an application program, the processor being configured to run the application program in the memory to perform an image processing method including:

acquiring a target image comprising a first object for motion transfer and a plurality of source images corresponding to the target image, wherein each of the plurality of source images includes a same object, which is a second object different from the first object in the target image;

for each source image of the plurality of source images:

performing multi-dimensional feature extraction on the source image and the target image to acquire keypoint feature information of corresponding keypoints in the source image and the target image, and appearance feature information corresponding to the source image, the keypoint feature information comprising keypoint perspective information;

performing perspective transformation on the keypoints according to the keypoint perspective information to acquire optic flow information of the keypoints, the optic flow information indicating transformation information of the keypoints from the source image to the target image;

determining motion information corresponding to the keypoints based on the optic flow information and the keypoint feature information, wherein the motion information includes attitude information and occlusion information of the source image;

fusing the attitude information, the occlusion information and the appearance feature information to acquire initial image information corresponding to the source image; and

fusing the initial image information corresponding to the plurality of source images to acquire a processed image based on a weighted combination using attention weights associated with the plurality of source images respectively, the processed image being an image acquired after transferring motion of the first object in the target image to the second object in the plurality of source images.

9 . The electronic device according to claim 8 , wherein the keypoint feature information further comprises keypoint position information and keypoint deformation information, and the performing perspective transformation on the keypoints according to the keypoint perspective information to acquire optic flow information of the keypoints comprises:

fusing the keypoint perspective information and the keypoint deformation information to acquire a perspective transformation parameter; and

performing perspective transformation on the keypoints according to the perspective transformation parameter and the keypoint position information to acquire the optic flow information of the keypoints.

10 . The electronic device according to claim 9 , wherein the fusing the keypoint perspective information and the keypoint deformation information to acquire a perspective transformation parameter comprises:

respectively constructing a first initial perspective transformation parameter of the source image and a second initial perspective transformation parameter of the target image based on the keypoint perspective information and the keypoint deformation information; and

fusing the first initial perspective transformation parameter and the second initial perspective transformation parameter to acquire the perspective transformation parameter.

11 . The electronic device according to claim 9 , wherein the performing perspective transformation on the keypoints according to the perspective transformation parameter and the keypoint position information to acquire the optic flow information of the keypoint comprises:

acquiring unit optic flow information of the keypoints, the unit optic flow information being optic flow information corresponding to a warping operation between the source image and the target image;

respectively screening out first position information of the keypoints in the source image and second position information of the corresponding keypoints in the target image from the keypoint position information;

performing the perspective transformation on the keypoints according to the unit optic flow information, the second position information and the perspective transformation parameter to acquire initial optic flow information of the keypoints; and

fusing the first position information and the initial optic flow information to acquire the optic flow information of the keypoints.

12 . The electronic device according to claim 11 , wherein the determining motion information corresponding to the keypoints based on the optic flow information and the keypoint feature information comprises:

performing the warping operation on the source image according to the optic flow information to acquire at least one warped source image;

fusing the unit optic flow information, the first position information, the second position information and the warped source image to acquire fused source image information; and

recognizing the motion information corresponding to the keypoints from the fused source image information.

13 . The electronic device according to claim 12 , wherein the fusing the unit optic flow information, the first position information, the second position information and the warped source image to acquire fused source image information comprises:

determining a first position difference value of the first position information and the unit optic flow information;

determining a second position difference value of the second position information and the unit optic flow information;

determining transformation position information of the keypoints according to the first position difference value and the second position difference value; and

stitching the transformation position information and the warped source image to acquire the fused source image information.

14 . The electronic device according to claim 12 , wherein the recognizing motion information corresponding to the keypoints from the fused source image information comprises:

performing multi-dimensional feature extraction on the fused source image information by using a trained processing model to acquire local motion feature information of the keypoints, the occlusion information of the source image and an attention weight;

weighting the local motion feature information to acquire weighted motion feature information of the keypoints; and

fusing the weighted motion feature information to acquire the attitude information of the first object in the target image, and using the attitude information, the occlusion information and the attention weight as the motion information corresponding to the keypoints.

15 . A non-transitory computer readable storage medium, storing a plurality of instructions, the instructions being suitable for being loaded by a processor of an electronic device, causing the electronic device to perform an image processing method including:

acquiring a target image comprising a first object for motion transfer and a plurality of source images corresponding to the target image, wherein each of the plurality of source images includes a same object, which is a second object different from the first object in the target image;

for each source image of the plurality of source images:

performing multi-dimensional feature extraction on the source image and the target image to acquire keypoint feature information of corresponding keypoints in the source image and the target image, and appearance feature information corresponding to the source image, the keypoint feature information comprising keypoint perspective information;

performing perspective transformation on the keypoints according to the keypoint perspective information to acquire optic flow information of the keypoints, the optic flow information indicating transformation information of the keypoints from the source image to the target image;

determining motion information corresponding to the keypoints based on the optic flow information and the keypoint feature information, wherein the motion information includes attitude information and occlusion information of the source image:

fusing the attitude information, the occlusion information and the appearance feature information to acquire initial image information corresponding to the source image; and

fusing the initial image information corresponding to the plurality of source images to acquire a processed image based on a weighted combination using attention weights associated with the plurality of source images respectively, the processed image being an image acquired after transferring motion of the first object in the target image to the second object in the plurality of source images.

16 . The non-transitory computer readable storage medium according to claim 15 , wherein the keypoint feature information further comprises keypoint position information and keypoint deformation information, and the performing perspective transformation on the keypoints according to the keypoint perspective information to acquire optic flow information of the keypoints comprises:

fusing the keypoint perspective information and the keypoint deformation information to acquire a perspective transformation parameter; and

performing perspective transformation on the keypoints according to the perspective transformation parameter and the keypoint position information to acquire the optic flow information of the keypoints.

17 . The non-transitory computer readable storage medium according to claim 16 , wherein the fusing the keypoint perspective information and the keypoint deformation information to acquire a perspective transformation parameter comprises:

respectively constructing a first initial perspective transformation parameter of the source image and a second initial perspective transformation parameter of the target image based on the keypoint perspective information and the keypoint deformation information; and

fusing the first initial perspective transformation parameter and the second initial perspective transformation parameter to acquire the perspective transformation parameter.

18 . The non-transitory computer readable storage medium according to claim 16 , wherein the performing perspective transformation on the keypoints according to the perspective transformation parameter and the keypoint position information to acquire the optic flow information of the keypoint comprises:

acquiring unit optic flow information of the keypoints, the unit optic flow information being optic flow information corresponding to a warping operation between the source image and the target image;

respectively screening out first position information of the keypoints in the source image and second position information of the corresponding keypoints in the target image from the keypoint position information;

performing the perspective transformation on the keypoints according to the unit optic flow information, the second position information and the perspective transformation parameter to acquire initial optic flow information of the keypoints; and

fusing the first position information and the initial optic flow information to acquire the optic flow information of the keypoints.

19 . The non-transitory computer readable storage medium according to claim 18 , wherein the determining motion information corresponding to the keypoints based on the optic flow information and the keypoint feature information comprises:

performing the warping operation on the source image according to the optic flow information to acquire at least one warped source image;

fusing the unit optic flow information, the first position information, the second position information and the warped source image to acquire fused source image information; and

recognizing the motion information corresponding to the keypoint from the fused source image information.

20 . The non-transitory computer readable storage medium according to claim 19 , wherein the fusing the unit optic flow information, the first position information, the second position information and the warped source image to acquire fused source image information comprises:

determining a first position difference value of the first position information and the unit optic flow information;

determining a second position difference value of the second position information and the unit optic flow information;

determining transformation position information of the keypoints according to the first position difference value and the second position difference value; and

stitching the transformation position information and the warped source image to acquire the fused source image information.