Method, device, and computer program product for image processing

1 . A method comprising:

implementing a processor-based machine learning system comprising at least first and second processing paths arranged in parallel with one another, with the first processing path comprising a first model and the second processing path comprising a second model different than the first model;

acquiring a plurality of images at an input of the first processing path and a first video at an input of the second processing path, wherein the plurality of images indicate visual information of a source object from a plurality of perspectives, and the first video indicates animation of a target object;

generating, utilizing the first model in the first processing path, a three-dimensional model for the source object based on the plurality of images, by applying key image frames of the plurality of images to an input of the first model, the first model providing at an output thereof the three-dimensional model for the source object;

generating, utilizing the second model in the second processing path, a plurality of animation models for the target object based on the first video, by applying key video frames of the first video to an input of the second model, the second model providing at an output thereof the plurality of animation models for the target object; and

fusing the three-dimensional model for the source object and the plurality of animation models for the target object, at least in part by aligning one or more outputs of each of the first and second processing paths to provide an additional model different than the first and second models, to generate a second video for the source object, wherein in the second video, the target object in the first video is replaced with the source object;

wherein fusing the three-dimensional model for the source object and the plurality of animation models for the target object further comprises:

separating background information from at least a subset of the key video frames of the first video;

determining model weights of one or more of the plurality of animation models for the target object;

utilizing the model weights and the additional model to generate a two-dimensional projection; and

combining at least portions of the background information and the two-dimensional projection to generate the second video.

2 . The method according to claim 1 , wherein generating a three- dimensional model for the source object based on the plurality of images comprises:

sampling the plurality of images to obtain a plurality of key image frames;

determining a sparse point cloud for the target object based on the plurality of key image frames;

training the first model according to a preset training condition based on the sparse point cloud for the target object; and

determining the three-dimensional model using the trained first model based on the plurality of key image frames.

3 . The method according to claim 2 , wherein the preset training condition comprises:

iteratively training the first model for a first preset number of times based on the sparse point cloud of the target object; and

reducing a learning rate of the first model in response to iterative training being performed for a second preset number of times, wherein the second preset number of times is less than the first preset number of times.

4 . The method according to claim 1 , wherein generating a plurality of animation models for the target object based on the first video comprises:

sampling various video frames of the first video to obtain a plurality of key video frames;

generating one animation model for the target object using the second model based on one key video frame of the plurality of key video frames;

extracting deformation information of the target object from each of the plurality of key video frames; and

adjusting the one animation model for the target object based on the deformation information of the target object in each key video frame to obtain the animation model of the target object in each key video frame.

5 . The method according to claim 4 , wherein extracting deformation information of the target object from each of the plurality of key video frames comprises:

extracting posture information and shape information of the target object from each of the plurality of key video frames;

and adjusting the one animation model for the target object based on the deformation information of the target object in each key video frame comprises:

for each key video frame, adjusting the one animation model for the target object using the posture information and shape information of the target object extracted from the key video frame to obtain the animation model of the target object in each key video frame.

6 . The method according to claim 4 , wherein fusing the three-dimensional model for the source object and the plurality of animation models for the target object to generate a second video for the source object comprises:

determining a typical animation model from the plurality of animation models for the target object, wherein the typical animation model for the target object has a posture and shape consistent with the three-dimensional model for the source object;

aligning the three-dimensional model with the typical animation model to obtain an aligned three-dimensional model;

transferring a plurality of skinning weights of the animation models for the target object to the aligned three-dimensional model; and

generating a second video for the source object using the aligned three-dimensional model based on the plurality of skinning weights.

7 . The method according to claim 6 , wherein aligning the three-dimensional model with the typical animation model comprises:

rotating the three-dimensional model by a first angle based on a rotation vector to obtain a first three-dimensional model, wherein the first angle is an angle indicated by the rotation vector;

displacing the first three-dimensional model along a first direction by a first distance based on a displacement vector to obtain a second three-dimensional model, wherein the first direction and the first distance are a direction and a distance indicated by the displacement vector respectively;

determining a distance between the second three-dimensional model and the typical animation model; and

determining the second three-dimensional model as an aligned three-dimensional model for the source object in response to the distance being less than a preset threshold.

8 . The method according to claim 7 , wherein determining a distance between the second three-dimensional model and the typical animation model comprises:

for each point in the second three-dimensional model, determining a distance between the point and a corresponding point in the typical animation model as a first distance of the point; and

determining a sum of first distances of various points in the second three-dimensional model as the distance between the second three-dimensional model and the typical animation model.

9 . The method according to claim 6 , wherein generating a second video for the source object by using the aligned three-dimensional model based on the plurality of skinning weights comprises:

controlling the aligned three-dimensional model based on the plurality of skinning weights to obtain a three-dimensional animation for the source object; and

determining a two-dimensional projection of the three-dimensional animation for the source object from a preset perspective as the second video.

10 . The method according to claim 9 , further comprising:

determining background information based on the first video, wherein the background information does not include the target object; and

embedding the background information of the first video into the two-dimensional projection as an optimized second video for the source object.

11 . An electronic device, comprising:

at least one processor; and

memory coupled to the at least one processor and having instructions stored therein, wherein the instructions, when executed by the at least one processor, cause the electronic device to perform actions comprising:

implementing a processor-based machine learning system comprising at least first and second processing paths arranged in parallel with one another, with the first processing path comprising a first model and the second processing path comprising a second model different than the first model;

acquiring a plurality of images at an input of the first processing path and a first video at an input of the second processing path, wherein the plurality of images indicate visual information of a source object from a plurality of perspectives, and the first video indicates animation of a target object;

generating, utilizing the first model in the first processing path, a three-dimensional model for the source object based on the plurality of images, by applying key image frames of the plurality of images to an input of the first model, the first model providing at an output thereof the three- dimensional model for the source object;

generating, utilizing the second model in the second processing path, a plurality of animation models for the target object based on the first video, by applying key video frames of the first video to an input of the second model, the second model providing at an output thereof the plurality of animation models for the target object; and

fusing the three-dimensional model for the source object and the plurality of animation models for the target object, at least in part by aligning one or more outputs of each of the first and second processing paths to provide an additional model different than the first and second models, to generate a second video for the source object, wherein in the second video, the target object in the first video is replaced with the source object;

wherein fusing the three-dimensional model for the source object and the plurality of animation models for the target object further comprises:

separating background information from at least a subset of the key video frames of the first video;

determining model weights of one or more of the plurality of animation models for the target object;

utilizing the model weights and the additional model to generate a two-dimensional projection; and

combining at least portions of the background information and the two-dimensional projection to generate the second video.

12 . The electronic device according to claim 11 , wherein generating a three-dimensional model for the source object based on the plurality of images comprises:

sampling the plurality of images to obtain a plurality of key image frames;

determining a sparse point cloud for the target object based on the plurality of key image frames;

training the first model according to a preset training condition based on the sparse point cloud for the target object; and

determining the three-dimensional model using the trained first model based on the plurality of key image frames.

13 . The electronic device according to claim 12 , wherein the preset training condition comprises:

iteratively training the first model for a first preset number of times based on the sparse point cloud of the target object; and

reducing a learning rate of the first model in response to iterative training being performed for a second preset number of times, wherein the second preset number of times is less than the first preset number of times.

14 . The electronic device according to claim 11 , wherein generating a plurality of animation models for the target object based on the first video comprises:

sampling various video frames of the first video to obtain a plurality of key video frames;

generating one animation model for the target object using the second model based on one key video frame of the plurality of key video frames;

extracting deformation information of the target object from each of the plurality of key video frames; and

adjusting the one animation model for the target object based on the deformation information of the target object in each key video frame to obtain the animation model of the target object in each key video frame.

15 . The electronic device according to claim 14 , wherein extracting deformation information of the target object from each of the plurality of key video frames comprises:

extracting posture information and shape information of the target object from each of the plurality of key video frames;

and adjusting the one animation model for the target object based on the deformation information of the target object in each key video frame comprises:

for each key video frame, adjusting the one animation model for the target object using the posture information and shape information of the target object extracted from the key video frame to obtain the animation model of the target object in each key video frame.

16 . The electronic device according to claim 14 , wherein fusing the three- dimensional model for the source object and the plurality of animation models for the target object to generate a second video for the source object comprises:

determining a typical animation model from the plurality of animation models for the target object, wherein the typical animation model for the target object has a posture and shape consistent with the three-dimensional model for the source object;

aligning the three-dimensional model with the typical animation model to obtain an aligned three-dimensional model;

transferring a plurality of skinning weights of the animation models for the target object to the aligned three-dimensional model; and

generating a second video for the source object using the aligned three-dimensional model based on the plurality of skinning weights.

17 . The electronic device according to claim 16 , wherein aligning the three- dimensional model with the typical animation model comprises:

rotating the three-dimensional model by a first angle based on a rotation vector to obtain a first three-dimensional model, wherein the first angle is an angle indicated by the rotation vector;

displacing the first three-dimensional model along a first direction by a first distance based on a displacement vector to obtain a second three-dimensional model, wherein the first direction and the first distance are a direction and a distance indicated by the displacement vector respectively;

determining a distance between the second three-dimensional model and the typical animation model; and

determining the second three-dimensional model as an aligned three-dimensional model for the source object in response to the distance being less than a preset threshold.

18 . The electronic device according to claim 17 , wherein determining a distance between the second three-dimensional model and the typical animation model comprises:

for each point in the second three-dimensional model, determining a distance between the point and a corresponding point in the typical animation model as a first distance of the point; and

determining a sum of first distances of various points in the second three-dimensional model as the distance between the second three-dimensional model and the typical animation model.

19 . The electronic device according to claim 16 , wherein generating a second video for the source object using the aligned three-dimensional model based on the plurality of skinning weights comprises:

controlling the aligned three-dimensional model based on the plurality of skinning weights to obtain a three-dimensional animation for the source object; and

determining a two-dimensional projection of the three-dimensional animation for the source object from a preset perspective as the second video.

20 . A computer program product comprising a non-transitory computer-readable medium having machine-executable instructions stored therein, wherein the machine-executable instructions, when executed by a machine, cause the machine to perform actions comprising:

implementing a processor-based machine learning system comprising at least first and second processing paths arranged in parallel with one another, with the first processing path comprising a first model and the second processing path comprising a second model different than the first model;

acquiring a plurality of images at an input of the first processing path and a first video at an input of the second processing path, wherein the plurality of images indicate visual information of a source object from a plurality of perspectives, and the first video indicates animation of a target object;

generating, utilizing the first model in the first processing path, a three-dimensional model for the source object based on the plurality of images, by applying key image frames of the plurality of images to an input of the first model, the first model providing at an output thereof the three-dimensional model for the source object;

generating, utilizing the second model in the second processing path, a plurality of animation models for the target object based on the first video, by applying key video frames of the first video to an input of the second model, the second model providing at an output thereof the plurality of animation models for the target object; and

fusing the three-dimensional model for the source object and the plurality of animation models for the target object, at least in part by aligning one or more outputs of each of the first and second processing paths to provide an additional model different than the first and second models, to generate a second video for the source object, wherein in the second video, the target object in the first video is replaced with the source object;

wherein fusing the three-dimensional model for the source object and the plurality of animation models for the target object further comprises:

separating background information from at least a subset of the key video frames of the first video;

determining model weights of one or more of the plurality of animation models for the target object;

utilizing the model weights and the additional model to generate a two-dimensional projection; and

combining at least portions of the background information and the two-dimensional projection to generate the second video.