Method and apparatus for training image processing model, electronic device, computer-readable storage medium, and computer program product

1 . A model training method, performed by an electronic device, comprising:

obtaining a training sample set including a triplet training sample, the triplet training sample including a source image, a template image, and a true value image, wherein the source image is obtained by:

obtaining a face image collected by a terminal in a use environment of a first image processing model;

performing image augmentation processing on the face image;

determining a corresponding face position based on a processing result of the image augmentation, and intercepting a face image including a background image based on the face position; and

cropping the face image to obtain the source image;

performing face swapping on the source image and the template image through the first image processing model to obtain a first face-swapped image, the first image processing model being of a re-parameterization structure;

obtaining a second image processing model corresponding to the first image processing model, the second image processing model being a pre-trained image processing model;

calculating a fusion loss function of the first image processing model according to the second image processing model, the first face-swapped image, and the true value image, comprising:

obtaining a second face-swapped image outputted by the second image processing model, and calculating a reconstruction loss function of the first image processing model using the first face-swapped image and the second face-swapped image;

calculating a feature loss function of the first image processing model based on the first face-swapped image and the second face-swapped image;

calculating an estimation loss function of the first image processing model based on the first face-swapped image and the source image;

calculating an adversarial loss function of the first image processing model based on the first face-swapped image and the true value image; and

performing fusion processing on the reconstruction loss function, the feature loss function, the estimation loss function, and the adversarial loss function, to obtain the fusion loss function of the first image processing model; and

training the first image processing model according to the fusion loss function, and determining a model parameter of the first image processing model in response to a training convergence condition of the first image processing model being reached.

2 . The model training method according to claim 1 ,

wherein the training sample set is a first training sample set;

the method further comprising:

determining a dynamic noise threshold matching a use environment of the first image processing model; and

performing denoising processing on the first training sample set according to the dynamic noise threshold, to form a second training sample set matching the dynamic noise threshold.

3 . The model training method according to claim 1 ,

wherein the training sample set is a first training sample set;

the method further comprising:

determining a fixed noise threshold corresponding to the second image processing model; and

performing denoising processing on the first training sample set according to the fixed noise threshold, to form a second training sample set matching the fixed noise threshold.

4 . The model training method according to claim 1 , wherein calculating the reconstruction loss function of the first image processing model includes:

calculating a pixel-level difference between the first face-swapped image and the second face-swapped image; and

determining the reconstruction loss function of the first image processing model according to the pixel-level difference.

5 . The model training method according to claim 1 , wherein calculating the feature loss function of the first image processing model includes:

performing feature extraction on the first face-swapped image through a pre-trained feature extraction network, to obtain features of a plurality of levels of the first face-swapped image;

performing feature extraction on the second face-swapped image through the pre-trained feature extraction network, to obtain features of a plurality of levels of the second face-swapped image; and

determining the feature loss function of the first image processing model based on a difference between the features of the plurality of levels of the first face-swapped image and the features of the plurality of levels of the second face-swapped image.

6 . The model training method according to claim 1 , wherein calculating the estimation loss function of the first image processing model includes:

extracting a face-swapped image feature vector of the first face-swapped image;

extracting a source image feature vector of the source image; and

calculating the estimation loss function of the first image processing model using a similarity between the face-swapped image feature vector and the source image feature vector.

7 . The model training method according to claim 1 ,

the method further comprising:

obtaining a target face image and a candidate face image to be replaced, after the first image processing model is trained and deployed in a mobile terminal;

encoding the target face image and the candidate face image through an encoder network of the first image processing model, to obtain a face image vector; and

decoding the face image vector through a decoder network of the first image processing model, to obtain the second face-swapped image.

8 . An electronic device comprising:

at least one memory storing one or more computer-executable instructions; and

at least one processor configured to execute the one or more computer-executable instructions to:

obtain a training sample set including a triplet training sample, the triplet training sample including a source image, a template image, and a true value image, wherein the source image is obtained by:

obtaining a face image collected by a terminal in a use environment of a first image processing model;

performing image augmentation processing on the face image;

determining a corresponding face position based on a processing result of the image augmentation, and intercepting a face image including a background image based on the face position; and

cropping the face image to obtain the source image;

perform face swapping on the source image and the template image through the first image processing model to obtain a face-swapped image, the first image processing model being of a re-parameterization structure;

obtain a second image processing model corresponding to the first image processing model, the second image processing model being a pre-trained image processing model;

calculate a fusion loss function of the first image processing model according to the second image processing model, the first face-swapped image, and the true value image, comprising:

obtaining a second face-swapped image outputted by the second image processing model, and calculating a reconstruction loss function of the first image processing model using the first face-swapped image and the second face-swapped image;

calculating a feature loss function of the first image processing model based on the first face-swapped image and the second face-swapped image;

calculating an estimation loss function of the first image processing model based on the first face-swapped image and the source image;

calculating an adversarial loss function of the first image processing model based on the first face-swapped image and the true value image; and

performing fusion processing on the reconstruction loss function, the feature loss function, the estimation loss function, and the adversarial loss function, to obtain the fusion loss function of the first image processing model; and

train the first image processing model according to the fusion loss function, and determining a model parameter of the first image processing model in response to a training convergence condition of the first image processing model being reached.

9 . The electronic device according to claim 8 , wherein:

the training sample set is a first training sample set; and

the at least one processor is further configured to execute the one or more computer-executable instructions to:

determine a dynamic noise threshold matching a use environment of the first image processing model; and

perform denoising processing on the first training sample set according to the dynamic noise threshold, to form a second training sample set matching the dynamic noise threshold.

10 . The electronic device according to claim 8 , wherein:

the training sample set is a first training sample set;

the at least one processor is further configured to execute the one or more computer-executable instructions to:

determine a fixed noise threshold corresponding to the second image processing model; and

perform denoising processing on the first training sample set according to the fixed noise threshold, to form a second training sample set matching the fixed noise threshold.

11 . The electronic device according to claim 8 , wherein the at least one processor is further configured to execute the one or more computer-executable instructions to, when calculating the reconstruction loss function of the first image processing model:

calculate a pixel-level difference between the first face-swapped image and the second face-swapped image; and

determine the reconstruction loss function of the first image processing model according to the pixel-level difference.

12 . The electronic device according to claim 8 , wherein the at least one processor is further configured to execute the one or more computer-executable instructions to, when calculating the feature loss function of the first image processing model:

perform feature extraction on the first face-swapped image through a pre-trained feature extraction network, to obtain features of a plurality of levels of the first face-swapped image;

perform feature extraction on the second face-swapped image through the pre-trained feature extraction network, to obtain features of a plurality of levels of the second face-swapped image; and

determine the feature loss function of the first image processing model based on a difference between the features of the plurality of levels of the first face-swapped image and the features of the plurality of levels of the second face-swapped image.

13 . The electronic device according to claim 8 , wherein the at least one processor is further configured to execute the one or more computer-executable instructions to, when calculating the estimation loss function of the first image processing model:

extract a face-swapped image feature vector of the first face-swapped image;

extract a source image feature vector of the source image; and

calculate the estimation loss function of the first image processing model using a similarity between the face-swapped image feature vector and the source image feature vector.

14 . The electronic device according to claim 8 , wherein:

the at least one processor is further configured to execute the one or more computer-executable instructions to:

obtain a target face image and a candidate face image to be replaced, after the first image processing model is trained and deployed in a mobile terminal;

encode the target face image and the candidate face image through an encoder network of the first image processing model, to obtain a face image vector; and

decode the face image vector through a decoder network of the first image processing model, to obtain a second face-swapped image.

15 . A non-transitory computer-readable storage medium storing one or more computer-executable instructions that, when executed by at least one processor, cause the at least one processor to:

obtain a training sample set including a triplet training sample, the triplet training sample including a source image, a template image, and a true value image, wherein the source image is obtained by:

obtaining a face image collected by a terminal in a use environment of a first image processing model;

performing image augmentation processing on the face image;

determining a corresponding face position based on a processing result of the image augmentation, and intercepting a face image including a background image based on the face position; and

cropping the face image to obtain the source image;

perform face swapping on the source image and the template image through the first image processing model to obtain a first face-swapped image, the first image processing model being of a re-parameterization structure;

obtain a second image processing model corresponding to the first image processing model, the second image processing model being a pre-trained image processing model;

calculate a fusion loss function of the first image processing model according to the second image processing model, the first face-swapped image, and the true value image, comprising:

obtaining a second face-swapped image outputted by the second image processing model, and calculating a reconstruction loss function of the first image processing model using the first face-swapped image and the second face-swapped image;

calculating a feature loss function of the first image processing model based on the first face-swapped image and the second face-swapped image;

calculating an estimation loss function of the first image processing model based on the first face-swapped image and the source image;

calculating an adversarial loss function of the first image processing model based on the first face-swapped image and the true value image; and

performing fusion processing on the reconstruction loss function, the feature loss function, the estimation loss function, and the adversarial loss function, to obtain the fusion loss function of the first image processing model; and

train the first image processing model according to the fusion loss function, and determining a model parameter of the first image processing model in response to a training convergence condition of the first image processing model being reached.

16 . The non-transitory computer-readable storage medium according to claim 15 , wherein:

the training sample set is a first training sample set; and

the one or more computer-executable instructions, when executed by at least one processor, further cause the at least one processor to:

determine a dynamic noise threshold matching a use environment of the first image processing model; and

perform denoising processing on the first training sample set according to the dynamic noise threshold, to form a second training sample set matching the dynamic noise threshold.