Granted Patent B2

US 12,315,113 · App. 17/395,861 · Granted May 27, 2025

Inventors: Gao Wu (Shenzhen, CN); Bingkeng Peng (Shenzhen, CN); Nianhua Xie (Shenzhen, CN)

Assignee: TENCENT TECHNOLOGY (SHENZHEN) COMPANY LIMITED

G06T5/50G06F18/21G06N3/08G06T11/00G06V10/454G06V10/82G06V20/41G06V20/46G06V20/49G06V30/19173G06V30/274

Abstract

Claims (83)

1. An image processing method, comprising:

obtaining a harmonized image, the harmonized image being harmonized with a promotional content image;

performing context feature extraction on the harmonized image using a plurality of convolutional layers in a first convolutional neural network, to obtain multi-layer context features of the harmonized image, the multi-layer context features comprise an output-layer context feature and a middle-layer context feature;

performing multi-level semantic extraction on the output-layer context feature to obtain an intermediate multi-level semantic information of an object in the harmonized image;

harmonizing the intermediate multi-level semantic information with a middle-layer context feature at a same level in a manner of feature value summation, to obtain the multi-level semantic information of the object in the harmonized image; and

performing image reconstruction based on the multi-layer context features and the multi-level semantic information, to obtain a reconstructed image.

2. The image processing method of claim 1 , wherein the performing the multi-level semantic extraction on the output-layer context feature comprises:

determining the output-layer context feature as a current extraction feature;

performing semantic extraction on the current extraction feature, to obtain target-level semantic information of the object in the harmonized image;

harmonizing the target-level semantic information with a middle-layer context feature at a same level, to obtain a target-level harmonized feature of the object in the harmonized image;

updating a current input feature to the target-level harmonized feature; and

in response to a semantic extraction termination condition being met, outputting the multi-level semantic information of the object in the harmonized image.

3. The image processing method of claim 2 , wherein the performing the multi-level semantic extraction on the output-layer context feature comprises:

in response to the semantic extraction termination condition being not met, reperforming the semantic extraction on the current extraction feature.

4. The image processing method of claim 2 , wherein the multi-layer context features comprise the output-layer context feature; and

the performing the image reconstruction based on the multi-layer context features and the multi-level semantic information, to obtain the reconstructed image comprises:

performing multi-level feature restoration on the output-layer context feature to obtain a restored feature; and

harmonizing the restored feature and semantic information at a same level, to obtain the reconstructed image.

5. The image processing method of claim 4 , wherein the multi-layer context features comprise the output-layer context feature and intermediate context features with a plurality of levels, and the performing the image reconstruction based on the multi-layer context features and the multi-level semantic information, to obtain the reconstructed image comprises:

determining the output-layer context feature as a current restoration feature;

performing feature restoration on the current restoration feature, to obtain a restored feature;

harmonizing the restored feature and semantic information at a same level, to obtain a harmonized feature;

updating the current restoration feature to the harmonized feature; and

in response to a reconstruction termination condition being met, outputting the current restoration feature to obtain the reconstructed image.

6. The image processing method of claim 5 , wherein the performing the image reconstruction based on the multi-layer context features and the multi-level semantic information, to obtain the reconstructed image comprises:

in response to a reconstruction termination condition being not met, reperforming the feature restoration on the current restoration feature.

7. The image processing method of claim 4 , wherein the performing the image reconstruction based on the multi-layer context features and the multi-level semantic information, to obtain the reconstructed image comprises:

inputting the output-layer context feature into a second convolutional neural network, the second convolutional neural network comprising a plurality of convolutional layers;

performing feature restoration on a feature outputted by a previous convolutional layer using a convolutional layer in the second convolutional neural network, to obtain a restored feature;

harmonizing the restored feature with semantic information at a same level, and

inputting the harmonized feature into a next convolutional layer to obtain the reconstructed image.

8. The image processing method of claim 1 , wherein the first convolutional neural network comprises an encoding module and a decoding module, the encoding module comprising a plurality of encoding convolutional layers, and the decoding module comprising a plurality of decoding convolutional layers in skip connections with the plurality of encoding convolutional layers respectively;

the performing the context feature extraction on the harmonized image using the plurality of convolutional layers in the first convolutional neural network comprises:

performing the context feature extraction on the harmonized image using the plurality of encoding convolutional layers in the encoding module; and

the extracting the multi-level semantic information of the object in the harmonized image based on the multi-layer context features of the harmonized image comprises:

for each of the plurality of decoding convolutional layers, performing semantic extraction on a feature outputted by a previous convolutional layer using the decoding convolutional layer; and

harmonizing an extracted semantic feature and a middle-layer context feature at a same level, the middle-layer context feature at the same level being a middle-layer context feature outputted by an encoding convolutional layer in a skip connection with the decoding convolutional layer.

9. The image processing method of claim 1 , wherein the obtaining a harmonized image comprises:

splitting a video to obtain at least one video frame;

determining a target video frame in the at least one video frame;

harmonizing the promotional content image to a harmonized region of the target video frame, to obtain a harmonized video frame; and

using the harmonized video frame as the harmonized image.

10. The image processing method of claim 1 , further comprising:

determining a harmonized region in the reconstructed image; and

replacing an image of the harmonized region in the reconstructed image with the promotional content image.

11. An image processing apparatus, comprising:

a memory operable to store computer-readable instructions; and

a processor operable to read the computer-readable instructions, the processor when executing the computer-readable instructions is configured to:

obtain a harmonized image, the harmonized image being harmonized with a promotional content image;

perform context feature extraction on the harmonized image using a plurality of convolutional layers in a first convolutional neural network, to obtain multi-layer context features of the harmonized image, the multi-layer context features comprise an output-layer context feature and a middle-layer context feature;

perform multi-level semantic extraction on the output-layer context feature to obtain an intermediate multi-level semantic information of an object in the harmonized image;

harmonize the intermediate multi-level semantic information with a middle-layer context feature at a same level in a manner of feature value summation, to obtain the multi-level semantic information of the object in the harmonized image;

perform image reconstruction based on the multi-layer context features and the multi-level semantic information, to obtain a reconstructed image.

12. The image processing apparatus of claim 11 , wherein the processor is configured to:

perform the context feature extraction on the harmonized image using a plurality of convolutional layers in a first convolutional neural network, to obtain multi-layer context features of the harmonized image;

extract the multi-level semantic information of the object in the harmonized image based on the multi-layer context features of the harmonized image; and

perform the image reconstruction based on the multi-layer context features and the multi-level semantic information, to obtain the reconstructed image.

13. The image processing apparatus of claim 12 , wherein the multi-layer context features comprise an output-layer context feature and a middle-layer context feature, and the processor is configured to:

perform multi-level semantic extraction on the output-layer context feature to obtain an intermediate multi-level semantic information; and

harmonize the intermediate multi-level semantic information with a middle-layer context feature at a same level, to obtain the multi-level semantic information of the object in the harmonized image.

14. The image processing apparatus of claim 13 , wherein the processor is configured to:

determine the output-layer context feature as a current extraction feature;

perform semantic extraction on the current extraction feature, to obtain target-level semantic information of the object in the harmonized image;

harmonize the target-level semantic information with a middle-layer context feature at a same level, to obtain a target-level harmonized feature of the object in the harmonized image;

update a current input feature to the target-level harmonized feature; and

in response to a semantic extraction termination condition being met, output the multi-level semantic information of the object in the harmonized image.

15. The image processing apparatus of claim 14 , wherein the processor is configured to:

in response to the semantic extraction termination condition being not met, reperform the semantic extraction on the current extraction feature.

16. The image processing apparatus of claim 14 , wherein the multi-layer context features comprise the output-layer context feature, and the processor is configured to:

perform multi-level feature restoration on the output-layer context feature to obtain a restored feature; and

harmonize the restored feature and semantic information at a same level, to obtain the reconstructed image.

17. The image processing apparatus of claim 16 , wherein the multi-layer context features comprise the output-layer context feature and intermediate context features with a plurality of layers, and the processor is configured to:

determine the output-layer context feature as a current restoration feature;

perform feature restoration on the current restoration feature, to obtain a restored feature;

harmonize the restored feature and semantic information at a same level, to obtain a harmonized feature;

update the current restoration feature to the harmonized feature; and

in response to a reconstruction termination condition being met, output the current restoration feature to obtain the reconstructed image.

18. A non-transitory computer-readable storage medium, having processor executable instructions stored thereon for causing a processor to:

obtain a harmonized image, the harmonized image being harmonized with a promotional content image;

perform context feature extraction on the harmonized image using a plurality of convolutional layers in a first convolutional neural network, to obtain multi-layer context features of the harmonized image, the multi-layer context features comprise an output-layer context feature and a middle-layer context feature;

perform multi-level semantic extraction on the output-layer context feature to obtain an intermediate multi-level semantic information of an object in the harmonized image;

harmonize the intermediate multi-level semantic information with a middle-layer context feature at a same level in a manner of feature value summation, to obtain the multi-level semantic information of the object in the harmonized image; and

perform image reconstruction based on the multi-layer context features and the multi-level semantic information, to obtain a reconstructed image.

Assignments (1)

Priority Claims (1)

Continuity (2)

References Cited (17)