IP Library Granted Patent US 11,276,207
Granted Patent B2
US 11,276,207 · App. 16/880,883 · Granted Mar 15, 2022

Image processing method, storage medium, and computer device

Inventors: Minjun Li (Shenzhen, CN); Haozhi Huang (Shenzhen, CN); Lin Ma (Shenzhen, CN); Wei Liu (Shenzhen, CN); Yugang Jiang (Shenzhen, CN)
Assignee: TENCENT TECHNOLOGY (SHENZHEN) COMPANY LIMITED
G06T11/001G06T3/40G06T9/00G06T2207/20081
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,276,207
App. No.
16/880,883
Granted
Mar 15, 2022
Kind
B2
Abstract

An image processing method for a computer device. The method includes obtaining a to-be-processed image belonging to a first image category; inputting the to-be-processed image into a first stage image conversion model, to obtain a first intermediate image; and converting the first intermediate image into a second intermediate image through a second stage image conversion model. The method also includes determining a first weight matrix corresponding to the first intermediate image; determining a second weight matrix corresponding to the second intermediate image; and fusing the first intermediate image and the second intermediate image according to the corresponding first weight matrix and second weight matrix, to obtain a target image corresponding to the to-be-processed image and belonging to a second image category. A sum of the first weight matrix and the second weight matrix being a preset matrix.

Claims (118)

1. An image processing method for a computer device, comprising:

obtaining a to-be-processed image belonging to a first image category;

inputting the to-be-processed image into a first stage image conversion model, to obtain a first intermediate image;

converting the first intermediate image into a second intermediate image through a second stage image conversion model;

determining a first weight matrix corresponding to the first intermediate image and a second weight matrix corresponding to the second intermediate image, by inputting the to-be-processed image, the first intermediate image and the second intermediate image jointly into a first weight prediction model; and

fusing the first intermediate image and the second intermediate image according to the corresponding first weight matrix and second weight matrix, to obtain a target image corresponding to the to-be-processed image and belonging to a second image category, wherein a sum of the first weight matrix and the second weight matrix being a preset matrix.

2. The method according to claim 1 , wherein:

the inputting the to-be-processed image into a first stage image conversion model, to obtain a first intermediate image comprises:

performing down-sampling on the to-be-processed image, to obtain a compressed image with a reduced image size; and

inputting the compressed image into the first stage image conversion model, to output the first intermediate image; and

the converting the first intermediate image into a second intermediate image through a second stage image conversion model comprises:

performing up-sampling on the first intermediate image, to obtain an enlarged image processed image; and

inputting the enlarged image into the second stage image conversion model, to output the second intermediate image.

3. The method according to claim 1 , wherein the fusing the first intermediate image and the second intermediate image according to the corresponding first weight matrix and second weight matrix, to obtain a target image corresponding to the to-be-processed image and belonging to a second image category comprises:

multiplying pixel values of the first intermediate image by matrix elements of the first weight matrix in bit, to obtain a first target image belonging to the second image category;

multiplying pixel values of the second intermediate image by matrix elements of the second weight matrix in bit, to obtain a second target image belonging to the second image category; and

obtaining, according to the first target image and the second target image, the target image corresponding to the to-be-processed image and belonging to the second image category.

4. The method according to claim 1 , further comprising:

obtaining a first sample belonging to the first image category and a second sample belonging to the second image category;

making the first sample pass through the first stage image conversion model and a first stage image reverse conversion model, to obtain a first sample single stage recovery image;

making the second sample pass through the first stage image reverse conversion model and the first stage image conversion model, to obtain a second sample single stage recovery image; and

adjusting the first stage image conversion model and the first stage image reverse conversion model according to a difference between the first sample and the first sample single stage recovery image, and a difference between the second sample and the second sample single stage recovery image.

5. The method according to claim 4 , further comprising:

obtaining a first sample single stage conversion image outputted by the first stage image conversion model after the first sample passes through the first stage image conversion model;

obtaining a second sample single stage conversion image outputted by the first stage image reverse conversion model after the second sample passes through the first stage image reverse conversion model;

respectively inputting the first sample and the second sample single stage conversion image into a first stage image reverse conversion discrimination model, to respectively obtain a discrimination confidence of the first sample and a discrimination confidence of the second sample single stage conversion image; and

respectively inputting the second sample and the first sample single stage conversion image into a first stage image conversion discrimination model, to respectively obtain a discrimination confidence of the second sample and a discrimination confidence of the first sample single stage conversion image.

6. The method according to claim 4 , further comprising:

making the first sample pass through the first stage image conversion model and the second stage image conversion model, to obtain a first sample first-order conversion image outputted by the first stage image conversion model, and a first sample second-order conversion image outputted by the second stage image conversion model;

obtaining, according to the first sample first-order conversion image and the first sample second-order conversion image, a first sample conversion image corresponding to the first sample and belonging to the second image category;

making the first sample conversion image pass through the first stage image reverse conversion model and a second stage image reverse conversion model, to obtain a first sample first-order recovery image outputted by the first stage image reverse conversion model, and a first sample second-order recovery image outputted by the second stage image reverse conversion model;

obtaining, according to the first sample first-order recovery image and the first sample second-order recovery image, a first sample recovery image corresponding to the first sample and belonging to the first image category;

making the second sample pass through the first stage image reverse conversion model and the second stage image reverse conversion model, to obtain a second sample first-order conversion image outputted by the first stage image reverse conversion model, and a second sample second-order conversion image outputted by the second stage image reverse conversion model;

obtaining, according to the second sample first-order conversion image and the second sample second-order conversion image, a second sample conversion image corresponding to the second sample and belonging to the first image category;

making the second sample conversion image pass through the first stage image conversion model and the second stage image conversion model, to obtain a second sample first-order recovery image outputted by the first stage image conversion model, and a second sample second-order recovery image outputted by the second stage image conversion model;

obtaining, according to the second sample first-order recovery image and the second sample second-order recovery image, a second sample recovery image corresponding to the second sample and belonging to the second image category; and

adjusting the second stage image conversion model and the second stage image reverse conversion model according to a difference between the first sample and the first sample recovery image, and a difference between the second sample and the second sample recovery image.

7. The method according to claim 6 , further comprising:

respectively inputting the first sample and the second sample conversion image into a second stage image reverse conversion discrimination model, to respectively obtain a discrimination confidence of the first sample and a discrimination confidence of the second sample conversion image; and

respectively inputting the second sample and the first sample conversion image into a second stage image conversion discrimination model, to respectively obtain a discrimination confidence of the second sample and a discrimination confidence of the first sample conversion image.

8. The method according to claim 6 , wherein:

the obtaining, according to the first sample first-order conversion image and the first sample second-order conversion image, a first sample conversion image corresponding to the first sample and belonging to the second image category comprises:

inputting the first sample, the first sample first-order conversion image, and the first sample second-order conversion image jointly into a first weight prediction model, to obtain a weight matrix corresponding to the first sample second-order conversion image;

obtaining a weight matrix corresponding to the first sample first-order conversion image according to the weight matrix; and

fusing the first sample first-order conversion image and the first sample second-order conversion image according to the respective corresponding weight matrices, to obtain the first sample conversion image corresponding to the first sample and belonging to the second image category; and

the obtaining, according to the first sample first-order recovery image and the first sample second-order recovery image, a first sample recovery image corresponding to the first sample and belonging to the first image category comprises:

inputting the first sample conversion image, the first sample first-order recovery image, and the first sample second-order recovery image jointly into a second weight prediction model, to obtain a weight matrix corresponding to the first sample second-order recovery image;

obtaining a weight matrix corresponding to the first sample first-order recovery image according to the weight matrix; and fusing the first sample first-order recovery image and the first sample second-order recovery image according to the respective corresponding weight matrices, to obtain the first sample recovery image corresponding to the first sample and belonging to the first image category.

9. A computer device, comprising: a memory storing computer-readable instructions; and a processor coupled to the memory for executing the computer-readable instructions to perform:

obtaining a to-be-processed image belonging to a first image category;

inputting the to-be-processed image into a first stage image conversion model, to obtain a first intermediate image;

converting the first intermediate image into a second intermediate image through a second stage image conversion model;

determining a first weight matrix corresponding to the first intermediate image and a second weight matrix corresponding to the second intermediate image, by inputting the to-be-processed image, the first intermediate image and the second intermediate image jointly into a first weight prediction model;

and

fusing the first intermediate image and the second intermediate image according to the corresponding first weight matrix and second weight matrix, to obtain a target image corresponding to the to-be-processed image and belonging to a second image category, wherein a sum of the first weight matrix and the second weight matrix being a preset matrix.

10. The computer device according to claim 9 , wherein:

the inputting the to-be-processed image into a first stage image conversion model, to obtain a first intermediate image comprises:

performing down-sampling on the to-be-processed image, to obtain a compressed image with a reduced image size; and

inputting the compressed image into the first stage image conversion model, to output the first intermediate image; and

the converting the first intermediate image into a second intermediate image through a second stage image conversion model comprises:

performing up-sampling on the first intermediate image, to obtain an enlarged image; and

inputting the enlarged image into the second stage image conversion model, to output the second intermediate image.

11. The computer device according to claim 10 , wherein the fusing the first intermediate image and the second intermediate image according to the corresponding first weight matrix and second weight matrix, to obtain a target image corresponding to the to-be-processed image and belonging to a second image category comprises:

multiplying pixel values of the first intermediate image by matrix elements of the first weight matrix in bit, to obtain a first target image belonging to the second image category;

multiplying pixel values of the second intermediate image by matrix elements of the second weight matrix in bit, to obtain a second target image belonging to the second image category; and

obtaining, according to the first target image and the second target image, the target image corresponding to the to-be-processed image and belonging to the second image category.

12. The computer device according to claim 9 , wherein the processor further performs:

obtaining a first sample belonging to the first image category and a second sample belonging to the second image category;

making the first sample pass through the first stage image conversion model and a first stage image reverse conversion model, to obtain a first sample single stage recovery image;

making the second sample pass through the first stage image reverse conversion model and the first stage image conversion model, to obtain a second sample single stage recovery image; and

adjusting the first stage image conversion model and the first stage image reverse conversion model according to a difference between the first sample and the first sample single stage recovery image, and a difference between the second sample and the second sample single stage recovery image.

13. The computer device according to claim 12 , wherein: the processor further performs:

obtaining a first sample single stage conversion image outputted by the first stage image conversion model after the first sample passes through the first stage image conversion model;

obtaining a second sample single stage conversion image outputted by the first stage image reverse conversion model after the second sample passes through the first stage image reverse conversion model;

respectively inputting the first sample and the second sample single stage conversion image into a first stage image reverse conversion discrimination model, to respectively obtain a discrimination confidence of the first sample and a discrimination confidence of the second sample single stage conversion image; and

respectively inputting the second sample and the first sample single stage conversion image into a first stage image conversion discrimination model, to respectively obtain a discrimination confidence of the second sample and a discrimination confidence of the first sample single stage conversion image.

14. The computer device according to claim 12 , wherein the processor further performs:

making the first sample pass through the first stage image conversion model and the second stage image conversion model, to obtain a first sample first-order conversion image outputted by the first stage image conversion model, and a first sample second-order conversion image outputted by the second stage image conversion model;

obtaining, according to the first sample first-order conversion image and the first sample second-order conversion image, a first sample conversion image corresponding to the first sample and belonging to the second image category; making the first sample conversion image pass through the first stage image reverse conversion model and a second stage image reverse conversion model, to obtain a first sample first-order recovery image outputted by the first stage image reverse conversion model, and a first sample second-order recovery image outputted by the second stage image reverse conversion model;

obtaining, according to the first sample first-order recovery image and the first sample second-order recovery image, a first sample recovery image corresponding to the first sample and belonging to the first image category;

making the second sample pass through the first stage image reverse conversion model and the second stage image reverse conversion model, to obtain a second sample first-order conversion image outputted by the first stage image reverse conversion model, and a second sample second-order conversion image outputted by the second stage image reverse conversion model;

obtaining, according to the second sample first-order conversion image and the second sample second-order conversion image, a second sample conversion image corresponding to the second sample and belonging to the first image category;

making the second sample conversion image pass through the first stage image conversion model and the second stage image conversion model, to obtain a second sample first-order recovery image outputted by the first stage image conversion model, and a second sample second-order recovery image outputted by the second stage image conversion model;

obtaining, according to the second sample first-order recovery image and the second sample second-order recovery image, a second sample recovery image corresponding to the second sample and belonging to the second image category; and

adjusting the second stage image conversion model and the second stage image reverse conversion model according to a difference between the first sample and the first sample recovery image, and a difference between the second sample and the second sample recovery image.

15. The computer device according to claim 14 , wherein: the processor further performs:

respectively inputting the first sample and the second sample conversion image into a second stage image reverse conversion discrimination model, to respectively obtain a discrimination confidence of the first sample and a discrimination confidence of the second sample conversion image; and

respectively inputting the second sample and the first sample conversion image into a second stage image conversion discrimination model, to respectively obtain a discrimination confidence of the second sample and a discrimination confidence of the first sample conversion image.

16. The computer device according to claim 14 , wherein:

the obtaining, according to the first sample first-order conversion image and the first sample second-order conversion image, a first sample conversion image corresponding to the first sample and belonging to the second image category comprises:

inputting the first sample, the first sample first-order conversion image, and the first sample second-order conversion image jointly into a first weight prediction model, to obtain a weight matrix corresponding to the first sample second-order conversion image;

obtaining a weight matrix corresponding to the first sample first-order conversion image according to the weight matrix; and

fusing the first sample first-order conversion image and the first sample second-order conversion image according to the respective corresponding weight matrices, to obtain the first sample conversion image corresponding to the first sample and belonging to the second image category; and

the obtaining, according to the first sample first-order recovery image and the first sample second-order recovery image, a first sample recovery image corresponding to the first sample and belonging to the first image category comprises:

inputting the first sample conversion image, the first sample first-order recovery image, and the first sample second-order recovery image jointly into a second weight prediction model, to obtain a weight matrix corresponding to the first sample second-order recovery image;

obtaining a weight matrix corresponding to the first sample first-order recovery image according to the weight matrix; and

fusing the first sample first-order recovery image and the first sample second-order recovery image according to the respective corresponding weight matrices, to obtain the first sample recovery image corresponding to the first sample and belonging to the first image category; and

the obtaining, according to the second sample first-order conversion image and the second sample second-order conversion image, a second sample conversion image corresponding to the second sample and belonging to the first image category comprises:

inputting the second sample, the second sample first-order conversion image, and the second sample second-order conversion image jointly into the second weight prediction model, to obtain a weight matrix corresponding to the second sample second-order conversion image;

obtaining a weight matrix corresponding to the second sample first-order conversion image according to the weight matrix; and

fusing the second sample first-order conversion image and the second sample second-order conversion image according to the respective corresponding weight matrices, to obtain the second sample conversion image corresponding to the second sample and belonging to the first image category; and

the obtaining, according to the second sample first-order recovery image and the second sample second-order recovery image, a second sample recovery image corresponding to the second sample and belonging to the second image category comprises:

inputting the second sample conversion image, the second sample first-order recovery image, and the second sample second-order recovery image jointly into the first weight prediction model, to obtain a weight matrix corresponding to the second sample second-order recovery image;

obtaining a weight matrix corresponding to the second sample first-order recovery image according to the weight matrix; and

fusing the second sample first-order recovery image and the second sample second-order recovery image according to the respective corresponding weight matrices, to obtain the second sample recovery image corresponding to the second sample and belonging to the second image category.

17. A non-transitory computer-readable storage medium storing computer program instructions executable by at least one processor to perform:

obtaining a to-be-processed image belonging to a first image category;

inputting the to-be-processed image into a first stage image conversion model, to obtain a first intermediate image;

converting the first intermediate image into a second intermediate image through a second stage image conversion model;

determining a first weight matrix corresponding to the first intermediate image and a second weight matrix corresponding to the second intermediate image, by inputting the to-be-processed image, the first intermediate image and the second intermediate image jointly into a first weight prediction model; and

fusing the first intermediate image and the second intermediate image according to the corresponding first weight matrix and second weight matrix, to obtain a target image corresponding to the to-be-processed image and belonging to a second image category, wherein a sum of the first weight matrix and the second weight matrix being a preset matrix.

18. The non-transitory computer-readable storage medium according to claim 17 , wherein:

the inputting the to-be-processed image into a first stage image conversion model, to obtain a first intermediate image comprises:

performing down-sampling on the to-be-processed image, to obtain a compressed image with a reduced image size; and

inputting the compressed image into the first stage image conversion model, to output the first intermediate image; and

the converting the first intermediate image into a second intermediate image through a second stage image conversion model comprises:

performing up-sampling on the first intermediate image, to obtain an enlarged image; and

inputting the enlarged image into the second stage image conversion model, to output the second intermediate image.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 21, 2020
From: LI, MINJUN; HUANG, HAOZHI; MA, LIN; LIU, WEI; JIANG, YUGANG
To: TENCENT TECHNOLOGY (SHENZHEN) COMPANY LIMITED
Reel/Frame 052730/0120 →
Priority Claims (1)
CN 201810078544.8 · Jan 26, 2018 · national
Continuity (2)
Continuation PCTCN2019072491 · Jan 21, 2019
Related Publication 20200286263A1 · Sep 10, 2020
Cited By (1)
US 12,620,141