Granted Patent B2

US 12,205,355 · App. 17/558,974 · Granted Jan 21, 2025

Inventors: Yan Li (Beijing, CN); Ni Zhang (Beijing, CN)

Assignee: NEC CORPORATION

G06V10/7747G06T5/50G06T5/70G06T7/10

Abstract

Claims (60)

1. A method of image processing, comprising:

obtaining a first image and at least one second image, wherein each image of the at least one second image is a transformed image of the first image;

generating a first semantic segmentation label for the first image based on at least one semantic prediction result of the at least one second image;

obtaining a plurality of images and a first set of semantic segmentation labels for the plurality of images, wherein the plurality of images comprise the first image, and the first set of semantic segmentation labels comprise the first semantic segmentation label;

obtaining a plurality of image masks corresponding to the plurality of images, wherein each image mask is used for selecting a target area in a corresponding image of the plurality of images;

generating a second set of semantic segmentation labels of the plurality of images based on the plurality of image masks and the first set of semantic segmentation labels; and

generating a mixup image and a semantic segmentation label for the mixup image based on the plurality of images and the second set of semantic segmentation labels;

wherein obtaining a plurality of image masks corresponding to the plurality of images comprises:

determining corresponding predicted semantic categories of a plurality of pixels in the first image;

determining corresponding scores of the plurality of pixels based on corresponding predicted semantic categories of the plurality of pixels, wherein the score of each pixel indicates a degree of correlation between the pixel and a noisy area in the first image; and

determining a first image mask of the first image based on corresponding scores of the plurality of pixels, wherein the first image mask is used for selecting at least part of pixels with lower scores among the plurality of pixels.

2. The method according to claim 1 , wherein obtaining the at least one second image comprises:

obtaining the at least one second image by performing image augmentation on the first image.

3. The method according to claim 2 , wherein the image augmentation comprises at least one of: mirror transformation, rotation, scaling, cropping, translation, brightness modification, adding noise, and changing color.

4. The method according to claim 1 , further comprising:

generating, using an image semantic segmentation model, at least one semantic segmentation label or at least one semantic confidence label for the at least one second image as the at least one semantic prediction result,

wherein the at least one semantic confidence label indicates a confidence of each pixel in the at least one second image belonging to a predetermined semantic category.

5. The method according to claim 2 , wherein generating the first semantic segmentation label comprises:

performing inverse transformation corresponding to the image augmentation on the at least one semantic prediction result respectively, to obtain at least one inversely-transformed semantic prediction result; and

generating the at least one semantic segmentation label based on the at least one inversely-transformed semantic prediction result.

6. The method according to claim 1 , wherein the plurality of image masks comprise a first image mask corresponding to the first image, and generating the second set of semantic segmentation labels comprises:

generating a second semantic segmentation label for the first image by performing a masking operation on the first semantic segmentation label and the first image mask, wherein the second set of semantic segmentation labels comprise the second semantic segmentation label.

7. The method according to claim 1 , wherein generating a mixup image and a semantic segmentation label for the mixup image comprises:

calculating a weighted sum of the plurality of images based on corresponding weights of the plurality of images, to generate the mixup image; and

calculating a weighted sum of the second set of semantic segmentation labels based on corresponding weights of the plurality of images, to generate a semantic segmentation label for the mixup image.

8. The method according to claim 1 , further comprising:

obtaining training data, wherein the training data comprises at least one of:

a first training sample, comprising the first image and the first semantic segmentation label; and

a second training sample, comprising the mixup image and the semantic segmentation label for the mixup image; and

training, based on the training data, a target semantic segmentation model to generate a semantic segmentation label for an input image.

9. A computing device, comprising:

at least one processor, configured to:

obtain a first image and at least one second image, wherein each image of the at least one second image is a transformed image of the first image;

generate a first semantic segmentation label for the first image based on at least one semantic prediction result of the at least one second image;

obtain a plurality of images and a first set of semantic segmentation labels for the plurality of images, wherein the plurality of images comprise the first image, and the first set of semantic segmentation labels comprise the first semantic segmentation label;

obtain a plurality of image masks corresponding to the plurality of images, wherein each image mask is used for selecting a target area in a corresponding image of the plurality of images;

generate a second set of semantic segmentation labels of the plurality of images based on the plurality of image masks and the first set of semantic segmentation labels;

generate a mixup image and a semantic segmentation label for the mixup image based on the plurality of images and the second set of semantic segmentation labels;

calculate a weighted sum of the plurality of images based on corresponding weights of the plurality of images, to generate the mixup image; and

calculate a weighted sum of the second set of semantic segmentation labels based on corresponding weights of the plurality of images, to generate a semantic segmentation label for the mixup image.

10. The computing device according to claim 9 , wherein the at least one processor is configured to:

obtain the at least one second image by performing image augmentation on the first image.

11. The computing device according to claim 10 , wherein the image augmentation comprises at least one of: mirror transformation, rotation, scaling, cropping, translation, brightness modification, adding noise, and changing color.

12. The computing device according to claim 9 , wherein the at least one processor is further configured to:

generate, using an image semantic segmentation model, at least one semantic segmentation label or at least one semantic confidence label for the at least one second image as the at least one semantic prediction result,

wherein the at least one semantic confidence label indicates a confidence of each pixel in the at least one second image belonging to a predetermined semantic category.

13. The computing device according to claim 10 , wherein the at least one processor is configured to:

perform inverse transformation corresponding to the image augmentation on the at least one semantic prediction result respectively, to obtain at least one inversely-transformed semantic prediction result; and

generate the at least one semantic segmentation label based on the at least one inversely-transformed semantic prediction result.

14. The computing device according to claim 9 , wherein the at least one processor is configured to:

determine corresponding predicted semantic categories of a plurality of pixels in the first image;

determine corresponding scores of the plurality of pixels based on corresponding predicted semantic categories of the plurality of pixels, wherein the score of each pixel indicates a degree of correlation between the pixel and a noisy area in the first image; and

determine a first image mask of the first image based on corresponding scores of the plurality of pixels, wherein the first image mask is used for selecting at least part of pixels with lower scores among the plurality of pixels.

15. The computing device according to claim 9 , wherein the plurality of image masks comprise a first image mask corresponding to the first image, and the at least one processor is configured to:

generate a second semantic segmentation label for the first image by performing a masking operation on the first semantic segmentation label and the first image mask, wherein the second set of semantic segmentation labels comprise the second semantic segmentation label.

16. The computing device according to claim 9 , wherein the at least one processor is further configured to:

obtain training data, wherein the training data comprises at least one of:

a first training sample, comprising the first image and the first semantic segmentation label; and

a second training sample, comprising the mixup image and the semantic segmentation label for the mixup image; and

train, based on the training data, a target semantic segmentation model to generate a semantic segmentation label for an input image.

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (25)