Granted Patent B2

US 12,608,813 · App. 18/251,228 · Granted Apr 21, 2026

Inventors: Tao Kong (Beijing, CN); Ya Jing (Beijing, CN); Lei Li (Beijing, CN)

Assignee: BEIJING YOUZHUJU NETWORK TECHNOLOGY CO., LTD.

G06T7/11G06T3/40G06T2207/20221

Abstract

Claims (64)

1 . An image segmentation method, comprising:

fusing a visual feature corresponding to an original image with a text feature corresponding to a description language to obtain a multimodal feature, wherein the description language is used for specifying a target object to be segmented in the original image;

determining a visual region of the target object according to an image corresponding to the multimodal feature and recording an image corresponding to the visual region as a response heat map; and

determining a segmentation result of the target object according to the image corresponding to the multimodal feature and the response heat map,

wherein determining the visual region of the target object according to the image corresponding to the multimodal feature comprises:

performing correlation filtering on the image corresponding to the multimodal feature to obtain the visual region of the target object.

2 . The method of claim 1 , wherein performing the correlation filtering on the image corresponding to the multimodal feature to obtain the visual region of the target object comprises:

determining a convolution kernel according to the text feature; and

performing a convolution operation on the image corresponding to the multimodal feature according to the convolution kernel to obtain the visual region of the target object.

3 . The method of claim 2 , wherein determining the segmentation result of the target object according to the image corresponding to the multimodal feature and the response heat map comprises:

inputting the image corresponding to the multimodal feature and the response heat map into a target image segmentation model and acquiring an output result of the target image segmentation model as the segmentation result of the target object.

4 . The method of claim 1 , wherein the visual feature comprises visual features extracted from the original image under at least two resolutions separately.

5 . The method of claim 4 , wherein fusing the visual feature corresponding to the original image with the text feature corresponding to the description language to obtain the multimodal feature comprises:

sorting at least two visual features in an order of sizes of the resolutions to obtain a sorting result; and mapping, through a mapping matrix, the text feature to a feature space where a first visual feature corresponding to a first resolution in the sorting result is located, wherein the first resolution has a smallest value;

splicing the first visual feature with the mapped text feature to obtain a first spliced feature; and

upsampling the first spliced feature, splicing the upsampled first spliced feature with a second visual feature corresponding to a second resolution in the sorting result to obtain a second spliced feature, and cyclically performing upsampling and splicing operations until an upsampled spliced feature is spliced with a visual feature having a largest resolution in the sorting result to obtain the multimodal feature, wherein the second resolution is greater than the first resolution and a smallest resolution except the first resolution.

6 . The method of claim 5 , wherein determining the segmentation result of the target object according to the image corresponding to the multimodal feature and the response heat map comprises:

inputting the image corresponding to the multimodal feature and the response heat map into a target image segmentation model and acquiring an output result of the target image segmentation model as the segmentation result of the target object.

7 . The method of claim 4 , wherein determining the segmentation result of the target object according to the image corresponding to the multimodal feature and the response heat map comprises:

inputting the image corresponding to the multimodal feature and the response heat map into a target image segmentation model and acquiring an output result of the target image segmentation model as the segmentation result of the target object.

8 . The method of claim 1 , wherein determining the segmentation result of the target object according to the image corresponding to the multimodal feature and the response heat map comprises:

inputting the image corresponding to the multimodal feature and the response heat map into a target image segmentation model and acquiring an output result of the target image segmentation model as the segmentation result of the target object.

9 . The method of claim 8 , wherein a training process of the target image segmentation model comprises:

acquiring a sample image and a sample description language and extracting a sample visual feature of the sample image and a sample text feature of the sample description language, wherein the sample description language is used for specifying a sample target object to be segmented in the sample image;

fusing the sample visual feature with the sample text feature to obtain a sample multimodal feature;

determining a sample visual region of the sample target object according to an image corresponding to the sample multimodal feature and recording an image corresponding to the sample visual region as a sample response heat map; and

training an initial image segmentation model according to the image corresponding to the sample multimodal feature and the sample response heat map to obtain the target image segmentation model.

10 . The method of claim 9 , wherein training the initial image segmentation model according to the image corresponding to the sample multimodal feature and the sample response heat map to obtain the target image segmentation model comprises:

inputting the image corresponding to the sample multimodal feature and the sample response heat map into the initial image segmentation model to obtain a plurality of first convolution results of the image corresponding to the sample multimodal feature and the sample response heat map, wherein the plurality of first convolution results are obtained through a first convolution operation performed on the image corresponding to the sample multimodal feature and the sample response heat map at different sampling rates;

splicing the plurality of first convolution results to obtain a spliced result;

performing a second convolution operation on the spliced result to obtain a second convolution result;

upsampling the second convolution result to obtain a sample segmentation result;

determining a loss value of the sample segmentation result relative to an actual segmentation result of the sample image; and

in a case where the loss value is less than a set threshold, stopping training the initial image segmentation model and using an image segmentation model with the loss value less than the set threshold as the target image segmentation model; and in a case where the loss value is not less than the set threshold, continuing to train the initial image segmentation model until the loss value is less than the set threshold.

11 . An electronic device, comprising:

at least one processor; and

a memory configured to store at least one program;

wherein the at least one program, when executed by the at least one processor, causes the at least one processor to perform the following steps:

fusing a visual feature corresponding to an original image with a text feature corresponding to a description language to obtain a multimodal feature, wherein the description language is used for specifying a target object to be segmented in the original image;

determining a visual region of the target object according to an image corresponding to the multimodal feature and recording an image corresponding to the visual region as a response heat map; and

determining a segmentation result of the target object according to the image corresponding to the multimodal feature and the response heat map,

wherein determining the visual region of the target object according to the image corresponding to the multimodal feature comprises:

performing correlation filtering on the image corresponding to the multimodal feature to obtain the visual region of the target object.

12 . The device of claim 11 , wherein performing the correlation filtering on the image corresponding to the multimodal feature to obtain the visual region of the target object comprises:

determining a convolution kernel according to the text feature; and

performing a convolution operation on the image corresponding to the multimodal feature according to the convolution kernel to obtain the visual region of the target object.

13 . The device of claim 11 , wherein the visual feature comprises visual features extracted from the original image under at least two resolutions separately.

14 . The device of claim 13 , wherein fusing the visual feature corresponding to the original image with the text feature corresponding to the description language to obtain the multimodal feature comprises:

sorting at least two visual features in an order of sizes of the resolutions to obtain a sorting result; and mapping, through a mapping matrix, the text feature to a feature space where a first visual feature corresponding to a first resolution in the sorting result is located, wherein the first resolution has a smallest value;

splicing the first visual feature with the mapped text feature to obtain a first spliced feature; and

upsampling the first spliced feature, splicing the upsampled first spliced feature with a second visual feature corresponding to a second resolution in the sorting result to obtain a second spliced feature, and cyclically performing upsampling and splicing operations until an upsampled spliced feature is spliced with a visual feature having a largest resolution in the sorting result to obtain the multimodal feature, wherein the second resolution is greater than the first resolution and a smallest resolution except the first resolution.

15 . The device of claim 11 , wherein determining the segmentation result of the target object according to the image corresponding to the multimodal feature and the response heat map comprises:

inputting the image corresponding to the multimodal feature and the response heat map into a target image segmentation model and acquiring an output result of the target image segmentation model as the segmentation result of the target object.

16 . The device of claim 15 , wherein a training process of the target image segmentation model comprises:

acquiring a sample image and a sample description language and extracting a sample visual feature of the sample image and a sample text feature of the sample description language, wherein the sample description language is used for specifying a sample target object to be segmented in the sample image;

fusing the sample visual feature with the sample text feature to obtain a sample multimodal feature;

determining a sample visual region of the sample target object according to an image corresponding to the sample multimodal feature and recording an image corresponding to the sample visual region as a sample response heat map; and

training an initial image segmentation model according to the image corresponding to the sample multimodal feature and the sample response heat map to obtain the target image segmentation model.

17 . A non-transitory computer-readable storage medium, which is configured to store a computer program which, when executed by a processor, causes the processor to perform the following steps:

fusing a visual feature corresponding to an original image with a text feature corresponding to a description language to obtain a multimodal feature, wherein the description language is used for specifying a target object to be segmented in the original image;

determining a visual region of the target object according to an image corresponding to the multimodal feature and recording an image corresponding to the visual region as a response heat map; and

determining a segmentation result of the target object according to the image corresponding to the multimodal feature and the response heat map,

wherein determining the visual region of the target object according to the image corresponding to the multimodal feature comprises:

performing correlation filtering on the image corresponding to the multimodal feature to obtain the visual region of the target object.

Priority Claims (1)

Continuity (1)

References Cited (46)