Granted Patent B2

US 12,288,371 · App. 17/884,607 · Granted Apr 29, 2025

Inventor: Robert Gonsalves (Wellesley, MA)

Assignee: AVID TECHNOLOGY, INC.

G06V10/25G06T5/94G06T7/11G06T7/70G06T9/00G06T2207/10016G06T2207/20081G06V2201/07

Abstract

Claims (56)

1. A method of determining semantic regions of interest within a source image, the method comprising:

receiving the source image;

using an automatic object-detection system to detect a plurality of objects within the source image;

subdividing the source image into a plurality of sub-images, each sub-image containing a portion of the source image that contains one of the detected plurality of objects;

using a trained neural network model to:

generate an image embedding for the source image; and

for each sub-image of the plurality of sub-images, generate an image embedding for the sub-image; and

for each sub-image of the plurality of sub-images:

determining a degree of similarity between the image embedding of the sub-image and the image embedding of the source image;

assigning a semantic interest to the detected object contained by the sub-image according to the determined degree of similarity between the image embedding of the sub-image corresponding to the detected object and the image embedding of the source image; and

outputting an indication of the semantic interest assigned to the detected object contained by the sub-image.

2. The method of claim 1 , wherein the automatic object-detection system is a trained neural-network model.

3. The method of claim 1 , wherein the trained neural network model that is used to generate the image embeddings is a multi-modal neural network.

4. The method of claim 1 , further comprising:

for each detected object of the plurality of detected objects, generating an object mask for the detected object;

generating an object mask image of the source image in which each detected object of the plurality of detected objects is replaced in the source image with a shaded silhouette of the object mask generated for the object; and

applying a visual indication to each shaded silhouette, wherein the visual indication is indicative of the semantic interest assigned to the detected object corresponding the object mask.

5. The method of claim 1 , wherein the indication of the semantic interest assigned to each detected object of the plurality of detected objects is used to enhance image-processing of the source image.

6. The method of claim 5 , wherein the image processing comprises image compression, and enhancing the image processing of the source image includes varying a number of bits allocated to compressing each sub-image of the plurality of sub-images in accordance with the semantic interest assigned to the detected object corresponding to the sub-image.

7. The method of claim 6 , wherein the source image is a frame of a video stream.

8. The method of claim 5 , wherein:

the image processing includes cropping a portion of the source image in order to achieve a desired aspect ratio of the source image; and

enhancing the image processing of the source image includes preferentially retaining within the cropped portion of the source image objects to which higher semantic interest have been assigned.

9. The method of claim 8 , wherein the objects that are preferentially retained within the cropped image include an object to which a maximum semantic interest has been assigned.

10. The method of claim 8 , further comprising:

selecting a subset of detected objects of the plurality of detected objects, wherein the selected subset of objects includes a set of objects to which high semantic interest has been assigned;

locating a centroid of the subset of the detected objects within the source image; and

cropping the source image such that the centroid of the subset of the detected objects within the source image is located at a center of the cropped image.

11. The method of claim 8 , wherein the source image is a frame of a video stream.

12. The method of claim 5 , wherein the image processing includes contrast enhancement, and the contrast enhancement includes boosting contrast in a region of the source image containing a detected object to which a high semantic interest has been assigned.

13. The method of claim 12 , wherein the source image is a frame of a video stream.

14. A computer program product comprising:

a non-transitory computer-readable medium with computer-readable instructions encoded thereon, wherein the computer-readable instructions, when processed by a processing device, instruct the processing device to perform a method of determining semantic regions of interest within a source image, the method comprising:

receiving the source image;

using an automatic object-detection system to detect a plurality of objects within the source image;

subdividing the source image into a plurality of sub-images, each sub-image containing a portion of the source image that contains one of the detected plurality of objects;

using a trained neural network model to:

generate an image embedding for the source image; and

for each sub-image of the plurality of sub-images, generate an image embedding for the sub-image; and

for each sub-image of the plurality of sub-images:

determining a degree of similarity between the image embedding of the sub-image and the image embedding of the source image;

assigning a semantic interest to the detected object contained by the sub-image according to the determined degree of similarity between the image embedding of the sub-image corresponding to the detected object and the image embedding of the source image; and

outputting an indication of the semantic interest assigned to the detected object contained by the sub-image.

15. A system comprising:

a memory for storing computer-readable instructions; and

a processor connected to the memory, wherein the processor, when executing the computer-readable instructions, causes the system to perform a method of determining semantic regions of interest within a source image, the method comprising:

receiving the source image;

using an automatic object-detection system to detect a plurality of objects within the source image;

subdividing the source image into a plurality of sub-images, each sub-image containing a portion of the source image that contains one of the detected plurality of objects;

using a trained neural network model to:

generate an image embedding for the source image; and

for each sub-image of the plurality of sub-images, generate an image embedding for the sub-image; and

for each sub-image of the plurality of sub-images:

determining a degree of similarity between the image embedding of the sub-image and the image embedding of the source image;

assigning a semantic interest to the detected object contained by the sub-image according to the determined degree of similarity between the image embedding of the sub-image corresponding to the detected object and the image embedding of the source image; and

outputting an indication of the semantic interest assigned to the detected object contained by the sub-image.

Assignments (1)

Continuity (1)

References Cited (30)