Granted Patent B2

US 12,586,608 · App. 18/937,679 · Granted Mar 24, 2026

Inventors: Nathan James Frey (Venice, CA); Zheng Sun (Sunnyvale, CA)

Assignee: Google LLC

G11B27/031G06T7/20G06V20/41G06T2207/10016

Abstract

Claims (66)

1 . A method performed by one or more computers, the method comprising:

receiving an input video comprising a sequence of video frames;

processing the input video to generate tracking data that identifies and tracks visual locations of one or more instances of at least one target object in the input video;

generating a set of multiple sub-videos based on the input video and the tracking data;

determining a respective score for each sub-video in the set of multiple sub-videos that characterizes a quality of the sub-video;

selecting a proper subset of the set of multiple sub-videos based on the scores;

generating an output video that comprises the proper subset of sub-videos; and

causing presentation of the output video at a user interface of a computing device.

2 . The method of claim 1 , wherein generating the set of multiple sub-videos based on the input video and the tracking data comprises, for each sub-video:

generating a respective sequence of sub-video frames that are each extracted from a respective video frame of the input video.

3 . The method of claim 2 , wherein for each sub-video, each of the sub-video frames includes a depiction of the at least one target object and excludes at least some content of the respective video frame that is outside of a bounding box that encloses the target object in each of the sub-video frames.

4 . The method of claim 1 , wherein each of the sub-videos has a duration that is shorter than the input video.

5 . The method of claim 1 , wherein processing the input video to generate tracking data that identifies and tracks visual locations of one or more instances of at least one target object in the input video comprises:

processing each video frame in the sequence of video frames of the input video using an object detection neural network to generate, for each of a plurality of the video frames, a respective object detection output that defines one or more bounding boxes that each enclose a respective object in the video frame; and

generating the tracking data using the object detection outputs for the video frames in the sequence of video frames of the input video.

6 . The method of claim 5 , wherein generating the tracking data using the object detection outputs for the video frames in the sequence of video frames in the input video comprises:

determining a similarity measure between a first bounding box in a first video frame and a second bounding box in a second video frame satisfies a threshold; and

in response, determining that the first bounding box and the second bounding box are included in a same target object trajectory that tracks a position of a target object from the first video frame to the second video frame.

7 . The method of claim 6 , wherein determining the similarity measure between the first bounding box in the first video frame and the second bounding box in the second video frame comprises:

determining an overlap between the first bounding and the second bounding box; and

determining the similarity measure based at least in part on the overlap between the first bounding box and the second bounding box.

8 . The method of claim 6 , wherein determining the similarity measure between the first bounding box in the first video frame and the second bounding box in the second video frame comprises:

determining a visual similarity between respective video frame content enclosed by the first bounding box and the second bounding box; and

determining the similarity measure based at least in part on the visual similarity between the respective video frame content enclosed by the first bounding box and the second bounding box.

9 . The method of claim 1 , wherein for each sub-video in the set of multiple sub-videos, determining the score for the sub-video comprises:

determining the score for the sub-video based at least in part on an amount of motion in the sub-video.

10 . The method of claim 1 , wherein for each sub-video in the set of multiple sub-videos, determining the score for the sub-video comprises:

determining the score for the sub-video based at least in part on a sharpness of the sub-video.

11 . The method of claim 1 , wherein causing presentation of the output video at the user interface of the computing device comprises:

receiving a request for a digital component;

determining that a digital component that includes the output video is responsive to the request; and

providing the digital component that includes the output video in response to the request to be presented alongside search results or on a third-party webpage.

12 . A system comprising:

one or more computers; and

one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:

receiving an input video comprising a sequence of video frames;

processing the input video to generate tracking data that identifies and tracks visual locations of one or more instances of at least one target object in the input video;

generating a set of multiple sub-videos based on the input video and the tracking data;

determining a respective score for each sub-video in the set of multiple sub-videos that characterizes a quality of the sub-video;

selecting a proper subset of the set of multiple sub-videos based on the scores;

generating an output video that comprises the proper subset of sub-videos; and

causing presentation of the output video at a user interface of a computing device.

13 . One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

receiving an input video comprising a sequence of video frames;

processing the input video to generate tracking data that identifies and tracks visual locations of one or more instances of at least one target object in the input video;

generating a set of multiple sub-videos based on the input video and the tracking data;

determining a respective score for each sub-video in the set of multiple sub-videos that characterizes a quality of the sub-video;

selecting a proper subset of the set of multiple sub-videos based on the scores;

generating an output video that comprises the proper subset of sub-videos; and

causing presentation of the output video at a user interface of a computing device.

14 . The one or more non-transitory computer storage media of claim 13 , wherein generating the set of multiple sub-videos based on the input video and the tracking data comprises, for each sub-video:

generating a respective sequence of sub-video frames that are each extracted from a respective video frame of the input video.

15 . The one or more non-transitory computer storage media of claim 14 , wherein for each sub-video, each of the sub-video frames includes a depiction of the at least one target object and excludes at least some content of the respective video frame that is outside of a bounding box that encloses the target object in each of the sub-video frames.

16 . The one or more non-transitory computer storage media of claim 13 , wherein each of the sub-videos has a duration that is shorter than the input video.

17 . The one or more non-transitory computer storage media of claim 13 , wherein processing the input video to generate tracking data that identifies and tracks visual locations of one or more instances of at least one target object in the input video comprises:

processing each video frame in the sequence of video frames of the input video using an object detection neural network to generate, for each of a plurality of the video frames, a respective object detection output that defines one or more bounding boxes that each enclose a respective object in the video frame; and

generating the tracking data using the object detection outputs for the video frames in the sequence of video frames of the input video.

18 . The one or more non-transitory computer storage media of claim 17 , wherein generating the tracking data using the object detection outputs for the video frames in the sequence of video frames in the input video comprises:

determining a similarity measure between a first bounding box in a first video frame and a second bounding box in a second video frame satisfies a threshold; and

in response, determining that the first bounding box and the second bounding box are included in a same target object trajectory that tracks a position of a target object from the first video frame to the second video frame.

19 . The one or more non-transitory computer storage media of claim 18 , wherein determining the similarity measure between the first bounding box in the first video frame and the second bounding box in the second video frame comprises:

determining an overlap between the first bounding and the second bounding box; and

determining the similarity measure based at least in part on the overlap between the first bounding box and the second bounding box.

20 . The one or more non-transitory computer storage media of claim 18 , wherein determining the similarity measure between the first bounding box in the first video frame and the second bounding box in the second video frame comprises:

determining a visual similarity between respective video frame content enclosed by the first bounding box and the second bounding box; and

determining the similarity measure based at least in part on the visual similarity between the respective video frame content enclosed by the first bounding box and the second bounding box.

Assignments (1)

Continuity (3)

References Cited (32)