Granted Patent B2

US 12,541,829 · App. 18/142,304 · Granted Feb 3, 2026

Inventors: Jenhao Hsiao (Palo Alto, CA); Zheng Zhang (Palo Alto, CA)

Assignee: INNOPEAK TECHNOLOGY, INC.

G06T5/77G06T5/50G06T7/246G06T7/33G06T7/73G06V10/776G06V20/46G06T2207/10016

Abstract

Claims (75)

1 . A method implemented by a computer system, the method comprising: receiving a sequence of video frames and a set of masks, wherein the sequence of video frames comprises an initial video frame including an undesired object and the set of masks comprises an initial mask corresponding to the undesired object; identifying from the sequence of video frames, a subset of video frames; creating, using the set of masks, a combined mask corresponding to the undesired object;

selecting, from the sequence of video frames, a reference video frame that comprises replacement pixels corresponding to pixels in a target video frame that are associated with the combined mask when:

an overlap between the combined mask in the reference video frame and the combined mask in the target video frame is less than a first threshold; and

an optical flow metric of the combined mask in the reference video frame compared to the combined mask in the target video frame is less than a second threshold;

aligning the reference video frame with the target video frame; and replacing, in the target video frame, pixels corresponding to the combined mask with pixels corresponding to the combined mask in the reference video frame and removing the undesired object from the target video frame.

2 . The method of claim 1 , wherein receiving the set of masks comprises generating the set of masks by identifying, in each video frame of the sequence of video frames, a set of pixels that correlates with a set of pixels of the initial mask.

3 . The method of claim 1 , wherein creating the combined mask comprises:

determining a subset of the set of masks for which a stability score is greater than a third threshold; and

forming, from the subset of the set of masks, the combined mask as a union of pixels corresponding to the undesired object in the subset of the set of masks.

4 . The method of claim 1 , wherein determining the overlap comprises:

identifying, in the reference video frame, a first set of pixels that correspond to pixels in the combined mask;

locating, in the target video frame, a second set of pixels that corresponds to the first set of pixels; and

computing a stability score using the first set of pixels and the second set of pixels.

5 . The method of claim 1 , further comprising outputting the target video frame to a display device.

6 . The method of claim 1 , further comprising determining the optical flow metric by:

determining a difference between pixels that correspond to the combined mask in the reference video frame and pixels that correspond to the combined mask in the target video frame; and

computing the optical flow metric based on the difference.

7 . The method of claim 1 , wherein aligning the reference video frame with the target video frame comprises:

a) determining a plurality of feature points in the target video frame and the reference video frame;

b) sampling a subset of the plurality of feature points to provide a validation feature set;

c) performing feature matching for the validation feature set;

d) calculating an overall matching score for the validation feature set;

iteratively performing b) through d) a predetermined number of times;

selecting the validation feature set having a highest overall matching score; and

computing a homography for the selected validation feature set.

8 . The method of claim 7 , wherein sampling the subset of the plurality of feature points comprises performing a random sampling.

9 . A computer system, including: one or more processors; and

one or more non-transitory computer-storage media storing instructions that, upon execution on by the one or more processors, cause the computer system to perform operations including:

receiving a sequence of video frames and a set of masks, wherein the sequence of video frames comprises an initial video frame including an undesired object and the set of masks comprises an initial mask corresponding to the undesired object;

identifying from the sequence of video frames, a subset of video frames;

creating, using the set of masks, a combined mask corresponding to the undesired object;

selecting, from the sequence of video frames, a reference video frame that comprises replacement pixels corresponding to pixels in a target video frame that are associated with the combined mask when:

an overlap between the combined mask in the reference video frame and the combined mask in the target video frame is less than a first threshold; and

an optical flow metric of the combined mask in the reference video frame compared to the combined mask in the target video frame is less than a second threshold;

aligning the reference video frame with the target video frame; and

replacing, in the target video frame, pixels corresponding to the combined mask with pixels corresponding to the combined mask in the reference video frame and removing the undesired object from the target video frame.

10 . The computer system of claim 9 , wherein receiving the set of masks comprises generating the set of masks by identifying, in each video frame of the sequence of video frames, a set of pixels that correlates with a set of pixels of the initial mask.

11 . The computer system of claim 9 , wherein creating the combined mask comprises:

determining a subset of the set of masks for which a stability score is greater than a third threshold; and

forming, from the subset of the set of masks, the combined mask as a union of pixels corresponding to the undesired object in the subset of the set of masks.

12 . The computer system of claim 9 , wherein the operations further comprise determining an optical flow metric by:

determining a difference between pixels that correspond to the combined mask in the reference video frame and pixels that correspond to the combined mask in the target video frame; and

computing the optical flow metric based on the difference.

13 . The computer system of claim 9 , wherein aligning the reference video frame with the target video frame comprises:

a) determining a plurality of feature points in the target video frame and the reference video frame;

b) sampling a subset of the plurality of feature points to provide a validation feature set;

c) performing feature matching for the validation feature set; d) calculating an overall matching score for the validation feature set;

iteratively performing b) through d) a predetermined number of times;

selecting the validation feature set having a highest overall matching score; and

computing a homography for the selected validation feature set.

14 . The computer system of claim 9 , wherein sampling the subset of the plurality of feature points comprises performing a random sampling.

15 . One or more non-transitory computer-storage media storing instructions that, upon execution on a computer system, cause the computer system to perform operations including:

receiving a sequence of video frames and a set of masks, wherein the sequence of video frames comprises an initial video frame including an undesired object and the set of masks comprises an initial mask corresponding to the undesired object;

identifying from the sequence of video frames, a subset of video frames; creating, using the set of masks, a combined mask corresponding to the undesired object;

selecting, from the sequence of video frames, a reference video frame that comprises replacement pixels corresponding to pixels in a target video frame that are associated with the combined mask when:

an overlap between the combined mask in the reference video frame and the combined mask in the target video frame is less than a first threshold; and

an optical flow metric of the combined mask in the reference video frame compared to the combined mask in the target video frame is less than a second threshold;

aligning the reference video frame with the target video frame; and

replacing, in the target video frame, pixels corresponding to the combined mask with pixels corresponding to the combined mask in the reference video frame and removing the undesired object from the target video frame.

16 . The non-transitory computer-storage media of claim 15 , wherein receiving the set of masks comprises generating the set of masks by identifying, in each video frame of the sequence of video frames, a set of pixels that correlates with a set of pixels of the initial mask.

17 . The non-transitory computer-storage media of claim 15 , wherein creating the combined mask comprises:

determining a subset of the set of masks for which a stability score is greater than a third threshold; and

forming, from the subset of the set of masks, the combined mask as a union of pixels corresponding to the undesired object in the subset of the set of masks.

18 . The non-transitory computer-storage media of claim 15 , wherein the operations further comprise determining the optical flow metric by:

determining a difference between pixels that correspond to the combined mask in the reference video frame and pixels that correspond to the combined mask in the target video frame; and

computing the optical flow metric based on the difference.

19 . The non-transitory computer-storage media of claim 15 , wherein aligning the reference video frame with the target video frame comprises:

a) determining a plurality of feature points in the target video frame and the reference video frame;

b) sampling a subset of the plurality of feature points to provide a validation feature set;

c) performing feature matching for the validation feature set;

d) calculating an overall matching score for the validation feature set;

iteratively performing b) through d) a predetermined number of times;

selecting the validation feature set having a highest overall matching score; and

computing a homography for the selected validation feature set.

20 . The non-transitory computer-storage media of claim 19 , wherein sampling the subset of the plurality of feature points comprises performing a random sampling.

Assignments (1)

Continuity (2)

References Cited (16)