IP Library › Granted Patent US 12705890

Granted Patent B2

US 12705890 · App. 18/361,707 · Granted Aug 11, 2026

Memory-based video object segmentation

Inventors: Joon-Young Lee (San Jose, CA); Seoung Wug Oh (San Jose, CA); John G. Nelson (Normandy Park, WA); Wujun Wang (Mercer Island, WA)

Assignee: Adobe Inc.

G06V20/41 G06V10/72

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 12705890
App. No.: 18/361,707
Filed: Jul 28, 2023
Granted: Aug 11, 2026
Kind: B2
Examiner: ALLEN, KYLA GUAN-PING TI
Art Unit: 2661
USPC: 382/100

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

Embodiments are disclosed for a method including obtaining a region of interest of a current frame of a video sequence depicting an object. The method may further include determining, by a mask propagation model, a likelihood of each pixel of the current frame being associated with the object in the region of interest of the current frame based on the region of interest of the current frame and a fixed number of previous frames of the video sequence including the object. The method may further include replacing a previous frame of the fixed number of previous frames with the current frame. The method may further include displaying the current frame of the video sequence including a masked object in the region of interest of the current frame based on the likelihood of one or more pixels of the current frame being associated with the object.

Claims (74)

1 . A method comprising:

determining a fixed number of storage units, wherein each storage unit of the fixed number of storage units stores a previous frame of a video sequence comprising a masked object;

obtaining a region of interest of a current frame of the video sequence depicting an object;

duplicating a ground truth frame masking the object based on the fixed number of storage units;

storing at least one of the ground truth frame or a duplicated ground truth frame in each storage unit;

determining, by a mask propagation model, a likelihood of each pixel of the current frame being associated with the object in the region of interest of the current frame based on the region of interest of the current frame and the masked objects in each of the previous frames stored in the fixed number of storage units;

replacing, in a storage unit of the fixed number of storage units, a previous frame of the video sequence with the current frame comprising a masked object in the region of interest; and

displaying the current frame of the video sequence including the masked object in the region of interest of the current frame based on the likelihood of one or more pixels of the current frame being associated with the object.

2 . The method of claim 1 , further comprising:

determining that the current frame is a reliable frame by comparing a reliability score associated with the current frame to a reliability threshold, wherein the reliability score is based on the likelihood of each pixel of the current frame being associated with the object in the region of interest of the current frame.

3 . The method of claim 1 , further comprising:

obtaining a region of interest of a second frame of the video sequence depicting the object;

determining, by the mask propagation model, a likelihood of each pixel of the second frame being associated with the object in the region of interest of the second frame based on the region of interest of the second frame including the object and the masked objects in each of the storage units of the fixed number of storage units;

determining that the second frame is not a reliable frame by comparing a reliability score associated with the second frame to a reliability threshold, wherein the reliability score is based on the likelihood of each pixel of the second frame being associated with the object in the region of interest of the second frame; and

discarding the second frame.

4 . The method of claim 1 , wherein duplicating a ground truth frame masking the object based on the fixed number of storage units further comprises:

obtaining the ground truth frame masking the object; and

duplicating the ground truth frame a number of times equal to the fixed number of storage units.

5 . The method of claim 4 , wherein replacing, in the storage unit of the fixed number of storage units, the previous frame of the video sequence comprising the masked object with the current frame further comprising a masked object in the region of interest comprises:

storing the ground truth frame at a first location in a memory; and

storing the previous frame at a location in the memory other than the first location in the memory.

6 . The method of claim 1 , further comprising:

determining the region of interest of the current frame based on a region of interest of a previous frame including the masked object.

7 . The method of claim 1 , further comprising:

rescaling the likelihood of each pixel of the current frame being associated with the object in the region of interest of the current frame based on the likelihood of each pixel of the current frame being associated with the object in the region of interest of the current frame based on the region of interest of the current frame and the masked objects in each of the storage units of the fixed number of storage units.

8 . The method of claim 1 , wherein determining, by the mask propagation model, the likelihood of each pixel of the current frame being associated with the object in the region of interest of the current frame based on the region of interest of the current frame and the masked objects in each of the previous frames stored in the fixed number of storage units further comprises:

comparing a similarity of the region of interest of the current frame to the masked objects in each of the previous frames stored in the fixed number of storage units.

9 . A system comprising:

a memory component; and

a processing device coupled to the memory component, the processing device to perform operations comprising:

determining a fixed number of storage units, wherein each storage unit of the fixed number of storage units stores a previous frame of a video sequence comprising a masked object;

obtaining a region of interest of a current frame of the video sequence depicting an object;

duplicating a ground truth frame masking the object based on the fixed number of storage units;

storing at least one of the ground truth frame or a duplicated ground truth frame in each storage unit;

replacing, in a storage unit of the fixed number of storage units, a previous frame of the video sequence with the current frame comprising a masked object in the region of interest; and

10 . The system of claim 9 , wherein the processing device performs further operations comprising:

11 . The system of claim 9 , wherein the processing device performs further operations comprising:

obtaining a region of interest of a second frame of the video sequence depicting the object;

discarding the second frame.

12 . The system of claim 9 , wherein performing the operation of duplicating a ground truth frame masking the object based on the fixed number of storage units causes the processing device further to perform operations comprising:

obtaining the ground truth frame masking the object; and

duplicating the ground truth frame a number of times equal to the fixed number of storage units.

13 . The system of claim 12 , wherein replacing, in the storage unit of the fixed number of storage units, the previous frame of the video sequence comprising the masked object with the current frame further comprising a masked object in the region of interest causes the processing device to perform further operations comprising:

storing the ground truth frame at a first location in a memory; and

storing the previous frame at a location in the memory other than the first location in the memory.

14 . The system of claim 9 , wherein the processing device performs further operations comprising:

determining the region of interest of the current frame based on a region of interest of a previous frame including the masked object.

15 . The system of claim 9 , wherein the processing device performs further operations comprising:

16 . The system of claim 9 , wherein determining, by the mask propagation model, the likelihood of each pixel of the current frame being associated with the object in the region of interest of the current frame based on the region of interest of the current frame and the masked objects in each of the previous frames stored in the fixed number of storage units further causes the processing device to perform further operations comprising:

comparing a similarity of the region of interest of the current frame to the masked objects in each of the previous frames stored in the fixed number of storage units.

17 . A non-transitory computer-readable medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:

determining a fixed number of storage units, wherein each storage unit of the fixed number of storage units stores a previous frame of a video sequence comprising a masked object;

duplicating a ground truth frame masking the object based on the fixed number of storage units;

storing at least one of the ground truth frame or a duplicated ground truth frame in each storage unit;

determining, by a mask propagation model, a likelihood of each pixel of a current frame of the video sequence being associated with an object of the current frame based on the current frame and the masked objects in each of the previous frames stored in the fixed number of storage units;

rescaling the likelihood of each pixel of the current frame being associated with the object of the current frame;

replacing, in a storage unit of the fixed number of storage units, a previous frame of the video sequence with the current frame comprising a masked object based on the rescaled likelihood of each pixel of the current frame being associated with the object of the current frame; and

displaying the current frame of the video sequence including the masked object in the current frame based on the rescaled likelihood of one or more pixels of the current frame being associated with the object.

18 . The non-transitory computer-readable medium of claim 17 , storing instructions that further cause the processing device to perform operations comprising:

determining that the current frame is a reliable frame by comparing a reliability score associated with the current frame to a reliability threshold, wherein the reliability score is based on the rescaled likelihood of each pixel of the current frame being associated with the object in the current frame.

19 . The non-transitory computer-readable medium of claim 17 , storing instructions that further cause the processing device to perform operations comprising:

obtaining a second frame of the video sequence depicting the object;

determining, by the mask propagation model, a likelihood of each pixel of the second frame being associated with the object in the second frame based on the second frame including the object and the masked objects in each of the storage units of the fixed number of storage units;

rescaling the likelihood of each pixel of the second frame being associated with the object of the current frame;

determining that the second frame is not a reliable frame by comparing a reliability score associated with the second frame to a reliability threshold, wherein the reliability score is based on the rescaled likelihood of each pixel of the second frame being associated with the object in the second frame; and

discarding the second frame.

20 . The non-transitory computer-readable medium of claim 17 , wherein determining, by the mask propagation model, the likelihood of each pixel of the current frame of the video sequence being associated with the object of the current frame based on the current frame and the masked objects in each of the previous frames stored in the fixed number of storage units further causes the processing device to perform further operations comprising:

comparing a similarity of a region of interest of the current frame to the masked objects in each of the previous frames stored in the fixed number of storage units.