Low-latency video matting
Techniques are disclosed herein for implementing a novel, low latency, guidance map-free video matting system, e.g., for use in extended reality (XR) platforms. The techniques may be designed to work with low resolution auxiliary inputs (e.g., binary segmentation masks) and to generate alpha mattes (e.g., alpha mattes configured to segment out any object(s) of interest, such as human hands, from a captured image) in near real-time and in a computationally efficient manner. Further, in a domain-specific setting, the system can function on a captured image stream alone, i.e., it would not require any auxiliary inputs, thereby reducing computational costs—without compromising on visual quality and user comfort. Once an alpha matte has been generated, various alpha-aware graphical processing operations may be performed on the captured images according to the generated alpha mattes (e.g., background replacement operations, synthetic shallow depth of field (SDOF) rendering operations, and/or various XR environment rendering operations).
1 . A device, comprising:
a memory;
one or more image capture devices;
a display screen; and
one or more processors operatively coupled to the memory, wherein the one or more processors are configured to execute instructions causing the one or more processors to:
obtain a first captured image of a scene from one of the one or more image capture devices, wherein the first captured image of the scene is obtained from a first image capture stream having a first frame rate;
obtain a binary segmentation mask for the first captured image of a scene, wherein the binary segmentation mask has a first resolution and segments out at least a first type of object from an image;
obtain a second captured image of the scene from one of the one or more image capture devices, wherein the second captured image of the scene is obtained from a second image capture stream having a second frame rate, and wherein the second captured image has a second resolution that is greater than the first resolution;
downscale a version of the second captured image to the first resolution;
feed the binary segmentation mask and the version of the second captured image that is downscaled to the first resolution as inputs to a first trained machine learning (ML) model, wherein at least one output from the first trained ML model for a previously-processed image is used as a prior constraint for generation of an alpha matte for a captured image the first trained ML model is currently processing, wherein an output of the first trained ML model comprises a first alpha matte generated for the second captured image, wherein the first alpha matte has the first resolution, and wherein values in the first alpha matte are indicative of the first trained ML model's confidence level in whether a corresponding portion of the second captured image comprises the first type of object;
upscale the first alpha matte from the first resolution to the second resolution; and
perform a first alpha-aware graphical processing operation on the second captured image according to the upscaled first alpha matte.
2 . The device of claim 1 , wherein the one or more processors are further configured to execute instructions causing the one or more processors to:
obtain a third captured image of the scene, wherein the third captured image has the second resolution and is captured subsequently to the second captured image;
downscale a version of the third captured image to the first resolution;
feed the binary segmentation mask and the version of the third captured image that is downscaled to the first resolution as input to the first trained ML model, wherein at least one output from the first trained ML model for the second captured image is used as a prior constraint to the first trained ML model for generation of a second alpha matte for the third captured image; and
perform a second alpha-aware graphical processing operation on the third captured image according to the second alpha matte.
3 . The device of claim 1 , wherein the first type of object comprises a particular domain of objects, and wherein the first trained ML model is trained to recognize the particular domain of objects in an image.
4 . The device of claim 1 , wherein the first captured image and the second captured image are captured at different times.
5 . The device of claim 1 , wherein the first captured image is captured by a first one of the one or more image capture devices and the second captured image is captured by a second one of the one or more image capture devices.
6 . The device of claim 1 , wherein the second frame rate is greater than the first frame rate.
7 . The device of claim 1 , wherein the first alpha matte comprises a plurality of pixels having continuous confidence values, and wherein higher confidence values represent a higher level of confidence that a given pixel reflects a presence of the first type of object in the scene.
8 . A non-transitory computer readable medium comprising programming instructions stored thereon to cause one or more processors to:
obtain a first captured image of a scene from one image capture device of one or more image capture devices, wherein the first captured image of the scene is obtained from a first image capture stream having a first frame rate;
obtain a binary segmentation mask for the first captured image of a scene, wherein the binary segmentation mask has a first resolution and segments out at least a first type of object from an image;
obtain a second captured image of the scene from one of the one or more image capture devices, wherein the second captured image of the scene is obtained from a second image capture stream having a second frame rate, and wherein the second captured image has a second resolution that is greater than the first resolution;
downscale a version of the second captured image to the first resolution;
feed the binary segmentation mask and the version of the second captured image that is downscaled to the first resolution as inputs to a first trained machine learning (ML) model, wherein at least one output from the first trained ML model for a previously-processed image is used as a prior constraint for generation of an alpha matte for a captured image the first trained ML model is currently processing, wherein an output of the first trained ML model comprises a first alpha matte generated for the second captured image, wherein the first alpha matte has the first resolution, and wherein values in the first alpha matte are indicative of the first trained ML model's confidence level in whether a corresponding portion of the second captured image comprises the first type of object;
upscale the first alpha matte from the first resolution to the second resolution; and
perform a first alpha-aware graphical processing operation on the second captured image according to the upscaled first alpha matte.
9 . The non-transitory computer readable medium of claim 8 , wherein the one or more processors are further configured to execute instructions causing the one or more processors to:
obtain a third captured image of the scene, wherein the third captured image has the second resolution and is captured subsequently to the second captured image;
downscale a version of the third captured image to the first resolution;
feed the binary segmentation mask and the version of the third captured image that is downscaled to the first resolution as input to the first trained ML model, wherein at least one output from the first trained ML model for the second captured image is used as a prior constraint to the first trained ML model for generation of a second alpha matte for the third captured image; and
perform a second alpha-aware graphical processing operation on the third captured image according to the second alpha matte.
10 . The non-transitory computer readable medium of claim 8 , wherein the first type of object comprises a particular domain of objects, and wherein the first trained ML model is trained to recognize the particular domain of objects in an image.
11 . The non-transitory computer readable medium of claim 8 , wherein the first captured image and the second captured image are captured at different times.
12 . The non-transitory computer readable medium of claim 8 , wherein the first captured image is captured by a first one of the one or more image capture devices and the second captured image is captured by a second one of the one or more image capture devices.
13 . The non-transitory computer readable medium of claim 8 , wherein the second frame rate is greater than the first frame rate.
14 . The non-transitory computer readable medium of claim 8 , wherein the first alpha matte comprises a plurality of pixels having continuous confidence values, and wherein higher confidence values represent a higher level of confidence that a given pixel reflects a presence of the first type of object in the scene.
15 . An image processing method, comprising:
obtaining a first captured image of a scene from one image capture device of one or more image capture devices, wherein the first captured image of the scene is obtained from a first image capture stream having a first frame rate;
obtaining a binary segmentation mask for the first captured image of a scene, wherein the binary segmentation mask has a first resolution and segments out at least a first type of object from an image;
obtaining a second captured image of the scene from one of the one or more image capture devices, wherein the second captured image of the scene is obtained from a second image capture stream having a second frame rate, and wherein the second captured image has a second resolution that is greater than the first resolution;
downscaling a version of the second captured image to the first resolution;
feeding the binary segmentation mask and the version of the second captured image that is downscaled to the first resolution as inputs to a first trained machine learning (ML) model, wherein at least one output from the first trained ML model for a previously-processed image is used as a prior constraint for generation of an alpha matte for a captured image the first trained ML model is currently processing, wherein an output of the first trained ML model comprises a first alpha matte generated for the second captured image, wherein the first alpha matte has the first resolution, and wherein values in the first alpha matte are indicative of the first trained ML model's confidence level in whether a corresponding portion of the second captured image comprises the first type of object;
upscaling the first alpha matte from the first resolution to the second resolution; and
performing a first alpha-aware graphical processing operation on the second captured image according to the upscaled first alpha matte.
16 . The method of claim 15 , further comprising:
obtaining a third captured image of the scene, wherein the third captured image has the second resolution and is captured subsequently to the second captured image;
downscaling a version of the third captured image to the first resolution;
feeding the binary segmentation mask and the version of the third captured image that is downscaled to the first resolution as input to the first trained ML model, wherein at least one output from the first trained ML model for the second captured image is used as a prior constraint to the first trained ML model for generation of a second alpha matte for the third captured image; and
performing a second alpha-aware graphical processing operation on the third captured image according to the second alpha matte.
17 . The method of claim 15 , wherein the first type of object comprises a particular domain of objects, and wherein the first trained ML model is trained to recognize the particular domain of objects in an image.
18 . The method of claim 15 , wherein the first captured image and the second captured image are captured at different times.
19 . The method of claim 15 , wherein the second frame rate is greater than the first frame rate.
20 . The method of claim 15 , wherein the first alpha matte comprises a plurality of pixels having continuous confidence values, and wherein higher confidence values represent a higher level of confidence that a given pixel reflects a presence of the first type of object in the scene.