Depth assisted images refinement
In some examples in accordance with the present description, an electronic device is provided. The electronic device includes a controller to implement an image segmentation process. The controller is to obtain color information of an image. The controller also is to obtain depth information of the image. The controller also is to determine a depth of a face represented in the color information. The controller also is to segment a foreground of the image from a background of the image according to the color information and the depth information based on the depth of the face.
1 . An electronic device, comprising:
an image sensor;
a depth sensor; and
a controller to:
receive, from the image sensor, red-green-blue (RGB) information of an image in separate R, G, and B channels;
receive, from the depth sensor, depth information of the image;
perform facial detection to identify a face in the RGB information;
truncate the depth information to exclude information for depth points not within a threshold distance from a depth of the identified face;
process the R, G, and B channels of the image and the truncated depth information according to a trained machine learning process to separate a foreground of the image from a background of the image, the trained machine learning process to receive the R, G, and B channels of the image and the truncated depth information as input and to output a segmentation result;
form an image mask by processing the RGB information and the truncated depth information according to the machine learning process;
apply the image mask to the image to obtain a separate representation of a foreground;
manipulate a background of the image by at least one of blurring or replacing the background;
overlay the separate foreground representation on the manipulated background to provide video data; and
transmit, via a network interface, the video data to another electronic device participating in a video conferencing session.
2 . The electronic device of claim 1 , wherein the controller is to truncate the depth information according to Euclidean distance clustering.
3 . The electronic device of claim 1 , wherein the controller is to sample the depth information within a bounding box that bounds the identified face to determine the depth of the identified face.
4 . The electronic device of claim 1 , wherein the controller is to process the R, G, and B channels of the image and the truncated depth information according to a convolutional neural network.
5 . An electronic device, comprising:
a controller to implement an image segmentation process to:
obtain color information of an image in separate color channels;
obtain depth information of the image;
determine a depth of a face represented in the color information;
provide the color information of the image in the separate color channels and the depth of the face to a trained machine learning process to separate a foreground of the image from a background of the image, the trained machine learning process to receive the color channels of the image and the truncated depth information as input and to output a segmentation result,
perform a depth cutoff of points of the depth information and separate the foreground of the image from the background of the image by processing the image and the cutoff depth information according to the trained machine learning process to form an image mask;
concatenate the separate color channels with the truncated depth information to form a four-channel input to the trained machine learning process;
process the four-channel input to generate the segmentation result; and
generate an updated face bounding box based on the segmentation result, the updated face bounding box for use in performing depth sampling for a subsequently received image.
6 . The electronic device of claim 5 , wherein the controller is to perform facial detection on the color channels of the image to define a region of the image including the face and sample the depth information of the image within the region to determine the depth of the face.
7 . The electronic device of claim 6 , wherein the controller is to perform a depth cutoff of points of the depth information that have a greater distance from a viewpoint than the depth of the face plus a threshold value.
8 . The electronic device of claim 5 , wherein the controller is to apply the image mask to the image to separate the foreground of the image from the background of the image.
9 . A non-transitory computer-readable medium storing machine-readable instructions which, when executed by a controller of an electronic device, cause the controller to:
obtain color information of an image in separate red, green, and blue color channels;
obtain depth information of the image;
determine a depth of a face present in the image;
perform a depth cutoff of the depth information for points having greater than a threshold distance from the depth of the face;
provide a 4-channel input to a machine learning process to process the image according to the cutoff depth information and the color information to separate a foreground of the image from a background of the image, the 4-channel input including the cutoff depth information, the red color channel, the green color channel, and the blue color channel, the trained machine learning process to receive the 4-channel input and to output a segmentation result,
form an image mask by processing the 4-channel input according to the machine learning process;
apply the image mask to the image to obtain a separate representation of a foreground;
manipulate a background of the image by at least one of blurring or replacing the background; and
overlay the separate foreground representation on the manipulated background to provide video data for a video conferencing session.
10 . The computer-readable medium of claim 9 , wherein execution of the executable code causes the controller to determine a bounding box surrounding the face in the color information of the image and determine the depth of the face by sampling the depth information at points with a region bounded by the bounding box.
11 . The computer-readable medium of claim 9 , wherein execution of the executable code causes the controller to perform the depth cutoff according to Euclidean distance clustering.
12 . The computer-readable medium of claim 9 , wherein execution of the executable code causes the controller to overlay the foreground over a manipulated representation of the image.
13 . The computer-readable medium of claim 12 , wherein the manipulated representation is a blurring of the image or a replacement of the image.
14 . The electronic device of claim 1 , wherein the controller is to concatenate the RGB information with the truncated depth information to generate a 4-channel input for the trained machine learning process.
15 . The electronic device of claim 1 , wherein the controller is to manipulate the RGB information to overlay the RGB information of the foreground of the image on a manipulated representation of the RGB information of the background of the image.
16 . The electronic device of claim 5 , wherein the controller is to manipulate the color information to overlay the color information of the foreground of the image on a manipulated representation of the color information of the background of the image.
17 . The computer-readable medium of claim 9 , wherein execution of the executable code causes the controller to concatenate the red, green, and blue color channels with the truncated depth information to generate the 4-channel input for the trained machine learning process.
18 . The computer-readable medium of claim 9 , wherein execution of the executable code causes the controller to manipulate the red, green, and blue color channels to overlay the color information of the foreground of the image on a manipulated representation of the color information of the background of the image.
19 . The electronic device of claim 1 , wherein applying the image mask comprises separating foreground pixels corresponding to the face from background pixels prior to manipulating the background.
20 . The electronic device of claim 5 , wherein the truncated depth information is generated by excluding depth points having distances greater than the determined depth of the face plus a threshold value.