Object removal during video conferencing
A video conferencing system includes an image of a participant in a video conference and a depth map of the image. The system identifies objects in the background of the image, identifies objects in the foreground of the image, and identifies objects in the middle-ground of the image. The system removes the objects from the middle-ground, and replaces the removed objects from the middle-ground with the objects from the background that are located behind the removed objects. The system then uses the image with the removed and replaced objects in a video stream of the video conference.
1. A process comprising:
receiving into a computer processor an image of a participant in a video conference;
receiving a depth map of the image;
identifying one or more objects forming a background in the image using the depth map;
identifying one or more objects forming a foreground in the image using the depth map;
identifying one or more objects forming a middle-ground in the image using the depth map, wherein the middle-ground comprises an area between an object or a person in the foreground and the background;
removing the one or more objects from the middle-ground;
replacing the removed one or more objects from the middle-ground with the one or more objects from the background that are located behind the removed one or more objects; and
using the image with the removed and replaced one or more objects in a video stream associated with the video conference;
wherein the identifying the one or more objects forming a middle-ground in the image using the depth map, the removing the one or more objects from the middle-ground, the replacing the removed one or more objects from the middle-ground with the one or more objects from the background that are located behind the removed one or more objects, and the using the image with the removed and replaced one or more objects in a video stream associated with the video conference are executed a plurality of times during the video conference.
2. The process of claim 1 , wherein the image is received into the computer processor prior to the participant joining the video conference.
3. The process of claim 1 , wherein the one or more objects forming the foreground comprises at least the participant.
4. The process of claim 1 , wherein the depth map is created using one or more of a structured light (SL) camera or a time-of-flight (TOF) camera.
5. The process of claim 1 , wherein the removed and replaced one or more objects from the middle-ground are removed and replaced using a machine learning algorithm.
6. The process of claim 1 , wherein the removed and replaced one or more objects from the middle-ground are removed and replaced by synthesizing the one or more objects from the background in the image that were occluded by the removed and replaced one or more objects from the middle ground.
7. A non-transitory machine-readable medium comprising instructions that when executed by a computer processor execute a process comprising:
receiving into the computer processor an image of a participant in a video conference;
receiving a depth map of the image;
identifying one or more objects forming a background in the image using the depth map;
identifying one or more objects forming a foreground in the image using the depth map;
identifying one or more objects forming a middle-ground in the image using the depth map;
removing the one or more objects from the middle-ground;
replacing the removed one or more objects from the middle-ground with the one or more objects from the background that are located behind the removed one or more objects; and
using the image with the removed and replaced one or more objects in a video stream associated with the video conference;
wherein the identifying the one or more objects forming a middle-ground in the image using the depth map, the removing the one or more objects from the middle-ground, the replacing the removed one or more objects from the middle-ground with the one or more objects from the background that are located behind the removed one or more objects, and the using the image with the removed and replaced one or more objects in a video stream associated with the video conference are executed a plurality of times during the video conference.
8. The non-transitory machine-readable medium of claim 7 , wherein the image is received into the computer processor prior to the participant joining the video conference.
9. The non-transitory machine-readable medium of claim 7 , wherein the one or more objects forming the foreground comprises at least the participant.
10. The non-transitory machine-readable medium of claim 7 , wherein the depth map is created using one or more of a structured light (SL) camera or a time-of-flight (TOF) camera.
11. The non-transitory machine-readable medium of claim 7 , wherein the removed and replaced one or more objects from the middle-ground are removed and replaced using a machine learning algorithm.
12. The non-transitory machine-readable medium of claim 7 , wherein the removed and replaced one or more objects from the middle-ground are removed and replaced by synthesizing the one or more objects from the background in the image that were occluded by the removed and replaced one or more objects from the middle ground.
13. A system comprising:
a computer processor; and
a computer memory coupled to the computer processor;
wherein one or more of the computer processor and the computer memory are operable for:
receiving into the computer processor an image of a participant in a video conference;
receiving a depth map of the image;
identifying one or more objects forming a background in the image using the depth map;
identifying one or more objects forming a foreground in the image using the depth map;
identifying one or more objects forming a middle-ground in the image using the depth map, removing the one or more objects from the middle-ground;
replacing the removed one or more objects from the middle-ground with the one or more objects from the background that are located behind the removed one or more objects; and
using the image with the removed and replaced one or more objects in a video stream associated with the video conference;
wherein the identifying the one or more objects forming a middle-ground in the image using the depth map, the removing the one or more objects from the middle-ground, the replacing the removed one or more objects from the middle-ground with the one or more objects from the background that are located behind the removed one or more objects, and the using the image with the removed and replaced one or more objects in a video stream associated with the video conference are executed a plurality of times during the video conference.
14. The system of claim 13 , wherein the image is received into the computer processor prior to the participant joining the video conference.
15. The system of claim 13 , wherein the one or more objects forming the foreground comprises at least the participant.
16. The system of claim 13 , wherein the depth map is created using one or more of a structured light (SL) camera or a time-of-flight (TOF) camera.
17. The system of claim 13 , wherein the removed and replaced one or more objects from the middle-ground are removed and replaced using a machine learning algorithm.
18. The system of claim 13 , wherein the removed and replaced one or more objects from the middle-ground are removed and replaced by synthesizing the one or more objects from the background in the image that were occluded by the removed and replaced one or more objects from the middle ground.