Multi-stream video encoding for screen sharing within a communications session
Methods, systems, and apparatus, including computer programs encoded on computer storage media related to multi-stream video encoding for screen sharing a communications session. The system may determine a sub-area and a remaining area of a display region. A first video stream of the sub-area is generated at a first frame rate. A second video stream of the remaining area is generated at second frame rate, where the second frame rate is a frame rate lower than the first frame rate. A client device may transmit the first video stream and the second video stream to a second client device.
1. A computer-implemented method comprising:
determining a sub-area of a display region, the display region including a group of pixels and the sub-area including a subset of the group of pixels;
determining a remaining area of the display region, the remaining area including a subset of pixels from the group of pixels that are outside of the sub-area;
generating a first video stream of at least a portion of the sub-area of the display region, the first video stream having a first frame rate;
generating a second video stream of at least a portion of the remaining area of the display region, the second video stream having a second frame rate that is a frame rate lower than the first frame rate;
transmitting, by a first client device, the first video stream and the second video stream to a second client device;
determining a second sub-area of the display region;
generating a third video stream of the second sub-area, the third video stream having a third frame rate; and
transmitting, by the first client device, the third video stream to a second client device concurrently with the first video stream and the second video stream.
2. The computer-implemented method of claim 1 , wherein the sub-area of the display region includes an area of changing pixel values among consecutive frames of the display region.
3. The computer-implemented method of claim 1 , wherein the remaining area of the display region includes a portion of the display region that has a group of static pixel values among consecutive frames of the display region.
4. The computer-implemented method of claim 1 , further comprising:
determining whether pixels about a boundary of the sub-area are changing among frames of the display region; and
increasing a size of the sub-area to include those pixels that are determined to be changing in values, or decreasing a size of the sub-area to exclude those pixels that are determined to be not-change in values.
5. The computer-implemented method of claim 1 , wherein the first video stream is transmitted at a variable bitrate, and the second video stream is transmitted at a constant bitrate.
6. A non-transitory computer readable medium that stores executable program instructions that when executed by one or more computing devices configure the one or more computing devices to perform operations comprising:
determining a sub-area of a display region, the display region including a group of pixels and the sub-area including a subset of the group of pixels;
determining a remaining area of the display region, the remaining area including a subset of pixels from the group of pixels that are outside of the sub-area;
generating a first video stream of at least a portion of the sub-area of the display region, the first video stream having a first frame rate;
generating a second video stream of at least a portion of the remaining area of the display region, the second video stream having a second frame rate that is a frame rate lower than the first frame rate;
transmitting, by a first client device, the first video stream and the second video stream to a second client device;
determining a second sub-area of the display region;
generating a third video stream of the second sub-area, the third video stream having a third frame rate; and
transmitting, by the first client device, the third video stream to a second client device concurrently with the first video stream and the second video stream.
7. The non-transitory computer readable medium of claim 6 , wherein the sub-area of the display region includes an area of changing pixel values among consecutive frames of the display region.
8. The non-transitory computer readable medium of claim 6 , wherein the remaining area of the display region includes a portion of the display region that has a group of static pixel values among consecutive frames of the display region.
9. The non-transitory computer readable medium of claim 6 , the operations further comprising:
determining whether pixels about a boundary of the sub-area are changing among frames of the display region; and
increasing a size of the sub-area to include those pixels that are determined to be changing in values, or decreasing a size of the sub-area to exclude those pixels that are determined to be not-change in values.
10. The non-transitory computer readable medium of claim 6 , wherein the first video stream is transmitted at a variable bitrate, and the second video stream is transmitted at a constant bitrate.
11. A system comprising one or more processors configured to perform the operations of:
determining a sub-area of a display region, the display region including a group of pixels and the sub-area including a subset of the group of pixels;
determining a remaining area of the display region, the remaining area including a subset of pixels from the group of pixels that are outside of the sub-area;
generating a first video stream of at least a portion of the sub-area of the display region, the first video stream having a first frame rate;
generating a second video stream of at least a portion of the remaining area of the display region, the second video stream having a second frame rate that is a frame rate lower than the first frame rate;
transmitting, by a first client device, the first video stream and the second video stream to a second client device;
determining a second sub-area of the display region;
generating a third video stream of the second sub-area, the third video stream having a third frame rate; and
transmitting, by the first client device, the third video stream to a second client device concurrently with the first video stream and the second video stream.
12. The system of claim 11 , wherein the sub-area of the display region includes an area of changing pixel values among consecutive frames of the display region.
13. The system of claim 11 , wherein the remaining area of the display region includes a portion of the display region that has a group of static pixel values among consecutive frames of the display region.
14. The system of claim 11 , the operations further comprising:
determining whether pixels about a boundary of the sub-area are changing among frames of the display region; and
increasing a size of the sub-area to include those pixels that are determined to be changing in values, or decreasing a size of the sub-area to exclude those pixels that are determined to be not-change in values.
15. The system of claim 11 , wherein the first video stream is transmitted at a variable bitrate, and the second video stream is transmitted at a constant bitrate.
16. A computer-implemented method comprising:
determining a sub-area of a display region, the display region including a group of pixels and the sub-area including a subset of the group of pixels;
determining a remaining area of the display region, the remaining area including a subset of pixels from the group of pixels that are outside of the sub-area;
generating a first video stream of at least a portion of the sub-area of the display region, the first video stream having a first frame rate;
generating a second video stream of at least a portion of the remaining area of the display region, the second video stream having a second frame rate that is a frame rate lower than the first frame rate;
transmitting, by a first client device, the first video stream and the second video stream to a second client device;
receiving, by the first client device, a third video stream having a third frame rate, the third video stream generated by the second client device;
receiving, by the first client device, a fourth video stream having a fourth frame rate, the fourth frame rate being a frame rate greater than the third frame rate;
combining, by the first client device, the third video stream and the fourth video stream to form a composite video stream; and
providing for display the composite video stream to a user interface of the first client device, wherein the composite video stream displays at the frame rate of the third video stream.
17. A non-transitory computer readable medium that stores executable program instructions that when executed by one or more computing devices configure the one or more computing devices to perform operations comprising:
determining a sub-area of a display region, the display region including a group of pixels and the sub-area including a subset of the group of pixels;
determining a remaining area of the display region, the remaining area including a subset of pixels from the group of pixels that are outside of the sub-area;
generating a first video stream of at least a portion of the sub-area of the display region, the first video stream having a first frame rate;
generating a second video stream of at least a portion of the remaining area of the display region, the second video stream having a second frame rate that is a frame rate lower than the first frame rate;
transmitting, by a first client device, the first video stream and the second video stream to a second client device;
receiving, by the first client device, a third video stream having a third frame rate, the third video stream generated by the second client device;
receiving, by the first client device, a fourth video stream having a fourth frame rate, the fourth frame rate being a frame rate greater than the third frame rate;
combining, by the first client device, the third video stream and the fourth video stream to form a composite video stream; and
providing for display the composite video stream to a user interface of the first client device, wherein the composite video stream displays at the frame rate of the third video stream.