Granted Patent B2

US 10,326,815 · App. 15/385,367 · Granted Jun 18, 2019

Abstract

Claims (96)

1. A method performed by a computing device, the method comprising:

generating an encoded video stream having a plurality of ordered layers of increasing quality, a lowest layer including video content of key frames of the video stream and excluding at least one delta frame of the video stream, a highest layer including video content of the key frames and all delta frames of the video stream, each layer of quality but the lowest layer including video content of at least one lower quality layer, each layer of quality having a respective target data rate larger than target data rates of all lower quality layers, wherein:

each delta frame includes difference data which, in combination with video content of at least one previous frame of the video stream, reconstructs that delta frame, and

each key frame contains a complete picture encoded independently of other frames;

generating metadata for each layer of the video stream, the metadata for each layer identifying particular video content to be used to reconstruct video at that layer of quality; and

sending the video content of the encoded video stream and the metadata for each layer of the video stream to a streaming server for selective bandwidth-based distribution to receivers having differing network bandwidths.

2. The method of claim 1 wherein:

the streaming server is remote from the computing device; and

sending the video content of the encoded video stream and the metadata for each layer of the video stream to the streaming server includes sending the video content and the metadata to the streaming server across a network connection.

3. The method of claim 1 wherein generating the encoded video stream includes taking, as input, a screen of a display of the computing device, the screen being controlled by a user of the computing device, the user sharing the screen with the receivers.

4. The method of claim 1 wherein:

an epoch is defined by a period of time during which each frame of the highest layer is displayed; and

generating metadata for each layer of the video stream includes, for each epoch of the video stream, generating an active set for each layer, wherein the active set for each given layer and each given epoch includes a reference to:

video content of a key frame upon which a display of that given epoch is based; and

video content of all delta frames which are needed to generate a display of that given epoch at that given layer of quality in combination with the video content of the key frame upon which that given epoch is based.

5. The method of claim 1 wherein generating the encoded video stream includes compressing video using an inter-frame compression scheme in a scalable manner, the receivers being configured to decode the video encoded in the inter-frame compression scheme.

6. The method of claim 5 wherein the streaming server is not configured to decode the video encoded in the inter-frame compression scheme.

7. The method of claim 5 wherein the method further comprises:

generating additional encoded video streams each having a plurality of ordered layers of increasing quality, the encoded video stream and the additional encoded video streams piecing together to form a larger video stream, the larger video stream representing a screen of a display of the computing device, at least one of the additional encoded video streams being compressed using a scheme different than the inter-frame compression scheme used for the encoded video stream;

generating additional metadata for each layer of the additional encoded video streams; and

sending video content of the additional encoded video streams and the additional metadata for each layer of the additional encoded video streams to the streaming server for selective bandwidth-based distribution to the receivers.

8. The method of claim 5 wherein:

the lowest layer includes all video content of key frames to the exclusion of video content of any delta frames of the video stream; and

an intermediate layer includes video content of key frames and a strict subset of delta frames of the video stream.

9. The method of claim 5 wherein:

the lowest layer includes a strict subset of the video content of key frames of the video stream, to the exclusion of video content of any delta frames of the video stream, the strict subset of the video content of the key frames generating the key frames at a low level of quality; and

an intermediate layer includes all video content of the key frames, wherein all video content of the key frames generates the key frames at a higher level of quality than the low level of quality.

10. The method of claim 5 wherein:

an intermediate layer includes video content of the key frames and all delta frames of the video stream, excluding any overlays; and

the highest layer includes video content of the key frames and all delta frames of the video stream as well as overlay information to be overlaid over the key frames and all delta frames.

11. The method of claim 5 wherein the method further comprises:

generating an additional layer of the encoded video stream, the additional layer including video content of the video stream encoded independent of video content of the ordered layers, the additional layer being encoded using a scheme different than the inter-frame compression scheme, the additional layer not including video content of the ordered layers;

generating additional metadata for the additional layer; and

sending the additional metadata to the streaming server together with the video content of the encoded video stream and the metadata for each ordered layer of the video stream.

12. The method of claim 1 wherein the method further includes receiving definition information regarding the ordered layers of quality from the streaming server, the definition information including:

a number of layers to be used in the encoded video stream; and

the target data rate of each layer of quality.

13. The method of claim 12 wherein the method further includes receiving updated definition information regarding the ordered layers of quality from the streaming server based on feedback provided to the streaming server by the receivers during operation.

14. The method of claim 1 wherein the method further includes:

receiving, from the streaming server, aggregated feedback from each of the receivers; and

in response to receiving the aggregated feedback, updating:

a number of layers to be used in the encoded video stream; and

the target data rate of each layer of quality.

15. The method of claim 1 wherein the method further comprises encrypting the video content of the encoded video stream prior to sending the video content of the encoded video stream to the streaming server, the receivers being configured to decrypt the encrypted video content, the streaming server not being configured to decrypt the encrypted video content.

16. The method of claim 1 wherein the method further comprises:

generating a cryptographic signature of the video content of the encoded video stream using a key;

sending the generated cryptographic signature to the streaming server, the receivers being configured to confirm integrity of the video content with reference to the generated cryptographic signature, the streaming server not having the key.

17. A method performed by a streaming server device, the method comprising:

receiving, from a source device over a network connection, an encoded video stream having a plurality of ordered layers of increasing quality, a lowest layer including video content of key frames of the video stream and excluding at least one delta frame of the video stream, a highest layer including video content of the key frames and all delta frames of the video stream, each layer of quality but the lowest layer including video content of at least one lower quality layer, each layer of quality having a respective target data rate larger than target data rates of all lower quality layers, wherein:

each delta frame includes difference data which, in combination with video content of at least one previous frame of the video stream, reconstructs that delta frame, and

each key frame contains a complete picture encoded independently of other frames;

receiving, from the source device over the network connection, metadata for each layer of the video stream, the metadata for each layer identifying particular video content to be used to reconstruct video at that layer of quality;

for each of a plurality of remote recipients having different available bandwidths, assigning a layer of quality to that remote recipient such that the target data rate of the assigned layer of quality is exceeded by the available bandwidth of that remote recipient; and

for each remote recipient, selectively sending to that remote recipient video content that is identified by the metadata for the layer of quality assigned to that remote recipient.

18. The method of claim 17 wherein:

an epoch is defined by a period of time during which each frame of the highest layer is displayed; and

receiving metadata for each layer of the video stream includes, for each epoch of the video stream, receiving an active set for each layer, wherein the active set for each given layer and each given epoch includes a reference to:

video content of a key frame upon which a display of that given epoch is based; and

video content of all delta frames which are needed to generate a display of that given epoch at that given layer of quality in combination with the video content of the key frame upon which that given epoch is based.

19. The method of claim 18 wherein, for each remote recipient, selectively sending to that remote recipient video content that is identified by the metadata for the layer of quality assigned to that remote recipient includes, for a particular epoch at which the highest layer includes video content of a delta frame:

sending, at the particular epoch, video content to a first recipient device that is assigned the highest layer of quality; and

refraining from sending, at the particular epoch, any video content to a second recipient device that is assigned the lowest layer of quality.

20. The method of claim 17 wherein the streaming server device is not configured to decode the received video stream.

21. The method of claim 17 wherein receiving the encoded video stream includes receiving the video content in encrypted form, the streaming server device not being configured to decrypt the video content, the remote recipients being configured to decrypt the video content.

22. The method of claim 17 ,

wherein the method further includes receiving feedback from the remote recipients, the feedback from each remote recipient indicating whether the video content sent to that remote recipient is:

too fast for the available bandwidth of that remote recipient,

too slow for the available bandwidth of that remote recipient, or

suitable for the available bandwidth of that remote recipient; and

wherein, for each remote recipient, assigning the layer of quality to that remote recipient includes:

for a remote recipient from which the feedback indicates that the video content sent to that remote recipient is too fast, reassigning a lower layer of quality to that remote recipient;

for a remote recipient from which the feedback indicates that the video content sent to that remote recipient is too slow, reassigning a higher layer of quality to that remote recipient; and

for a remote recipient from which the feedback indicates that the video content sent to that remote recipient is suitable, leaving the layer of quality assigned to that remote recipient unchanged.

23. The method of claim 22 wherein, for each remote recipient, selectively sending to that remote recipient video content that is identified by the metadata for the layer of quality assigned to that remote recipient includes, for a particular epoch at which the highest layer includes video content of a first delta frame and at which a lower layer does not include video content of the first delta frame:

sending, at the particular epoch, video content of only the first delta frame to a first recipient device that is assigned the highest layer of quality; and

sending, at the particular epoch, video content of both the first delta frame and a second delta frame, the second delta frame being from an earlier epoch, to a second recipient device which was reassigned to a higher layer of quality.

24. The method of claim 22 wherein the method further comprises, in response to receiving the feedback:

determining that a new set of target data rates different than the target data rates for each layer of quality would be more efficient for the remote recipients in light of the feedback; and

sending the new set of target data rates to the source device over the network connection for use in encoding the video stream in the future.

25. The method of claim 22 wherein the method further comprises, in response to receiving the feedback, sending the feedback from the remote recipients to the source device in an aggregated manner.

26. The method of claim 17 wherein the method further comprises:

using a virtual buffer for each remote recipient to model its video reception; and

selectively reassigning the remote recipients to new layers of quality based on underrun and overrun conditions in their respective virtual buffers.

27. A computer program product comprising a non-transitory computer-readable storage medium storing a set of instructions, which when executed by a computing device, cause the computing device to perform the following operations:

generating an encoded video stream having a plurality of ordered layers of increasing quality, a lowest layer including video content of key frames of the video stream and excluding at least one delta frame of the video stream, a highest layer including video content of the key frames and all delta frames of the video stream, each layer of quality but the lowest layer including video content of at least one lower quality layer, each layer of quality having a respective target data rate larger than target data rates of all lower quality layers, wherein:

each delta frame includes difference data which, in combination with video content of at least one previous frame of the video stream, reconstructs that delta frame, and

each key frame contains a complete picture encoded independently of other frames;

generating metadata for each layer of the video stream, the metadata for each layer identifying particular video content to be used to reconstruct video at that layer of quality; and

sending the video content of the encoded video stream and the metadata for each layer of the video stream to a streaming server for selective bandwidth-based distribution to receivers having differing network bandwidths.

28. A computer program product comprising a non-transitory computer-readable storage medium storing a set of instructions, which when executed by a computing device, cause the computing device to perform the following operations:

receiving, from a source device over a network connection, an encoded video stream having a plurality of ordered layers of increasing quality, a lowest layer including video content of key frames of the video stream and excluding at least one delta frame of the video stream, a highest layer including video content of the key frames and all delta frames of the video stream, each layer of quality but the lowest layer including video content of at least one lower quality layer, each layer of quality having a respective target data rate larger than target data rates of all lower quality layers, wherein:

each delta frame includes difference data which, in combination with video content of at least one previous frame of the video stream, reconstructs that delta frame, and

each key frame contains a complete picture encoded independently of other frames;

receiving, from the source device over the network connection, metadata for each layer of the video stream, the metadata for each layer identifying particular video content to be used to reconstruct video at that layer of quality;

for each of a plurality of remote recipients having different available bandwidths, assigning a layer of quality to that remote recipient such that the target data rate of the assigned layer of quality is exceeded by the available bandwidth of that remote recipient; and

for each remote recipient, selectively sending to that remote recipient video content that is identified by the metadata for the layer of quality assigned to that remote recipient.

Assignments (12)