Loss-resilient real-time video streaming
Systems, methods, and computer program products are provided for streaming video over a network. In various embodiments, a source video including at least a source frame is read. The source frame is encoded into a corresponding tensor representation by a machine learning model. The corresponding tensor representation is decomposed into a plurality of sub-tensors. Each of the plurality of sub-tensors is encoded into a corresponding packet and transmitted via a network from a source node to a receiver node.
1. A method of streaming video over a network, the method comprising:
reading a source video, the source video comprising at least a source frame;
encoding the source frame into a corresponding tensor representation by a machine learning model;
decomposing the corresponding tensor representation into a plurality of sub-tensors;
encoding each of the plurality of sub-tensors into a corresponding packet;
transmitting each of the corresponding packets via a network from a source node to a receiver node.
2. The method of claim 1 , wherein encoding the source frame comprises determining an I-frame corresponding to the source frame.
3. The method of claim 1 , wherein encoding the source frame comprises determining a P-frame corresponding to the source frame.
4. The method of claim 1 , wherein encoding the source frame comprises determining a composite frame comprising at least two regions, the first region encoded as a P-frame and the second region encoded as an I-frame.
5. The method of claim 4 , further comprising:
encoding a second source frame by determining a composite frame comprising a third region, different than the second region, the third region being encoded as an I-frame.
6. The method of claim 1 , wherein the machine learning model comprises an artificial neural network.
7. The method of claim 6 , wherein encoding each of the plurality of source frames into its corresponding tensor representation comprises applying an autoencoder.
8. The method of claim 1 , wherein decomposing the tensor representation comprises randomly selecting the plurality of sub-tensors.
9. The method of claim 1 wherein encoding each of the plurality of sub-tensors comprises applying entropy encoding.
10. The method of claim 1 , further comprising:
receiving at least some of the packets via the network at the receiver node;
decoding the received packets, thereby obtaining at least some of the plurality of sub-tensors;
combining the at least some of the plurality of sub-tensors to form a playback tensor;
decoding the playback tensor to determine a playback frame.
11. The method of claim 10 , wherein decoding the received packets comprises applying entropy encoding.
12. The method of claim 10 , wherein decoding the playback tensor comprises applying an artificial neural network.
13. The method of claim 12 , wherein decoding the playback tensor comprises applying an autoencoder.
14. The method of claim 10 , wherein the playback frame comprises a P-frame, an I-frame, or a composite thereof.
15. The method of claim 10 , wherein at least one of the plurality of sub-tensors is not obtained, and wherein the playback tensor comprises zero-values in place of the at least one of the plurality of sub-tensors not obtained.
16. The method of claim 10 , wherein at least one of the plurality of sub-tensors is not obtained, the method further comprising:
sending from the receiver node to the source node an indication of the at least one of the plurality of sub-tensors not obtained;
in response to said indication, determining the playback frame at the source node;
determining a resynchronized frame configured to correct the omission of the at least one of the plurality of sub-tensors not obtained;
transmitting the resynchronized frame from the source node to the receiver node.
17. A method of playing streaming video, the method comprising:
receiving a plurality of packets via a network from a source node, each of the plurality of packets comprising an encoded sub-tensor of a tensor;
decoding each of the plurality of packets, thereby obtaining a plurality of sub-tensors of the tensor;
combining the plurality of sub-tensors to form a playback tensor;
decoding the playback tensor to determine a playback frame.
18. A computer program product for streaming video over a network, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to perform a method, the method comprising:
reading a source video, the source video comprising at least a source frame;
encoding the source frame into a corresponding tensor representation by a machine learning model;
decomposing the corresponding tensor representation into a plurality of sub-tensors;
encoding each of the plurality of sub-tensors into a corresponding packet; and
transmitting each of the corresponding packets via a network from a source node to a receiver node.
19. A computer program product for playing streaming video, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to perform a method, the method comprising:
receiving a plurality of packets via a network from a source node, each of the plurality of packets comprising an encoded sub-tensor of a tensor;
decoding each of the plurality of packets, thereby obtaining a plurality of sub-tensors of the tensor;
combining the plurality of sub-tensors to form a playback tensor; and
decoding the playback tensor to determine a playback frame.
20. A system comprising:
a source node configured to
read a source video, the source video comprising at least a source frame, encode the source frame into a corresponding tensor representation by a machine learning model,
decompose the corresponding tensor representation into a plurality of sub-tensors, encode each of the plurality of sub-tensors into a corresponding packet,
transmit each of the corresponding packets via a network to the receiver node; and
a receiver node configured to
receive at least some of the packets via the network,
decode the received packets, thereby obtaining at least some of the plurality of sub-tensors,
combine the at least some of the plurality of sub-tensors to form a playback tensor,
decode the playback tensor to determine a playback frame.