Machine-learning based data compression for streaming media
Technology to encode/compress and decode/uncompress video streams is described. According to one example, a computer-implemented method includes receiving, by a server and from a first device associated with a first participant of a videoconference involving a second device associated with a second participant, a first compressed video stream and a first decoding model. The first decoding model is trained via machine learning, and the first compressed video stream is compressed using a first encoding model trained via machine learning. The first encoding and decoding models are customized to the first participant. The server is adapted to transmit to the second device, the first decoding model and the first compressed video stream.
1. A computer-implemented method, comprising:
receiving a set of original video frames comprising images of a user;
dividing the set of original video frames into a training subset and a test subset wherein the test subset includes one or more sets of reconstructed frames;
training a machine learning (ML) module using the training subset until a training error is less than a specified training threshold, to obtain a first encoding model that is customized to the user based at least in part on the training subset, a first decoding model that is customized to the user based at least in part on the training subset, one or more second encoding models that are customized to one or more other users, and one or more second decoding models that are customized to the one or more other users;
confirming using the test subset that a test error is less than a specified test threshold, wherein the test error is obtained using an adversarial network, wherein the adversarial network classifies one or more pairs of frames, wherein an individual pair of the one or more pairs comprises: (i) a first frame of the set of original video frames from the training subset, and (ii) a second frame of the set of original video frames from the test subset, and wherein the test error is based at least in part on a probability that the frames of the individual pair belong to a same or similar class; and
upon confirming that the test error is less than the specified test threshold, transmitting: (i) the first encoding model and the one or more second encoding models to a user device used by the user or to a video-conferencing server, and (ii) the first decoding model and the one or more second decoding models to the video-conferencing server or to one or more other user devices.
2. The computer-implemented method of claim 1 , wherein the set of original video frames comprises: (i) raw, unencoded video frames obtained from a camera, or (ii) encoded video frames obtained from a standardized video encoder.
3. The computer-implemented method of claim 1 , wherein each step of the method is performed at the user device used by the user or at the video-conferencing server.
4. The computer-implemented method of claim 1 , wherein the first decoding model comprises configuration of a decoding stage of an autoencoder.
5. The computer-implemented method of claim 1 , wherein the machine learning comprises training of an autoencoder using video obtained from the user or the one or more other users.
6. The computer-implemented method of claim 1 , wherein the probability is determined according to a simple difference function or a weighted difference function.
7. The computer-implemented method of claim 1 , wherein the set of original video frames comprise encoded video frames obtained from a standardized video encoder, wherein the encoded video frames include memcodes, and wherein the first decoding model reconstructs video frames using the memcodes and subject-specific code vectors.
8. A system comprising:
a processor; and
a memory in communication with the processor and comprising instructions which, when executed by the processor, program the processor to:
receive a set of original video frames comprising images of a user;
divide the set of original video frames into a training subset and a test subset, wherein the test subset includes one or more sets of reconstructed frames;
train a machine learning (ML) module using the training subset until a training error is less than a specified training threshold, to obtain a first encoding model that is customized to the user based at least in part on the training subset, a first decoding model that is customized to the user based at least in part on the training subset, a second encoding model that is customized to a second user, and a second decoding model that is customized to the second user;
confirm using the test subset that a test error is less than a specified test threshold, wherein the test error is obtained using an adversarial network, wherein the adversarial network classifies one or more pairs of frames, wherein an individual pair of the one or more pairs of frames comprises: (i) a first frame of the set of original video frames from the training subset, and (ii) a second frame of the set of original video frames from the test subset, and wherein the test error is based at least in part on a probability that the frames of the individual pair belong to a same or similar class; and
upon confirming that the test error is less than the specified test threshold, transmit: (i) the first encoding model or the second encoding model to a user device used by the user or to a video-conferencing server, and (ii) the first decoding model or the second decoding model to the video-conferencing server or to one or more other user devices.
9. The system of claim 8 , wherein the set of original video frames comprises: (i) raw, unencoded video frames obtained from a camera, or (ii) encoded video frames obtained from a standardized video encoder.
10. The system of claim 8 , wherein each step is performed at the user device used by the user or at the video-conferencing server.
11. The system of claim 8 , wherein the first decoding model comprises configuration of a decoding stage of an autoencoder.
12. The system of claim 8 , wherein the machine learning comprises training of an autoencoder using video obtained from the second user.
13. The system of claim 8 , wherein the probability is determined according to a simple difference function or a weighted difference function.
14. One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving a set of original video frames comprising images of a user;
dividing the set of original video frames into a training subset and a test subset wherein the test subset includes one or more sets of reconstructed frames;
training a machine learning (ML) module using the training subset until a training error is less than a specified training threshold, to obtain a first encoding model that is customized to the user based at least in part on the training subset, a first decoding model that is customized to the user based at least in part on the training subset, a second encoding model that is customized to a second user, and a second decoding model that are customized to the second user;
confirming using the test subset that a test error is less than a specified test threshold, wherein the test error is obtained using an adversarial network, wherein the adversarial network classifies one or more pairs of frame sequences, wherein an individual pair of the one or more pairs of frame sequences comprises: (i) a first frame sequence of the set of original video frames from the training subset, and (ii) a second frame sequence of the set of original video frames from the test subset, and wherein the test error is based at least in part on a probability that a majority of frames from the individual pair belong to a same or similar class; and
upon confirming that the test error is less than the specified test threshold, transmitting: (i) the first encoding model or the second encoding model to a user device used by the user or to a video-conferencing server, and (ii) the first decoding model or the second decoding model to the video-conferencing server or to one or more other user devices.
15. The one or more non-transitory computer-readable media of claim 14 , wherein the set of original video frames comprises: (i) raw, unencoded video frames obtained from a camera, or (ii) encoded video frames obtained from a standardized video encoder.
16. The one or more non-transitory computer-readable media of claim 14 , wherein each step is performed at the user device used by the user or at the video-conferencing server.
17. The one or more non-transitory computer-readable media of claim 14 , wherein the first decoding model comprises configuration of a decoding stage of an autoencoder.
18. The one or more non-transitory computer-readable media of claim 14 , wherein the machine learning comprises training of an autoencoder using video obtained from the second user.
19. The one or more non-transitory computer-readable media of claim 14 , wherein the probability is determined according to a simple difference function or a weighted difference function.
20. The one or more non-transitory computer-readable media of claim 14 , wherein the frame sequences correspond to a pre-determined time window.