Dynamic control for a machine learning autoencoder
An autoencoder is configured to encode content at different quality levels. The autoencoder includes an encoding system and a decoding system with neural network layers forming an encoder network and a decoder network. The encoder network and decoder network are configured to include branching paths through the networks that include different subnetworks. During deployment, content is provided to the encoding system with a quality signal indicating a quality at which the content can be reconstructed. The quality signal determines which of the paths through the encoder network are activated for encoding the content into one or more tensors, which are compressed into a bitstream and later used by the decoding system to reconstruct the content. The autoencoder is trained by randomly or systematically selecting different combinations of tensors to use to encode content and backpropagating error values from loss functions through the network paths associated with the selected tensors.
1. An encoding system stored on a non-transitory computer readable storage medium, wherein the encoding system is manufactured by a process comprising:
obtaining training content that comprises a plurality of examples;
initializing an autoencoder that comprises:
an encoding portion that comprises a neural network having a plurality of paths, where the encoding portion receives an input and generates a plurality of tensors at each of the paths, wherein one or more sub-networks in the encoding portion are shared by the plurality of paths, and wherein each of the plurality of sub-networks in the encoding portion are associated with a respective path, and
a decoding portion that comprises a neural network having a plurality of paths that correspond to the paths of the encoding portion, where the decoding portion receives the plurality of tensors at the corresponding plurality of paths and reconstructs the input therefrom, wherein one or more sub-networks in the encoding portion are shared by the plurality of paths, and wherein each of the plurality of sub-networks in the encoding portion are associated with a respective path;
for each of a plurality of the examples of the training content:
selecting a quality level indicating a required quality of reconstructed input;
selecting one or more corresponding paths through the encoder and decoder portions of the autoencoder based on the selected quality level;
repeatedly backpropagating one or more error terms obtained from a loss function to update a set of parameters of the autoencoder, the backpropagating performed only through the selected paths, wherein the loss function comprises:
a reconstruction loss indicating a dissimilarity between the input and the reconstructed input, and
an encoding loss indicating a length of a bitstream generated by compressing information about the one or more tensors output by the encoding portion; and
storing the set of parameters of the encoding portion of the autoencoder on the computer readable storage medium as parameters of the encoding system, wherein the encoding system is configured to receive an input and produce one or more tensors using the stored parameters, and where the encoding system further comprises a bitstream generator that is coupled to output a compressed bitstream representing the one or more tensors generated by the encoding system using the stored parameters.
2. The encoding system of claim 1 , wherein the paths of the encoding portion are organized in a branching structure, and wherein the paths of the decoding portion are organized into a corresponding branching structure.
3. The encoding system of claim 1 , wherein the quality level indicates a random combination of a plurality of paths through the encoder and decoder portions of the autoencoder to use to encode the training content.
4. The encoding system of claim 1 , wherein the quality level further indicates which of the paths through the encoder and decoder portions of the autoencoder to apply to the training content for generating the loss function.
5. The encoding system of claim 1 , wherein input to the encoding system includes images.
6. The encoding system of claim 1 , wherein input to the encoding system includes frames of a video.
7. The encoding system of claim 1 , wherein the encoding portion of the autoencoder stored on the computer readable storage medium further includes an encoding controller that is coupled to identify which of the one or more produced tensors to include in the compressed bitstream based on the quality level.
8. The encoding system of claim 1 , wherein generating the bitstream comprises combining the one or more tensors and compressing the combination.
9. The encoding system of claim 8 , wherein combining the one or more tensors includes combining the quality level with the one or more tensors.
10. The encoding system of claim 1 , wherein the quality level may be in the form of a tensor that indicates a plurality of quality level values for use in encoding the input, each of the quality level values associated with a different portion of the input.
11. The encoding system of claim 1 , wherein selecting a quality level comprises determining a quality level that will maximize a reconstruction quality for a given target bandwidth.
12. The encoding system of claim 1 , wherein selecting a quality level comprises determining a quality level that will minimize an amount of bandwidth required for a given target reconstruction quality.
13. A decoding system stored on a non-transitory computer readable storage medium, wherein the decoding system is manufactured by a process comprising:
obtaining training content that comprises a plurality of examples;
initializing an autoencoder that comprises:
an encoding portion that comprises a neural network having a plurality of paths, where the encoding portion receives an input and generates a plurality of tensors at each of the paths, wherein one or more sub-networks in the encoding portion are shared by the plurality of paths, and wherein each of the plurality of sub-networks in the encoding portion are associated with a respective path, and
a decoding portion that comprises a neural network having a plurality of paths that correspond to the paths of the encoding portion, where the decoding portion receives the plurality of tensors at the corresponding plurality of paths and reconstructs the input therefrom, wherein one or more sub-networks in the encoding portion are shared by the plurality of paths, and wherein each of the plurality of sub-networks in the encoding portion are associated with a respective path;
for each of a plurality of the examples of the training content:
selecting a quality level indicating a required quality for the reconstructed input;
selecting one or more corresponding paths through the encoder and decoder portions of the autoencoder based on the selected quality level;
repeatedly backpropagating one or more error terms obtained from a loss function to update a set of parameters of the autoencoder, the backpropagating performed only through the selected paths, wherein the loss function comprises:
a reconstruction loss indicating a dissimilarity between the input and the reconstructed input, and
an encoding loss indicating a length of a bitstream generated by compressing information about the one or more tensors output by the encoding portion; and
storing the set of parameters of the decoding portion of the autoencoder on the computer readable storage medium as parameters of the decoding system, wherein the decoding system comprises a tensor generator that is configured to receive and decompress a compressed bitstream comprising one or more tensors representing an input, and where the decoding portion is configured to receive the one or more tensors and generate a reconstruction of the input therefrom using the stored parameters.
14. The decoding system of claim 13 , wherein the paths of the encoding portion are organized in a branching structure, and wherein the paths of the decoding portion are organized into a corresponding branching structure.
15. The decoding system of claim 13 , wherein the quality level indicates a random combination of a plurality of paths through the encoder and decoder portions of the autoencoder to use to encode the training content.
16. The decoding system of claim 13 , wherein the quality level further indicates which of the paths through the encoder and decoder portions of the autoencoder to apply to the training content for generating the loss function.
17. The decoding system of claim 13 , wherein output of the decoding system includes images.
18. The decoding system of claim 13 , wherein output of the decoding system includes frames of a video.
19. The decoding system of claim 13 , wherein the decoding portion of the autoencoder stored on the computer readable storage medium further includes a decoding controller that is coupled to identify which of the one or more received tensors to input into the decoding portion for generating the reconstruction.
20. The decoding system of claim 13 , wherein generating the bitstream comprises combining the one or more tensors and compressing the combination.
21. The decoding system of claim 20 , wherein combining the one or more tensors includes combining the quality level with the one or more tensors.
22. The decoding system of claim 13 , wherein the quality level may be in the form of a tensor that indicates a plurality of quality level values for use in encoding the input, each of the quality level values associated with a different portion of the input.
23. The decoding system of claim 13 , wherein selecting a quality level comprises determining a quality level that will maximize a reconstruction quality for a given target bandwidth.
24. The decoding system of claim 13 , wherein selecting a quality level comprises determining a quality level that will minimize an amount of bandwidth required for a given target reconstruction quality.
25. A method for generating an encoding system, comprising:
obtaining training content that comprises a plurality of examples;
initializing an autoencoder that comprises:
an encoding portion that comprises a neural network having a plurality of paths, where the encoding portion receives an input and generates a plurality of tensors at each of the paths, wherein one or more sub-networks in the encoding portion are shared by the plurality of paths, and wherein each of the plurality of sub-networks in the encoding portion are associated with a respective path, and
a decoding portion that comprises a neural network having a plurality of paths that correspond to the paths of the encoding portion, where the decoding portion receives the plurality of tensors at the corresponding plurality of paths and reconstructs the input therefrom, wherein one or more sub-networks in the encoding portion are shared by the plurality of paths, and wherein each of the plurality of sub-networks in the encoding portion are associated with a respective path;
for each of a plurality of the examples of the training content:
selecting a quality level indicating a required quality of reconstructed input;
selecting one or more corresponding paths through the encoder and decoder portions of the autoencoder based on the selected quality level;
repeatedly backpropagating one or more error terms obtained from a loss function to update a set of parameters of the autoencoder, the backpropagating performed only through the selected paths, wherein the loss function comprises:
a reconstruction loss indicating a dissimilarity between the input and the reconstructed input, and
an encoding loss indicating a length of a bitstream generated by compressing information about the one or more tensors output by the encoding portion; and
storing the set of parameters of the encoding portion of the autoencoder as parameters of the encoding system, wherein the encoding system is configured to receive an input and produce one or more tensors using the stored parameters, and where the encoding system further comprises a bitstream generator that is coupled to output a compressed bitstream representing the one or more tensors generated by the encoding system using the stored parameters.
26. A method for generating a decoding system, comprising:
obtaining training content that comprises a plurality of examples;
initializing an autoencoder that comprises:
an encoding portion that comprises a neural network having a plurality of paths, where the encoding portion receives an input and generates a plurality of tensors at each of the paths, wherein one or more sub-networks in the encoding portion are shared by the plurality of paths, and wherein each of the plurality of sub-networks in the encoding portion are associated with a respective path, and
a decoding portion that comprises a neural network having a plurality of paths that correspond to the paths of the encoding portion, where the decoding portion receives the plurality of tensors at the corresponding plurality of paths and reconstructs the input therefrom, wherein one or more sub-networks in the encoding portion are shared by the plurality of paths, and wherein each of the plurality of sub-networks in the encoding portion are associated with a respective path;
for each of a plurality of the examples of the training content:
selecting a quality level indicating a required quality for the reconstructed input;
selecting one or more corresponding paths through the encoder and decoder portions of the autoencoder based on the selected quality level;
repeatedly backpropagating one or more error terms obtained from a loss function to update a set of parameters of the autoencoder, the backpropagating performed only through the selected paths, wherein the loss function comprises:
a reconstruction loss indicating a dissimilarity between the input and the reconstructed input, and
an encoding loss indicating a length of a bitstream generated by compressing information about the one or more tensors output by the encoding portion; and
storing the set of parameters of the decoding portion of the autoencoder as parameters of the decoding system, wherein the decoding system comprises a tensor generator that is configured to receive and decompress a compressed bitstream comprising one or more tensors representing an input, and where the decoding portion is configured to receive the one or more tensors and generate a reconstruction of the input therefrom using the stored parameters.