Instance-adaptive image and video compression in a network parameter subspace using machine learning systems
Techniques are described for compressing data using machine learning systems. An example process can include receiving input data for compression by a neural network compression system. The process can include determining, based on the input data, a set of updated model parameters for the neural network compression system, wherein the set of updated model parameters is selected from a subspace of model parameters. The process can include generating at least one bitstream including a compressed version of the input data and a compressed version of one or more subspace coordinates that correspond to the set of updated model parameters. The process can include outputting the at least one bitstream for transmission to a receiver.
1 . A method of processing image data, comprising:
receiving input data for compression by a neural network compression system;
determining, based on the input data, a set of updated model parameters for the neural network compression system, wherein the set of updated model parameters is selected from a subspace of model parameters based on minimizing, at inference time of a rate-distortion autoencoder (RD-AE) of the neural network compression system, a combined rate-distortion-model rate (RDM) loss corresponding to a rate-distortion loss of the RD-AE at the inference time and a number of model update bits used to represent the subspace of model parameters:
generating at least one bitstream including a compressed version of the input data and a compressed version of one or more subspace coordinates that correspond to the set of updated model parameters; and
outputting the at least one bitstream for transmission to a receiver.
2 . The method of claim 1 , wherein the subspace of model parameters includes a portion of a plurality of weight vectors.
3 . The method of claim 2 , wherein each of the plurality of weight vectors correspond to a weight vector used during training of the neural network compression system.
4 . The method of claim 2 , wherein the portion of the plurality of weight vectors is determined using at least one of principal component analysis (PCA), sparse principal component analysis (SPCA), and model-agnostic meta-learning (MAML).
5 . The method of claim 1 , further comprising:
generating a set of global model parameters based on a training dataset used to train the neural network compression system, wherein the one or more subspace coordinates that correspond to the set of updated model parameters are relative to the set of global model parameters.
6 . The method of claim 5 , wherein determining the set of updated model parameters from the subspace of model parameters comprises:
tuning the set of global model parameters using the input data, wherein the set of global model parameters are tuned based on a bit size of the compressed version of the input data and a distortion between the input data and reconstructed data generated from the compressed version of the input data.
7 . The method of claim 1 , further comprising:
quantizing the one or more subspace coordinates to yield one or more quantized subspace coordinates, wherein the at least one bitstream comprises a compressed version of the one or more quantized subspace coordinates.
8 . The method of claim 7 , wherein the at least one bitstream comprises a plurality of encoded quantization parameters used for quantizing the one or more subspace coordinates.
9 . The method of claim 1 , wherein generating the at least one bitstream comprises:
entropy encoding the one or more subspace coordinates using a model prior.
10 . The method of claim 1 , further comprising:
sending the subspace of model parameters to the receiver.
11 . An apparatus comprising:
at least one memory; and
at least one processor coupled to the at least one memory and configured to:
receive input data for compression by a neural network compression system;
determine, based on the input data, a set of updated model parameters for the neural network compression system, wherein the set of updated model parameters is selected from a subspace of model parameters based on minimizing, at inference time of a rate-distortion autoencoder (RD-AE) of the neural network compression system, a combined rate-distortion-model rate (RDM) loss corresponding to a rate-distortion loss of the RD-AE at the inference time and a number of model update bits used to represent the subspace of model parameters;
generate at least one bitstream including a compressed version of the input data and a compressed version of one or more subspace coordinates that correspond to the set of updated model parameters; and
output the at least one bitstream for transmission to a receiver.
12 . The apparatus of claim 11 , wherein the subspace of model parameters includes a portion of a plurality of weight vectors.
13 . The apparatus of claim 12 , wherein each of the plurality of weight vectors correspond to a weight vector used during training of the neural network compression system.
14 . The apparatus of claim 12 , wherein the portion of the plurality of weight vectors is determined using at least one of principal component analysis (PCA), sparse principal component analysis (SPCA), and model-agnostic meta-learning (MAML).
15 . The apparatus of claim 11 , where the at least one processor is further configured to:
generate a set of global model parameters based on a training dataset used to train the neural network compression system, wherein the one or more subspace coordinates that correspond to the set of updated model parameters are relative to the set of global model parameters.
16 . The apparatus of claim 15 , wherein to determine the set of updated model parameters from the subspace of model parameters the at least one processor is further configured to:
tune the set of global model parameters using the input data, wherein the set of global model parameters are tuned based on a bit size of the compressed version of the input data and a distortion between the input data and reconstructed data generated from the compressed version of the input data.
17 . The apparatus of claim 11 , where the at least one processor is further configured to:
quantize the one or more subspace coordinates to yield one or more quantized subspace coordinates, wherein the at least one bitstream comprises a compressed version of the one or more quantized subspace coordinates.
18 . The apparatus of claim 17 , wherein the at least one bitstream comprises a plurality of encoded quantization parameters used for quantizing the one or more subspace coordinates.
19 . The apparatus of claim 11 , wherein to generate the at least one bitstream the at least one processor is further configured to:
entropy encode the one or more subspace coordinates using a model prior.
20 . The apparatus of claim 11 , wherein the at least one processor is further configured to:
send the subspace of model parameters to the receiver.
21 . A computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to:
receive input data for compression by a neural network compression system;
determine, based on the input data, a set of updated model parameters for the neural network compression system, wherein the set of updated model parameters is selected from a subspace of model parameters based on minimizing, at inference time of a rate-distortion autoencoder (RD-AE) of the neural network compression system, a combined rate-distortion-model rate (RDM) loss corresponding to a rate-distortion loss of the RD-AE at the inference time and a number of model update bits used to represent the subspace of model parameters;
generate at least one bitstream including a compressed version of the input data and a compressed version of one or more subspace coordinates that correspond to the set of updated model parameters; and
output the at least one bitstream for transmission to a receiver.
22 . The computer-readable storage medium of claim 21 , wherein the subspace of model parameters includes a portion of a plurality of weight vectors.
23 . The computer-readable storage medium of claim 22 , wherein each of the plurality of weight vectors correspond to a weight vector used during training of the neural network compression system.
24 . The computer-readable storage medium of claim 22 , wherein the portion of the plurality of weight vectors is determined using at least one of principal component analysis (PCA), sparse principal component analysis (SPCA), and model-agnostic meta-learning (MAML).
25 . The computer-readable storage medium of claim 21 , wherein the instructions further cause the one or more processors to:
generate a set of global model parameters based on a training dataset used to train the neural network compression system, wherein the one or more subspace coordinates that correspond to the set of updated model parameters are relative to the set of global model parameters.
26 . The computer-readable storage medium of claim 25 , wherein to determine the set of updated model parameters from the subspace of model parameters, the instructions further cause the one or more processors to:
tune the set of global model parameters using the input data, wherein the set of global model parameters are tuned based on a bit size of the compressed version of the input data and a distortion between the input data and reconstructed data generated from the compressed version of the input data.
27 . The computer-readable storage medium of claim 21 , where the instructions further cause the one or more processors to:
quantize the one or more subspace coordinates to yield one or more quantized subspace coordinates, wherein the at least one bitstream comprises a compressed version of the one or more quantized subspace coordinates.
28 . The computer-readable storage medium of claim 27 , wherein the at least one bitstream comprises a plurality of encoded quantization parameters used for quantizing the one or more subspace coordinates.
29 . The computer-readable storage medium of claim 21 , wherein to generate the at least one bitstream, the instructions cause the one or more processors to:
entropy encode the one or more subspace coordinates using a model prior.
30 . The computer-readable storage medium of claim 21 , wherein the instructions cause the one or more processors to:
send the subspace of model parameters to the receiver.