Signal coding using a generative model and latent domain quantization
The present disclosure provides a decoder configured to receive a finite bitrate stream that includes a quantized latent frame, where the quantized latent frame includes a quantized representation of a current frame of a signal in a latent domain different from a first domain; to generate a reconstructed latent frame from the quantized latent frame; to use a generative neural network model to perform a task for which the general neural network model has been trained, wherein the task includes to generate parameters for an invertible mapping from the latent domain to the first domain; to reconstruct a current frame of the signal in the first domain, which includes to map the reconstructed latent frame to the first domain by use of the invertible mapping, and to use the reconstructed current frame of the signal in the first domain to update a state of the generative neural network model.
1 . A decoder, configured to:
receive a finite bitrate stream that includes a quantized latent frame, wherein the quantized latent frame includes a quantized representation of a current frame (t) of a signal in a latent domain different from a first domain;
generate a reconstructed latent frame from the quantized latent frame;
use a generative neural network model to perform a task for which the generative neural network model has been trained, wherein the task includes generating parameters for an invertible mapping from the latent domain to the first domain;
reconstruct a current frame of the signal in the first domain, which includes mapping the reconstructed latent frame to the first domain by use of the invertible mapping, wherein the mapping uses the generated parameters to reconstruct information lost during quantization, and
use the reconstructed current frame of the signal in the first domain to update a state of the generative neural network model.
2 . The decoder of claim 1 , wherein the generative neural network model has been trained to perform the task also based on conditioning information associated with at least one of the current frame (t) and a future frame (>t).
3 . The decoder of claim 1 , including means configured to receive, in a same finite bitrate frame stream, the quantized latent frame and either conditioning information associated with at least one of the current frame and a future frame or an indication that such conditioning information is not included in the same finite bitrate frame stream, and wherein the decoder is configured to, if the same finite bitrate frame stream includes the indication, perform the task without the use of such conditioning information.
4 . The decoder of claim 2 , including means configured to receive, in a same finite bitrate frame stream, the quantized latent frame and legacy codec data, and further configured to reconstruct a signal from the legacy codec data as at least part of the conditioning information.
5 . The decoder of claim 1 , wherein the task includes predicting a current frame of the signal in the first domain, and wherein to generate the current frame of the signal in the first domain includes correcting the predicted current frame by use of the reconstructed latent frame mapped to the first domain.
6 . An encoder, configured to:
receive a current frame of a signal in a first domain;
use a generative neural network model to perform a task for which the generative neural network model has been trained based on conditioning information associated with at least one of the current frame (t) and a future frame (>t), where the task includes providing parameters for an invertible mapping from the first domain to a latent domain different from the first domain;
generate a latent frame by mapping at least part of the current frame of the signal to the latent domain using the invertible mapping, wherein the latent frame includes a representation of the current frame (t) of the signal in the latent domain;
generate a quantized latent frame based on the generated latent frame, wherein the quantized latent frame conveys information from the generated latent frame for supplementing information that is not reconstructed by a decoder-side generative neural network during a reconstruction process, and
generate a finite bitrate stream that includes the quantized latent frame.
7 . The encoder of claim 6 , wherein the encoder is further configured to output a further finite bitrate stream including such conditioning information.
8 . The encoder of claim 6 , further configured to output a same finite bitrate frame stream including the quantized latent frame and either conditioning information associated with at least one of the current frame and a future frame or an indication that such conditioning information is not included in the same finite bitrate stream.
9 . The encoder of claim 6 , further configured to:
generate a reconstructed latent frame from the quantized latent frame;
generate a reconstructed current frame of the signal in the first domain, which includes mapping the reconstructed latent frame to the first domain by use of an inverse of the invertible mapping, and
use the reconstructed current frame of the signal in the first domain to update a state of the generative neural network model.
10 . The encoder of claim 9 , wherein the invertible mapping includes an affine transformation.
11 . The encoder of claim 9 , configured to generate the invertible mapping by use of a flow model.
12 . The encoder of claim 6 , wherein the bitrate of the finite bitrate stream that includes the quantized latent frame is allocated based on a perceptual rate allocation model ( 150 ).
13 . The encoder of claim 9 , wherein the bitrate of the finite bitrate stream that includes the quantized latent frame is allocated based on a difference between the received current frame of the signal in the first domain and the reconstructed current frame of the signal in the first domain.
14 . The encoder of claim 6 , configured to generate the quantized latent frame by use of subtractive dithering followed by a gain.
15 . The encoder of claim 6 , configured to generate the quantized latent frame by selecting from a plurality of quantizers with different quantization step-sizes, including a zero-rate noise-fill.
16 . A method of decoding a current frame of a signal, including:
receiving a finite bitrate stream including a quantized latent frame, the quantized latent frame including a quantized representation of the current frame (t) of the signal in a latent domain different from a first domain;
generating a reconstructed latent frame from the quantized latent frame;
using a generative neural network model to perform a task for which the generative neural network model has been trained, the task including generating parameters for an invertible mapping from the latent domain to the first domain;
reconstructing a current frame of the signal in the first domain, including mapping the reconstructed latent frame to the first domain using the invertible mapping, wherein the mapping uses the generated parameters to reconstruct information lost during quantization, and
using the reconstructed current frame of the signal in the first domain to update a state of the generative neural network model.
17 . A method of encoding a current frame of a signal, including:
receiving a current frame of the signal in a first domain;
using a generative neural network model to perform a task for which the generative neural network model has been trained task based on conditioning information associated with at least one of the current frame (t) and a future frame (>t), the task including providing parameters for an invertible mapping from the first domain to a latent domain different from the first domain;
generating a latent frame by mapping at least part of the current frame of the signal to the latent domain using the invertible mapping, the latent frame including a representation of the current frame of the signal in the latent domain;
generating a quantized latent frame based on the generated latent frame, wherein the quantized latent frame conveys information from the generated latent frame for supplementing information that is not reconstructed by a decoder-side generative neural network during a reconstruction process; and
generating a finite bitrate stream including the quantized latent frame and such conditioning information.
18 . A non-transitory computer readable medium storing instructions operable, when executed by at least one computer processor belonging to a computer hardware, to cause the computer hardware to perform the method of decoding a current frame of a signal according to claim 16 .
19 . A non-transitory computer readable medium storing instructions operable, when executed by at least one computer processor belonging to a computer hardware, to cause the computer hardware to perform the method of encoding a current frame of a signal according to claim 17 .