Stop code tolerant image compression neural networks
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for image compression and reconstruction. A request to generate an encoded representation of an input image is received. The encoded representation of the input image is then generated. The encoded representation includes a respective set of binary codes at each iteration. Generating the set of binary codes for the iteration from an initial set of binary includes: for any tiles that have already been masked off during any previous iteration, masking off the tile. For any tiles that have not yet been masked off during any of the previous iterations, a determination is made as to whether a reconstruction error of the tile when reconstructed from binary codes at the previous iterations satisfies an error threshold. When the reconstruction quality satisfies the error threshold, the tile is masked off.
1. A computer-implemented image compression and reconstruction method comprising:
receiving a request to generate an encoded representation of an input image; and
generating the encoded representation of the input image, wherein the encoded representation includes a respective set of binary codes at each of a plurality of iterations, wherein each set of binary codes includes a respective subset of binary codes for each of a plurality of tiles in the input image, and wherein the generating comprises, for each of the plurality of iterations:
processing an encoder input for the iteration using an encoder neural network, wherein the encoder neural network is configured to process the encoder input to generate an initial set of binary codes for the iteration, the initial set of binary codes including a respective subset of initial binary codes for each of the plurality of tiles; and
generating the set of binary codes for the iteration from the initial set of binary codes, comprising:
for any tiles that have already been masked off during any previous iteration of the plurality of iterations:
masking off the tile by setting each of the binary codes for the tile in the set of binary codes to zero;
for any tiles that have not yet been masked off during any of the previous iterations of the plurality of iterations:
determining whether a reconstruction error of the tile when reconstructed from binary codes at the previous iterations satisfies an error threshold; and
when the reconstruction quality satisfies the error threshold, masking off the tile by setting each of the binary codes for the tile in the set of binary codes to zero.
2. The method of claim 1 , wherein generating the set of binary codes for the iteration further comprises:
when the reconstruction error does not satisfy the error threshold, refraining from modifying the initial binary codes for the tile.
3. The method of claim 1 , further comprising:
compressing the input image by compressing the binary codes in the encoded representation using a data compression algorithm.
4. The method of claim 3 , wherein the data compression algorithm is a trained entropy coder.
5. The method of claim 3 , further comprising:
transmitting the compressed input image to a decoder system for decompression of the input image.
6. The method of claim 1 , wherein the encoder neural network is a recurrent neural network with a binarizer output layer.
7. The method of claim 6 , wherein the encoder input for a first iteration of the plurality of iterations is the input image.
8. The method of claim 6 , wherein the encoder input for each iteration of the plurality of iterations after the first iteration is a residual image between (i) the input image and (ii) a reconstruction generated by a decoder neural network from the set of binary codes at the previous iteration, wherein the decoder neural network is a recurrent neural network that is configured to, at each of the plurality of iterations, receive a decoder input comprising the set of binary codes at the iteration and to process the decoder input to generate a reconstruction of the input image.
9. The method of claim 8 , wherein determining whether a reconstruction error of the tile when reconstructed from binary codes at the previous iterations satisfies an error threshold comprises:
determining the reconstruction error for a portion corresponding to the tile in the residual image that is the encoder input at the iteration.
10. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:
receiving a request to generate an encoded representation of an input image; and
generating the encoded representation of the input image, wherein the encoded representation includes a respective set of binary codes at each of a plurality of iterations, wherein each set of binary codes includes a respective subset of binary codes for each of a plurality of tiles in the input image, and wherein the generating comprises, for each of the plurality of iterations:
processing an encoder input for the iteration using an encoder neural network, wherein the encoder neural network is configured to process the encoder input to generate an initial set of binary codes for the iteration, the initial set of binary codes including a respective subset of initial binary codes for each of the plurality of tiles; and
generating the set of binary codes for the iteration from the initial set of binary codes, comprising:
for any tiles that have already been masked off during any previous iteration of the plurality of iterations:
masking off the tile by setting each of the binary codes for the tile in the set of binary codes to zero;
for any tiles that have not yet been masked off during any of the previous iterations of the plurality of iterations:
determining whether a reconstruction error of the tile when reconstructed from binary codes at the previous iterations satisfies an error threshold; and
when the reconstruction quality satisfies the error threshold, masking off the tile by setting each of the binary codes for the tile in the set of binary codes to zero.
11. The method of claim 10 , wherein generating the set of binary codes for the iteration further comprises:
when the reconstruction error does not satisfy the error threshold, refraining from modifying the initial binary codes for the tile.
12. One or more computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
receiving a request to generate an encoded representation of an input image; and
generating the encoded representation of the input image, wherein the encoded representation includes a respective set of binary codes at each of a plurality of iterations, wherein each set of binary codes includes a respective subset of binary codes for each of a plurality of tiles in the input image, and wherein the generating comprises, for each of the plurality of iterations:
processing an encoder input for the iteration using an encoder neural network, wherein the encoder neural network is configured to process the encoder input to generate an initial set of binary codes for the iteration, the initial set of binary codes including a respective subset of initial binary codes for each of the plurality of tiles; and
generating the set of binary codes for the iteration from the initial set of binary codes, comprising:
for any tiles that have already been masked off during any previous iteration of the plurality of iterations:
masking off the tile by setting each of the binary codes for the tile in the set of binary codes to zero;
for any tiles that have not yet been masked off during any of the previous iterations of the plurality of iterations:
determining whether a reconstruction error of the tile when reconstructed from binary codes at the previous iterations satisfies an error threshold; and
when the reconstruction quality satisfies the error threshold, masking off the tile by setting each of the binary codes for the tile in the set of binary codes to zero.