User-guided variable-rate image compression

1. A computer-implemented method comprising:

receiving, via a user interface, a request to compress an image, the request including the image;

causing a segmentation mask corresponding to the image to be displayed in the user interface;

receiving, via the user interface, importance data for each region of the segmentation mask corresponding to the image and a target bitrate;

providing the image, the corresponding importance data, and the target bitrate to a compression network;

generating, by the compression network, a learned importance map and a representation of the image;

generating, by the compression network, a compressed representation of the image based on the learned importance map and the representation of the image;

dynamically updating the importance data for one or more regions of the segmentation mask using the user interface; and

generating, by the compression network, a second compressed representation of the image based on a second learned importance map and a second representation of the image.

2. The computer-implemented method of claim 1 , wherein the importance data includes a segmentation mask corresponding to the image and importance values for each region of the segmentation mask.

3. The computer-implemented method of claim 2 , wherein providing the image, the corresponding importance data, and the target bitrate to a compression network, further comprises:

concatenating the image and the segmentation mask; and

providing the concatenated image and segmentation mask to an encoder of the compression network.

4. The computer-implemented method of claim 3 , wherein generating, by the compression network, a learned importance map and a representation of the image, further comprises:

providing an intermediate representation generated by the encoder to an importance map network of the compression network;

generating, by the encoder, the representation of the image; and

generating, by the importance map network, the learned importance map from the intermediate representation.

5. The computer-implemented method of claim 4 , wherein the learned importance map defines an importance value for each pixel of the image and wherein within a region of the segmentation mask, high detail pixels receive a higher importance value than low detail pixels.

6. The computer-implemented method of claim 1 , wherein generating, by the compression network, a compressed representation of the image based on the learned importance map and the representation of the image, further comprises:

determining an element-wise product of the learned importance map and the representation of the image.

7. The computer-implemented method of claim 1 , further comprising:

receiving a request for the image, the request including at least a reference to the compressed representation of the image;

providing the compressed representation of the image to a reconstruction network; and

returning a reconstructed image generated by the reconstruction network.

8. The computer-implemented method of claim 1 , wherein a discriminator network and one or more loss functions are used to train the compression network to generate compressed representations at variable bitrates based on user-provided importance values and a reconstruction network to generate photorealistic images from the compressed representations.

9. The computer-implemented method of claim 8 , wherein the one or more loss functions includes an equivalence distortion loss function that trains the compression network to learn to allocate bits optimally in the learned importance map while staying within the target bitrate.

10. A non-transitory computer-readable storage medium including instructions stored thereon which, when executed by at least one processor, cause the at least one processor to:

receive, via a user interface, a request to compress an image, the request including the image;

cause a segmentation mask corresponding to the image to be displayed in the user interface;

receive, via the user interface, importance data for each region of the segmentation mask corresponding to the image and a target bitrate;

provide the image, the corresponding importance data, and the target bitrate to a compression network;

generate, by the compression network, a learned importance map and a representation of the image;

generate, by the compression network, a compressed representation of the image based on the learned importance map and the representation of the image;

dynamically updating the importance data for one or more regions of the segmentation mask using the user interface; and

generating, by the compression network, a second compressed representation of the image based on a second learned importance map and a second representation of the image.

11. The non-transitory computer-readable storage medium of claim 10 , wherein the importance data includes a segmentation mask corresponding to the image and importance values for each region of the segmentation mask.

12. The non-transitory computer-readable storage medium of claim 11 , wherein to provide the image, the corresponding importance data, and the target bitrate to a compression network, the instructions, when executed, further cause the at least one processor to:

concatenate the image and the segmentation mask; and

provide the concatenated image and segmentation mask to an encoder of the compression network.

13. The non-transitory computer-readable storage medium of claim 12 , wherein to generate, by the compression network, a learned importance map and a representation of the image, the instructions, when executed, further cause the at least one processor to:

provide an intermediate representation generated by the encoder to an importance map network of the compression network;

generate, by the encoder, the representation of the image; and

generate, by the importance map network, the learned importance map from the intermediate representation.

14. The non-transitory computer-readable storage medium of claim 13 , wherein the learned importance map defines an importance value for each pixel of the image and wherein within a region of the segmentation mask, high detail pixels receive a higher importance value than low detail pixels.

15. The non-transitory computer-readable storage medium of claim 10 , wherein to generate, by the compression network, a compressed representation of the image based on the learned importance map and the representation of the image, the instructions, when executed, further cause the at least one processor to:

determine an element-wise product of the learned importance map and the representation of the image.

16. The non-transitory computer-readable storage medium of claim 10 , wherein a discriminator network and one or more loss functions are used to train the compression network to generate compressed representations at variable bitrates based on user-provided importance values and a reconstruction network to generate photorealistic images from the compressed representations.

17. The non-transitory computer-readable storage medium of claim 16 , wherein the one or more loss functions includes an equivalence distortion loss function that trains the compression network to learn to allocate bits optimally in the learned importance map while staying within the target bitrate.

18. A system comprising:

at least one processor; and

a memory storing instructions which, when executed by the at least one processor, cause the system to:

receive, via a user interface, a request to compress an image, the request including the image;

cause a segmentation mask corresponding to the image to be displayed in the user interface;

receive, via the user interface, importance data for each region of the segmentation mask corresponding to the image and a target bitrate;

provide the image, the corresponding importance data, and the target bitrate to a compression network;

generate, by the compression network, a learned importance map and a representation of the image;

generate, by the compression network, a compressed representation of the image based on the learned importance map and the representation of the image;

dynamically updating the importance data for one or more regions of the segmentation mask using the user interface; and

generating, by the compression network, a second compressed representation of the image based on a second learned importance map and a second representation of the image.

19. The system of claim 18 , wherein the importance data includes a segmentation mask corresponding to the image and importance values for each region of the segmentation mask.