End-to-end watermarking system
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for jointly training an encoder that generates a watermark and a decoder that decodes a data item encoded within the watermark. The training comprises obtaining a plurality of training images and data items. For each training image, a first watermark is generated using an encoder and a subsequent second watermark is generated by tiling two or more first watermarks. The training image is watermarked using the second watermark to generate a first error value and distortions are added to the watermarked image. A distortion detector predicts the distortions based on which the distorted image is modified. The modified image is decoded by the decoder to generate a predicted data item and a second error value. The training parameters of the encoder and decoder are adjusted based on the first and the second error value.
1. A computer-implemented method for jointly training an encoder machine learning model that generates a digital watermark that is embedded into an image and a decoder machine learning model that decodes a first data item encoded within the digital watermark that is embedded into the image, wherein the training comprises:
obtaining a first plurality of training images and a plurality of data items, wherein each data item in the plurality of data items is a data item that is to be encoded within a digital watermark to be embedded into a training image;
for each training image in the first plurality of training images:
obtaining a data item from the plurality of data items;
generating, using the encoder machine learning model to which the data item is provided as input, a first digital watermark that encodes the data item;
tiling two or more instances of the first digital watermark to generate a second digital watermark;
combining the second digital watermark with the training image to obtain a watermarked training image;
applying one or more distortions to the watermarked training image;
predicting, using a distortion detector machine learning model, the one or more distortions present in the watermarked training image;
modifying the watermarked training image based on the predicted one or more distortions while preserving the second digital watermark embedded in the distorted, watermarked training image; and
decoding, using the decoder machine learning model, the modified watermarked training image to obtain a decoded data item that is predicted to be embedded in the second digital watermark embedded in the distorted, watermarked training image;
determining a first error value based on the watermarked training image and the training image;
determining a second error value based on the decoded data item and the data item; and
adjusting one or more training parameters of the encoder machine learning model and the decoder machine learning model to minimize the first error value and the second error value.
2. The method of claim 1 , wherein determining the first error value based on the watermarked training image and the training image, comprises:
computing an L 2 loss value based on the watermarked training image and the training image.
3. The method of claim 1 , wherein determining the second error value based on the decoded data item and the data item, comprises:
computing a sigmoid cross entropy value based on the decoded data item and the data item.
4. The method of claim 1 , wherein the one or more distortions include image compression, gaussian noise, or image scaling.
5. The method of claim 1 , further comprising training the distortion detector machine learning model, including:
obtaining a second plurality of training images and a plurality of data items, wherein each data item in the plurality of data items is a data item that is to be encoded within a digital watermark to be embedded into a training image;
fixing weights of the encoder machine learning model; and
for each training image in the second plurality of training images:
generating, using the encoder machine learning model, a watermarked training image;
applying one or more distortions to the watermarked training image;
predicting, using the distortion detector machine learning model, the one or more distortions present in the distorted watermarked training image;
predicting, using the distortion detector machine learning model, one or more distortions present in the watermarked training image;
determining a third error value based on the predicted, one or more distortions present in the distorted watermarked training image and the predicted one or more distortions present in the watermarked training image; and
adjusting one or more training parameters of the distortion detector machine learning model to minimize the third error value.
6. The method of claim 1 , wherein determining the third error value, comprises:
computing an L 2 loss value based on the predicted, one or more distortions present in the distorted watermarked training image and the predicted, one or more distortions present in the watermarked training image.
7. A system, comprising:
at least one processor;
at least one memory storing instructions that, when executed by the at least one processor, perform operations comprising:
training an encoder machine learning model and a decoder machine learning model, wherein the encoder machine learning model generates a digital watermark that is embedded into an image and the decoder machine learning model decodes a first data item encoded within the digital watermark that is embedded into the image, wherein the training comprises:
obtaining a first plurality of training images and a plurality of data items, wherein each data item in the plurality of data items is a data item that is to be encoded within a digital watermark to be embedded into a training image; and
for each training image in the first plurality of training images:
obtaining a data item from the plurality of data items;
generating, using the encoder machine learning model to which the data item is provided as input, a first digital watermark that encodes the data item;
tiling two or more instances of the first digital watermark to generate a second digital watermark;
combining the second digital watermark with the training image to obtain a watermarked training image;
applying one or more distortions to the watermarked training image;
predicting, using a distortion detector machine learning model, the one or more distortions present in the watermarked training image;
modifying the watermarked training image based on the predicted one or more distortions while preserving the second digital watermark embedded in the distorted, watermarked training image; and
decoding, using the decoder machine learning model, the modified watermarked training image to obtain a decoded data item that is predicted to be embedded in the second digital watermark embedded in the distorted, watermarked training image;
determining a first error value based on the watermarked training image and the training image;
determining a second error value based on the decoded data item and the data item; and
adjusting one or more training parameters of the encoder machine learning model and the decoder machine learning model to minimize the first error value and the second error value.
8. The system of claim 7 , wherein determining the first error value based on the watermarked training image and the training image, comprises:
computing an L 2 loss value based on the watermarked training image and the training image.
9. The system of claim 7 , wherein determining the second error value based on the decoded data item and the data item, comprises:
computing a sigmoid cross entropy value based on the decoded data item and the data item.
10. The system of claim 7 , wherein the one or more distortions include image compression, gaussian noise, or image scaling.
11. The system of claim 7 , wherein the operations further comprising training the distortion detector machine learning model, including:
obtaining a second plurality of training images and a plurality of data items, wherein each data item in the plurality of data items is a data item that is to be encoded within a digital watermark to be embedded into a training image;
fixing weights of the encoder machine learning model; and
for each training image in the second plurality of training images:
generating, using the encoder machine learning model, a watermarked training image;
applying one or more distortions to the watermarked training image;
predicting, using the distortion detector machine learning model, the one or more distortions present in the distorted watermarked training image;
predicting, using the distortion detector machine learning model, one or more distortions present in the watermarked training image;
determining a third error value based on the predicted, one or more distortions present in the distorted watermarked training image and the predicted one or more distortions present in the watermarked training image; and
adjusting one or more training parameters of the distortion detector machine learning model to minimize the third error value.
12. The system of claim 7 , wherein determining the third error value, comprises:
computing an L 2 loss value based on the predicted, one or more distortions present in the distorted watermarked training image and the predicted, one or more distortions present in the watermarked training image.
13. A non-transitory computer readable medium storing instructions that, when executed by one or more data processing apparatus, cause the one or more data processing apparatus to perform operations for jointly training an encoder machine learning model that generates a digital watermark that is embedded into an image and a decoder machine learning model that decodes a first data item encoded within the digital watermark that is embedded into the image, wherein the training comprises:
obtaining a first plurality of training images and a plurality of data items, wherein each data item in the plurality of data items is a data item that is to be encoded within a digital watermark to be embedded into a training image;
for each training image in the first plurality of training images:
obtaining a data item from the plurality of data items;
generating, using the encoder machine learning model to which the data item is provided as input, a first digital watermark that encodes the data item;
tiling two or more instances of the first digital watermark to generate a second digital watermark;
combining the second digital watermark with the training image to obtain a watermarked training image;
applying one or more distortions to the watermarked training image;
predicting, using a distortion detector machine learning model, the one or more distortions present in the watermarked training image;
modifying the watermarked training image based on the predicted one or more distortions while preserving the second digital watermark embedded in the distorted, watermarked training image; and
decoding, using the decoder machine learning model, the modified watermarked training image to obtain a decoded data item that is predicted to be embedded in the second digital watermark embedded in the distorted, watermarked training image;
determining a first error value based on the watermarked training image and the training image;
determining a second error value based on the decoded data item and the data item; and
adjusting one or more training parameters of the encoder machine learning model and the decoder machine learning model to minimize the first error value and the second error value.
14. The non-transitory computer readable medium of claim 13 , wherein determining the first error value based on the watermarked training image and the training image, comprises:
computing an L 2 loss value based on the watermarked training image and the training image.
15. The non-transitory computer readable medium of claim 13 , wherein determining the second error value based on the decoded data item and the data item, comprises:
computing a sigmoid cross entropy value based on the decoded data item and the data item.
16. The non-transitory computer readable medium of claim 13 , wherein the one or more distortions include image compression, gaussian noise, or image scaling.
17. The non-transitory computer readable medium of claim 13 , further comprising training the distortion detector machine learning model, including:
obtaining a second plurality of training images and a plurality of data items, wherein each data item in the plurality of data items is a data item that is to be encoded within a digital watermark to be embedded into a training image;
fixing weights of the encoder machine learning model; and
for each training image in the second plurality of training images:
generating, using the encoder machine learning model, a watermarked training image;
applying one or more distortions to the watermarked training image;
predicting, using the distortion detector machine learning model, the one or more distortions present in the distorted watermarked training image;
predicting, using the distortion detector machine learning model, one or more distortions present in the watermarked training image;
determining a third error value based on the predicted, one or more distortions present in the distorted watermarked training image and the predicted one or more distortions present in the watermarked training image; and
adjusting one or more training parameters of the distortion detector machine learning model to minimize the third error value.
18. The non-transitory computer readable medium of claim 13 , wherein determining the third error value, comprises:
computing an L 2 loss value based on the predicted, one or more distortions present in the distorted watermarked training image and the predicted, one or more distortions present in the watermarked training image.