Verifying the provenance of a digital object using watermarking and embeddings
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for verifying the provenance of a digital object generated by a neural network, such as an image or audio object. Also methods, systems, and apparatus, including computer programs, for training a watermarking neural network and a watermark decoding neural network. The described techniques make efficient use of computing resources and are robust to attack.
1 . A computer-implemented method of training a watermarking system comprising a watermark generation neural network and a watermark decoding neural network, comprising:
for a plurality of images:
processing the image using the watermark generation neural network to generate a watermark for the image;
combining the watermark for the image and the image to obtain a watermarked image;
applying an adversarial transformation to the watermarked image, to generate a perturbed watermarked data object;
applying an adversarial transformation to the image to generate a perturbed data object;
processing the perturbed watermarked data object using the watermark decoding neural network to generate a first watermarking signal that indicates whether or not the perturbed watermarked data object is predicted to be watermarked;
processing the perturbed data object using the watermark decoding neural network to generate a second watermarking signal that indicates whether or not the perturbed data object is predicted to be watermarked; and
jointly training the watermark decoding neural network and the watermark generation neural network, using the first watermarking signal and the second watermarking signal, to distinguish between the perturbed watermarked data object and the perturbed data object.
2 . The method of claim 1 , wherein jointly training the watermark decoding neural network and the watermark generation neural network comprises backpropagating gradients of a classification-based objective function that has a value that depends on classifying the first watermarking signal as indicating that the perturbed watermarked data object is watermarked and classifying the second watermarking signal as indicating that the perturbed data object as not watermarked.
3 . The method of claim 1 , comprising applying the adversarial transformation to the image to generate the perturbed data object, wherein applying the adversarial transformation comprises changing the values of pixels in the image data object.
4 . A computer implemented method of verifying the provenance of a digital object, wherein the digital object comprises an image, the method comprising:
maintaining an object verification system comprising:
a first interface to receive a digital object or a request to generate a digital object;
a second interface to provide a watermarked digital object for use;
an embedding neural network configured to process the digital object to generate an embedding of the digital object; and
an object verification database configured to store at least the embedding of the digital object; the method further comprising:
receiving a query digital object for verification, wherein the query digital object comprises an image;
processing the query digital object using a watermark decoding neural network to generate a watermarking signal for the query digital object;
processing the query digital object using the embedding neural network to generate a query embedding of the query digital object;
interrogating the object verification database using the query embedding to determine a set of one or more similarity scores for a corresponding set of one or more stored embeddings of digital objects that are similar to the query digital object; and
verifying a provenance of the query digital object based on a combination of the watermarking signal and the set of one or more similarity scores.
5 . The method of claim 4 , wherein the object verification system is an object generation and verification system, wherein the request comprises a request to generate the digital object, wherein maintaining the object verification system further comprises maintaining a generative neural network configured to process the request to generate the digital object in accordance with the request; the method further comprising:
receiving a request to generate the digital object;
processing the request using the generative neural network to generate the digital object;
processing the digital object using the watermarking neural network to generate the watermarked digital object by:
processing the digital object using a watermark generation neural network to generate a watermark for the digital object,
combining the watermark and the digital object to obtain a watermarked digital object;
providing the watermarked digital object for use;
processing the digital object using the embedding neural network to generate the embedding of the digital object;
storing the embedding of the digital object in the object verification database; and
verifying the provenance of the query digital object as generated by the object generation and verification system conditional upon the query embedding matching the embedding of the digital object stored in the object verification database.
6 . The method of claim 5 , further comprising:
storing the digital object in the object verification database; and
using the stored digital object to detect attempted removal of a watermark from the query digital object.
7 . The method of claim 4 , wherein the watermarking neural network and the watermark decoding neural network have been jointly trained to generate the watermarking signal under an adversarial perturbation of the watermarked digital object.
8 . The method of claim 4 , wherein the watermarking neural network comprises a neural network with a U-Net architecture and has more trainable parameters than the watermarking decoding neural network.
9 . A computer-implemented method of watermarking an image, the method comprising:
receiving the image, the image having an original size;
generating a resized image by resizing the image to have a target size;
processing the resized image using a watermark generation neural network to generate a watermark for the resized image;
generating a watermarked version of the resized image using the watermark for the image generated by the watermark generation neural network and the resized image; and
resizing the watermarked image to the original size.