Methods and systems for latent-aware image watermarking

1 . A computer-implemented method of digital image watermarking comprising:

performing a set of decoding operations to generate a watermarked image corresponding to a first latent feature tensor, comprising:

performing a first decoding operation in the set of decoding operations, comprising:

generating a first watermark embedding feature tensor that includes one or more representations of a watermark message embedded into the first latent feature tensor, and

upscaling the first watermark embedding feature tensor to generate a respective upscaled feature tensor;

performing one or more successive intermediate decoding operations in the set of decoding operations subsequent to the first decoding operation, each respective intermediate decoding operation comprising:

generating a respective intermediate watermark embedding feature tensor that includes one or more representations of the watermark message embedded into the upscaled feature tensor generated by a preceding decoding operation of the set of decoding operations, and

upscaling the respective intermediate watermark embedding feature tensor to generate a further upscaled feature tensor;

and

performing a final decoding operation in the set of decoding operations subsequent to a final intermediate decoding operation of the one or more successive intermediate decoding operations, the final decoding operation comprising:

generating a final watermark embedding feature tensor that includes one or more representations of the watermark message embedded into the upscaled feature tensor generated by the final intermediate decoding operation, and

upscaling the respective final watermark embedding feature tensor to generate the watermarked image.

2 . The method of claim 1 wherein each upscaling is performed by a respective layer of a multi-layer decoder model that is configured by a set of machine-learned parameters.

3 . The method of claim 1 wherein:

generating the first watermark embedding feature tensor comprises:

generating a first perturbation tensor that includes multiple perturbed representations of the watermark message combined with the first latent feature tensor, and

combining the first perturbation tensor and the first latent feature tensor to form the first watermark embedding feature tensor;

for each intermediate decoding operation, generating the respective intermediate watermark embedding feature tensor comprises:

generating a respective perturbation tensor that includes multiple perturbed representations of the watermark message combined with the upscaled feature tensor generated by the preceding decoding operation, and

combining the respective perturbation tensor and the upscaled feature tensor generated by the preceding decoding operation to form the respective intermediate watermark embedding feature tensor; and

generating the final watermark embedding feature tensor comprises:

generating a final perturbation tensor that includes multiple perturbed representations of the watermark message combined with the upscaled feature tensor generated by the final intermediate decoding operation, and

combining the final perturbation tensor and the upscaled feature tensor generated by the final intermediate decoding operation to form the final watermark embedding feature tensor.

4 . The method of claim 3 wherein:

generating the first perturbation tensor comprises:

combining repeated first representations of the watermark message to form a first watermark tensor having the same dimensions as the first latent feature tensor,

concatenating the first watermark tensor and the first latent feature tensor to form a first concatenated tensor, and

performing a first convolution operation on the first concatenated tensor to generate the first perturbation tensor, the first convolution operation being configured by a set of machine-learned first convolution operation parameters;

for each intermediate decoding operation, generating the respective perturbation tensor comprises:

combining repeated respective representations of the watermark message to form a respective watermark tensor having the same dimensions as the upscaled feature tensor generated by the preceding decoding operation,

concatenating the respective watermark tensor and the upscaled feature tensor generated by the preceding decoding operation to form a respective concatenated tensor, and

performing a respective convolution operation on the respective concatenated tensor to generate the respective perturbation tensor, the respective convolution operation being configured by a set of machine-learned respective convolution operation parameters; and

generating the final perturbation tensor comprises:

combining repeated final representations of the watermark message to form a final watermark tensor having the same dimensions as the upscaled feature tensor generated by the final intermediate decoding operation,

concatenating the final watermark tensor and the upscaled feature tensor generated by the final intermediate decoding operation to form a final concatenated tensor, and

performing a final convolution operation on the final concatenated tensor to generate the final perturbation tensor, the final convolution operation being configured by a set of machine-learned final convolution operation parameters.

5 . The method of claim 4 wherein the first latent feature tensor, the upscaled feature tensors, and the watermarked image are each represented as a respective matrix, each respective matrix having respective width, height and channel dimension sizes:

wherein when generating the first perturbation tensor, combining the repeated first representations of the watermark message to form the first watermark tensor comprises generating a first noise block representation of the watermark message by mapping the watermark message to a noise block that has a same channel dimension size as the first latent feature tensor and smaller width and height dimension sizes than the first latent feature tensor, the repeated first representations each being a repetition of the first noise block representation;

wherein, for each intermediate decoding operation, when generating the respective perturbation tensor, combining the repeated respective representations of the watermark message to form the respective watermark tensor comprises generating a respective noise block representation of the watermark message by mapping the watermark message to a noise block that has a same channel dimension size as, and smaller width and height dimension sizes than, the upscaled feature tensor generated by the preceding decoding operation, the repeated respective representations each being a repetition of the respective noise block representation; and

wherein when generating the final perturbation tensor, combining the repeated final representations of the watermark message to form the first watermark tensor comprises generating a final noise block representation of the watermark message by mapping the watermark message to a noise block that has a same channel dimension size, and smaller width and height dimension sizes than, the upscaled feature tensor generated by the final intermediate decoding operation.

6 . The method of claim 1 further comprising:

generating the first latent feature tensor by applying a latent diffusion process in respect of an input prompt that includes at least one of a text prompt and an input image.

7 . The method of claim 1 wherein a first latent code is generated by a latent encoder model in respect of an input prompt that includes an input image, and a latent diffusion process is applied to the first latent code to obtain the first latent feature tensor.

8 . The method of claim 7 wherein the set of decoding operations are performed by a watermark embedding decoder model, wherein the latent encoder model, the latent diffusion process and the watermark embedding decoder model collectively form a latent diffusion model configured to perform a task selected from a task group including: image super-resolution; inpainting; text-to-image generation; image manipulation; and unconditional generation.

9 . The method of claim 1 wherein the set of decoding operations is caused to be performed by a trained watermark embedding decoder model, the method further comprising obtaining the trained watermark embedding decoder model by:

receiving first model code and a set of pretrained upscaling parameters that collectively define a pretrained encoder model for causing a set of decoding operations that each include a respective upscaling function that are each configured by a respective subset of the pretrained upscaling parameters;

amending the first model code to define a further decoder model that includes, for each decoding operation, a respective watermark embedding feature tensor generating function that is configured by a respective subset of watermark embedding parameters; and

learning the respective subsets of watermark embedding parameters to enable the further encoder model to cause generation of watermarked images that are perceptually similar to images that are caused to be generated by the pretrained encoder model in respect of an input set of training latent feature tensors,

the trained watermark embedding decoder model being collectively defined by the amended the first model code, the set of pretrained upscaling parameters, and the learned respective subsets of watermark embedding parameters.

10 . The method of claim 1 further comprising:

processing an input image using a neural network that is configured to extract watermark messages to extract a message from the input image;

comparing the extracted message to the watermark message to determine if the extracted message and the watermark message match according to defined criteria; and

upon determining that the extracted message and the watermark message match according to the defined criteria, outputting an indication that the input image is watermarked with the watermark message.

11 . A system comprising:

one or more processors; and

one or more memories storing machine-executable instructions thereon which, when executed by the one or more processors, cause the system to perform a method comprising:

performing a set of decoding operations to generate a watermarked image corresponding to a first latent feature tensor, comprising:

performing a first decoding operation in the set of decoding operations, comprising:

generating a first watermark embedding feature tensor that includes one or more representations of a watermark message embedded into the first latent feature tensor, and

upscaling the first watermark embedding feature tensor to generate a respective upscaled feature tensor;

performing one or more successive intermediate decoding operations in the set of decoding operations subsequent to the first decoding operation, each respective intermediate decoding operation comprising:

generating a respective intermediate watermark embedding feature tensor that includes one or more representations of the watermark message embedded into the upscaled feature tensor generated by a preceding decoding operation of the set of decoding operations, and

upscaling the respective intermediate watermark embedding feature tensor to generate a further upscaled feature tensor; and

performing a final decoding operation in the set of decoding operations subsequent to a final intermediate decoding operation of the one or more successive intermediate decoding operations, the final decoding operation comprising:

generating a final watermark embedding feature tensor that includes one or more representations of the watermark message embedded into the upscaled feature tensor generated by the final intermediate decoding operation, and

upscaling the respective final watermark embedding feature tensor to generate the watermarked image.

12 . The system of claim 11 wherein each upscaling is performed by a respective layer of a multi-layer decoder model that is configured by a set of machine-learned parameters.

13 . The system method of claim 11 wherein:

generating the first watermark embedding feature tensor comprises:

generating a first perturbation tensor that includes multiple perturbed representations of the watermark message combined with the first latent feature tensor, and

combining the first perturbation tensor and the first latent feature tensor to form the first watermark embedding feature tensor;

for each intermediate decoding operation, generating the respective intermediate watermark embedding feature tensor comprises:

generating a respective perturbation tensor that includes multiple perturbed representations of the watermark message combined with the upscaled feature tensor generated by the preceding decoding operation, and

combining the respective perturbation tensor and the upscaled feature tensor generated by the preceding decoding operation to form the respective intermediate watermark embedding feature tensor; and

generating the final watermark embedding feature tensor comprises:

generating a final perturbation tensor that includes multiple perturbed representations of the watermark message combined with the upscaled feature tensor generated by the final intermediate decoding operation, and

combining the final perturbation tensor and the upscaled feature tensor generated by the final intermediate decoding operation to form the final watermark embedding feature tensor.

14 . The system of claim 13 wherein:

generating the first perturbation tensor comprises:

combining repeated first representations of the watermark message to form a first watermark tensor having the same dimensions as the first latent feature tensor,

concatenating the first watermark tensor and the first latent feature tensor to form a first concatenated tensor, and

performing a first convolution operation on the first concatenated tensor to generate the first perturbation tensor, the first convolution operation being configured by a set of machine-learned first convolution operation parameters;

for each intermediate decoding operation, generating the respective perturbation tensor comprises:

combining repeated respective representations of the watermark message to form a respective watermark tensor having the same dimensions as the upscaled feature tensor generated by the preceding decoding operation,

concatenating the respective watermark tensor and the upscaled feature tensor generated by the preceding decoding operation to form a respective concatenated tensor, and

performing a respective convolution operation on the respective concatenated tensor to generate the respective perturbation tensor, the respective convolution operation being configured by a set of machine-learned respective convolution operation parameters; and

generating the final perturbation tensor comprises:

combining repeated final representations of the watermark message to form a final watermark tensor having the same dimensions as the upscaled feature tensor generated by the final intermediate decoding operation,

concatenating the final watermark tensor and the upscaled feature tensor generated by the final intermediate decoding operation to form a final concatenated tensor, and

performing a final convolution operation on the final concatenated tensor to generate the final perturbation tensor, the final convolution operation being configured by a set of machine-learned final convolution operation parameters.

15 . The system of claim 14 wherein the first latent feature tensor, the upscaled feature tensors, and the watermarked image are each represented as a respective matrix, each respective matrix having respective width, height and channel dimension sizes:

wherein when generating the first perturbation tensor, combining the repeated first representations of the watermark message to form the first watermark tensor comprises generating a first noise block representation of the watermark message by mapping the watermark message to a noise block that has a same channel dimension size as the first latent feature tensor and smaller width and height dimension sizes than the first latent feature tensor, the repeated first representations each being a repetition of the first noise block representation;

wherein, for each intermediate decoding operation, when generating the respective perturbation tensor, combining the repeated respective representations of the watermark message to form the respective watermark tensor comprises generating a respective noise block representation of the watermark message by mapping the watermark message to a noise block that has a same channel dimension size as, and smaller width and height dimension sizes than, the upscaled feature tensor generated by the preceding decoding operation, the repeated respective representations each being a repetition of the respective noise block representation; and

wherein when generating the final perturbation tensor, combining the repeated final representations of the watermark message to form the first watermark tensor comprises generating a final noise block representation of the watermark message by mapping the watermark message to a noise block that has a same channel dimension size, and smaller width and height dimension sizes than, the upscaled feature tensor generated by the final intermediate decoding operation.

16 . The system of claim 11 further comprising:

generating the first latent feature tensor by applying a latent diffusion process in respect of an input prompt that includes at least one of a text prompt and an input image.

17 . The system of claim 11 wherein a first latent code is generated by a latent encoder model in respect of an input prompt that includes an input image, and a latent diffusion process is applied to the first latent code to obtain the first latent feature tensor.

18 . The system of claim 17 wherein the set of decoding operations are performed by a watermark embedding decoder model, wherein the latent encoder model, the latent diffusion process and the watermark embedding decoder model collectively form a latent diffusion model configured to perform a task selected from a task group including: image super-resolution; inpainting; text-to-image generation; image manipulation; and unconditional generation.

19 . The system of claim 11 wherein the set of decoding operations is caused to be performed by a trained watermark embedding decoder model, the method further comprising obtaining the trained watermark embedding decoder model by:

receiving first model code and a set of pretrained upscaling parameters that collectively define a pretrained encoder model for causing a set of decoding operations that each include a respective upscaling function that are each configured by a respective subset of the pretrained upscaling parameters;

amending the first model code to define a further encoder model that includes, for each decoding operation, a respective watermark embedding feature tensor generating function that is configured by a respective subset of watermark embedding parameters; and

learning the respective subsets of watermark embedding parameters to enable the further encoder model to cause generation of watermarked images that are perceptually similar to images that are caused to be generated by the pretrained encoder model in respect of an input set of training latent feature tensors,

the trained watermark embedding decoder model being collectively defined by the amended the first model code, the set of pretrained upscaling parameters, and the learned respective subsets of watermark embedding parameters.

20 . A non-transitory computer readable medium storing machine-executable instructions which, when executed by the one or more processors, cause the one or more processors to perform a method comprising:

performing a set of decoding operations to obtain a watermarked image corresponding to a first latent feature tensor, wherein one or more of the decoding operations in the set of decoding operations comprises:

generating a respective watermark embedding feature tensor that includes one or more representations of the watermark message embedded into an upscaled feature tensor generated based on the first latent feature tensor by a preceding decoding operation of the set of decoding operations, and

upscaling the respective watermark embedding feature tensor to generate a further upscaled feature tensor; and

outputting the watermarked image based on the further upscaled feature tensor.