Method and system for adapting a diffusion model
A method and a server for adapting a diffusion model are provided. The method comprising: receiving a given plurality of images of a given object; receiving a respective textual description for the given object; generating, based on the given plurality of images, a training set of data including a plurality of training digital objects; sampling, from the plurality of training digital objects, a given reference training digital object for using during a validation step; in cycles, executing the validation step by feeding a given reference training digital object to the diffusion model; in response to a convergence metric associated with a loss function at a given cycle of executing the validation step being below a predetermined threshold, terminating the adapting the diffusion model.
1 . A computer-implemented method for adapting a diffusion model, which has been pre-trained to generate images of objects based on textual descriptions thereof, to a given plurality of images of at least one object that the diffusion model has been pre-trained to generate, the method comprising:
receiving the given plurality of images, a given image of the given plurality of images being representative of a given object;
receiving a respective textual description for the given object depicted in the given image;
applying, to the given image, an image encoding algorithm, thereby generating a respective image vector for the given image;
applying, to the respective textual description associated with the given image, a text encoding algorithm, thereby generating a respective text vector for the respective textual description;
applying, to the respective image vector, during a respective number of diffusion steps, a respective level of a diffusion noise, thereby generating a respective noisy image vector for the given image;
compiling a training set of data including a plurality of training digital objects, a given one of which includes: (i) a set of images representative of the given object and respective noisy image vectors associated therewith; (ii) a respective set of textual descriptions of the given object and respective text vectors associated therewith;
sampling, from the plurality of training digital objects, at least one reference training digital object for using during a validation step;
excluding the at least one reference training digital object from the plurality of training digital objects, thereby generating an adaptation plurality of training digital objects;
feeding, at each training iteration, to the diffusion model, a respective training digital object of the adaptation plurality of training digital objects, thereby causing the diffusion model to generate, from the respective noisy image vector, a synthetic image representative of a respective object;
in each cycle having a predetermined number of training iterations, executing the validation step by feeding a respective reference training digital object from the at least one reference training digital object to the diffusion model, thereby causing the diffusion model to generate a respective synthetic reference image, wherein the at least one reference training digital object does not change between cycles;
determining, for a given cycle, a value of a loss function indicative of a difference between the given image of the respective reference training digital object and the respective synthetic reference image at the given cycle; and
in response to a convergence metric associated with the loss function being below a predetermined convergence threshold at the given cycle, terminating the adapting the diffusion model without using any further training digital objects.
2 . The method of claim 1 , wherein the text encoding algorithm comprises a Transformer-based machine-learning algorithm (MLA) that has been trained to generate text embeddings from input text.
3 . The method of claim 1 , wherein the loss function is expressed by a following equation:
ℒ
det
=
ϵ
-
ϵ
θ
(
z
t
(
ε
(
x
)
,
ϵ
)
,
c
(
y
)
,
t
)
2
2
,
where ϵ θ is the respective synthetic reference image generated at the given cycle in response to feeding, to the diffusion model, the given image x of the respective reference training digital object;
ε(x) is the respective noisy image vector of the given image x;
ϵ is the respective level of the diffusion noise applied to the given image x;
c(y) is the respective text vector for the respective textual description y of the given object depicted in the given image x; and
t is the respective number of diffusion steps over which the respective noisy image vector of the given image x has been generated.
4 . The method of claim 3 , wherein each one of (i) the respective noisy image vector of the given image; (ii) the respective level of the diffusion noise; (iii) the respective textual description for the given object depicted in the given image; and (iv) the respective number of diffusion steps does not change between cycles.
5 . The method of claim 3 , wherein at least one of (i) the respective noisy image vector of the given image; (ii) the respective level of the diffusion noise; (iii) the respective textual description for the given object depicted in the given image; and (iv) the respective number of diffusion steps does not change between cycles while other ones of (i) the respective noisy image vector of the given image; (ii) the respective level of the diffusion noise; (iii) the respective textual description for the given object depicted in the given image; and (iv) the respective number of diffusion steps are re-determined at each cycle.
6 . The method of claim 1 , wherein the set of images of the respective reference training digital object includes a predetermined number of images of the respective object.
7 . The method of claim 1 , wherein the convergence metric is representative of a ratio between: (i) a local variance of values of the loss function over a first number of cycles; and (ii) a global variance of values of the loss function over a second number of cycles, the second number of cycles being greater than the first number of cycles.
8 . The method of claim 1 , wherein the convergence metric is representative of a ratio between: (i) a current exponential moving average (EMA) of values of the loss function at the given cycle; and (ii) a past EMA, determined over a predetermined number of past cycles.
9 . The method of claim 8 , wherein the convergence metric is expressed by a following equation:
ℳ
EMA
=
EMA
(
t
)
-
EMA
(
t
-
n
)
EMA
(
t
-
n
)
,
where EMA(t) is the current EMA at the given cycle;
EMA(t−n) is the past EMA;
t is an index representing the given cycle; and
n is the predetermined number of cycles.
10 . The method of claim 1 , wherein the convergence metric comprises a Hall criterion that is representative of a ratio between: (i) a difference between a maximum value and minimum value of the loss function over a given number of cycles; and (ii) a mean value of the loss function over the given number of cycles.
11 . The method of claim 10 , wherein the convergence metric is expressed by a following equation:
ℳ
Hall
=
max
(
ℒ
det
n
)
-
min
(
ℒ
det
n
)
mean
(
ℒ
det
n
)
.
where det n is a given value of the loss function over the given number of cycles n.
12 . The method of claim 1 , wherein the convergence metric is a Trend metric that is representative of a slope of a graph of a linear regression that has been trained to predict values of the loss function based on past values thereof.
13 . The method of claim 1 , wherein the diffusion model is a Stable diffusion model.
14 . A server for adapting a diffusion model, which has been pre-trained to generate images of objects based on textual descriptions thereof, to a given plurality of images of at least one object that the diffusion model has been pre-trained to generate, the server comprising at least one processor and at least one non-transitory computer-readable memory storing executable instructions, which, when executed by the at least one processor, cause the server to:
receive the given plurality of images, a given image of the given plurality of images being representative of a given object;
receive a respective textual description for the given object depicted in the given image;
apply, to the given image, an image encoding algorithm, thereby generating a respective image vector for the given image;
apply, to the respective textual description associated with the given image, a text encoding algorithm, thereby generating a respective text vector for the respective textual description;
apply, to the respective image vector, during a respective number of diffusion steps, a respective level of a diffusion noise, thereby generating a respective noisy image vector for the given image;
compile a training set of data including a plurality of training digital objects, a given one of which includes: (i) a set of images representative of the given object and respective noisy image vectors associated therewith; (ii) a respective set of textual descriptions of the given object and respective text vectors associated therewith;
sample, from the plurality of training digital objects, at least one reference training digital object for using during a validation step;
exclude the at least one reference training digital object from the plurality of training digital objects, thereby generating an adaptation plurality of training digital objects;
feed, at each training iteration, to the diffusion model, a respective training digital object of the adaptation plurality of training digital objects, thereby causing the diffusion model to generate, from the respective noisy image vector, a synthetic image representative of a respective object;
in each cycle having a predetermined number of training iterations, execute the validation step by feeding a respective reference training digital object from the at least one reference training digital object to the diffusion model, thereby causing the diffusion model to generate a respective synthetic reference image, wherein the at least one reference training digital object does not change between cycles;
determine, for a given cycle, a value of a loss function indicative of a difference between the given image of the respective reference training digital object and the respective synthetic reference image at the given cycle; and
in response to a convergence metric associated with the loss function being below a predetermined convergence threshold at the given cycle, terminate the adapting the diffusion model without using any further training digital objects.
15 . The server of claim 14 , wherein the text encoding algorithm comprises a Transformer-based machine-learning algorithm (MLA) that has been trained to generate text embeddings from input text.
16 . The server of claim 14 , wherein the loss function is expressed by a following equation:
ℒ
det
=
ϵ
-
ϵ
θ
(
z
t
(
ε
(
x
)
,
ϵ
)
,
c
(
y
)
,
t
)
2
2
,
where ϵ θ is the respective synthetic reference image generated at the given cycle in response to feeding, to the diffusion model, the given image x of the respective reference training digital object;
ε(x) is the respective noisy image vector of the given image x;
ϵ is the respective level of the diffusion noise applied to the given image x;
c(y) is the respective text vector for the respective textual description y of the given object depicted in the given image x; and
t is the respective number of diffusion steps over which the respective noisy image vector of the given image x has been generated.
17 . The server of claim 16 , wherein each one of (i) the respective noisy image vector of the given image; (ii) the respective level of the diffusion noise; (iii) the respective textual description for the given object depicted in the given image; and (iv) the respective number of diffusion steps does not change between cycles.
18 . The server of claim 16 , wherein at least one of (i) the respective noisy image vector of the given image; (ii) the respective level of the diffusion noise; (iii) the respective textual description for the given object depicted in the given image; and (iv) the respective number of diffusion steps does not change between cycles while other ones of (i) the respective noisy image vector of the given image; (ii) the respective level of the diffusion noise; (iii) the respective textual description for the given object depicted in the given image; and (iv) the respective number of diffusion steps are re-determined at each cycle.
19 . The server of claim 14 , wherein the set of images of the respective reference training digital object includes a predetermined number of images of the respective object.
20 . The server of claim 14 , wherein the convergence metric is representative of a ratio between: (i) a local variance of values of the loss function over a first number of cycles; and (ii) a global variance of values of the loss function over a second number of cycles, the second number of cycles being greater than the first number of cycles.