Domain specific pre-training of cross modality transformer model

1. A method of pre-training a predictive model in a fashion domain, the method comprising:

receiving an input image of a fashion product and input text describing the fashion product;

tokenizing the input text to generate a plurality of input text tokens;

generating a plurality of patch groups of one or more image patches of the input image, each of the patch groups of one or more image patches covering a common portion of the input image;

masking one or more of the image patches and the input text tokens;

generating a training embedding from the image patches and the input text tokens; and

training a cross modality transformer-based model to predict the masked one or more of the image patches and the input text tokens using the training embedding.

2. The method of claim 1 , further comprising:

aligning one or more input text tokens with one or more aligned image patches based on a possible relevance of each of the input text tokens to respective ones of the aligned image patches.

3. The method of claim 2 , wherein masking the one or more of the image patches and the input text tokens comprises:

for one or more of the input text tokens aligned with one or more aligned image patches:

selecting either a respective input text token or one or more of the aligned image patches aligned with the respective input text token; and

masking the selected one of the respective input text token or the aligned image patches.

4. The method of claim 3 , wherein masking the selected input text token comprises replacing the input text token with one of:

a mask token; and

a token of a random word.

5. The method of claim 3 , wherein masking the selected one or more aligned image patches comprises replacing one or more of the aligned image patches with one or more of:

a patch covering a similar location of a similar image;

a greyed version of an aligned image patch; and

a blank version of the aligned image patch.

6. The method of claim 5 , wherein aligned image patches from a plurality of patch groups of the input image are masked.

7. The method of claim 2 , wherein masking the one or more of the image patches and the input text tokens comprises preferentially masking the aligned one or more input text tokens and the aligned image patches.

8. The method of claim 2 , wherein aligning one or more input text tokens with one or more aligned image patches comprises:

generating using a trained description model a text description of the input image, the trained description model provides an attention map for each token in the generated text description providing relevant regions of the input image that were relevant in generating the token;

matching one or more tokens of the generated text description with one or more tokens of the input text;

for each matched token of the input text, aligning the matched token with image patches corresponding to relevant regions of the attention map of the matched token.

9. The method of claim 8 , wherein each matched token is aligned with an image patch from a plurality of patch groups.

10. The method of claim 1 , wherein generating the training embedding comprises:

generating an image embedding from the image patches;

generating a text embedding from the input text tokens; and

generating the training embedding by concatenating the image embedding and the text embedding.

11. The method of claim 10 , wherein generating the image embedding comprises:

adding location information and segment information to each image patch; and

generating the image embedding from the location information, segment information and image patch, and

wherein generating the text embedding from the input text tokens comprises:

adding position information and segment information to each input text token; and

generating the text embedding from the position information, segment information and input text token.

12. The method of claim 1 , wherein training the transformer-based model comprises minimizing an overall loss function that is a summation of a plurality of individual loss functions for respective training tasks.

13. The method of claim 12 , wherein the training tasks and the individual loss functions include one or more of:

an aligned masked language modelling task that attempts to predict masked text tokens with an individual loss function, AMLM , defined as:

AMLM =ΣCE( t i , ( T,K,θ ) MSK_hidden ;

an image and text matching task that attempts to predict if the text describes the image with an individual loss function, ITM , defined as:

ITM =ΣCE( y m , ( T,K,θ ) CLS_hidden ;

a rotation recognition tasks that attempt to predict a rotation angle of an image patch with an individual loss, RR , function defined as:

RR =CE( y r , ( T,K,θ ) K 1 _hidden ;

a jigsaw puzzle solving task that attempts to predict a correct ordering of randomly ordered image patches with an individual loss function, JPS , defined as:

JPS =CE( y j , ( T,K,θ ) K 2 _hidden ;

a camouflage prediction task that attempts to predict an image patch that has been replaced with a corresponding image patch of a different image with an individual loss function, CP , defined as:

CP =CE( y c , ( T,K,θ ) K 3 _hidden ;

a grey-to-color modeling task that attempts to reconstruct color patch from a greyed image patch with an individual loss function, G2CM , defined as:

G2CM =ΣKLD( k 4i , ( T,K,θ ) K 4 _hidden ; and

a blank-to-color modeling task that attempts to reconstruct a patch from a blank patch with an individual loss function, B2CM , defined as:

B2CM =ΣKLD( k 5i , ( T,K,θ ) K 5 _hidden ,

where:

CE denotes the cross-entropy loss;

is the cross-modality transformer function;

(⋅) MSK_hidden denotes the hidden output of masked-out tokens;

(⋅) CLS_hidden denotes the hidden output of the CLS token;

(⋅) Kx_hidden denotes the hidden output of the image patches for an image patch level x;

T denotes the masked-out text sequence;

K denotes the masked-out kaleidoscope patch sequence;

θ is a rotation angle;

t i is the text token that has been masked out;

y m denotes the text and image match label

y r denotes the rotation angle

y j denotes the jigsaw permutation;

y c denotes the camouflaged patch;

KLD denotes the KL-divergence; and

k Ni denotes the masked-out patches of the K N kaleidoscope patches.

14. A non-transitory computer readable medium having stored thereon instructions, which when executed by one or more processors configure a computing system to perform a method of pre-training a predictive model in a fashion domain, the method comprising:

receiving an input image of a fashion product and input text describing the fashion product;

tokenizing the input text to generate a plurality of input text tokens;

generating a plurality of patch groups of one or more image patches of the input image, each of the patch groups of one or more image patches covering a common portion of the input image;

masking one or more of the image patches and the input text tokens;

generating a training embedding from the image patches and the input text tokens; and

training a cross modality transformer-based model to predict the masked one or more of the image patches and the input text tokens using the training embedding.

15. The computer readable medium of claim 14 , wherein the method further comprises:

aligning one or more input text tokens with one or more aligned image patches based on a possible relevance of each of the input text tokens to respective ones of the aligned image patches.

16. The computer readable medium of claim 15 , wherein masking the one or more of the image patches and the input text tokens comprises:

for one or more of the input text tokens aligned with one or more aligned image patches:

selecting either a respective input text token or one or more of the aligned image patches aligned with the respective input text token; and

masking the selected one of the respective input text token or the aligned image patches.

17. The computer readable medium of claim 15 , wherein masking the one or more of the image patches and the input text tokens comprises preferentially masking the aligned one or more input text tokens and the aligned image patches.

18. The computer readable medium of claim 15 , wherein aligning one or more input text tokens with one or more aligned image patches comprises:

generating using a trained description model a text description of the input image, the trained description model provides an attention map for each token in the generated text description providing relevant regions of the input image that were relevant in generating the token;

matching one or more tokens of the generated text description with one or more tokens of the input text;

for each matched token of the input text, aligning the matched token with image patches corresponding to relevant regions of the attention map of the matched token.

19. The computer readable medium of claim 18 , wherein each matched token is aligned with an image patch from a plurality of patch groups.

20. The computer readable medium of claim 14 , wherein training the transformer-based model comprises minimizing an overall loss function that is a summation of a plurality of individual loss functions for respective training tasks.

21. The computer readable medium of claim 20 , wherein the training tasks and the individual loss functions include one or more of:

an aligned masked language modelling task that attempts to predict masked text tokens with an individual loss function, AMLM , defined as:

AMLM =ΣCE( t i , ( T,K,θ ) MSK_hidden ;

an image and text matching task that attempts to predict if the text describes the image with an individual loss function, ITM , defined as:

ITM =ΣCE( y m , ( T,K,θ ) CLS_hidden ;

a rotation recognition tasks that attempt to predict a rotation angle of an image patch with an individual loss, RR , function defined as:

RR =CE( y r , ( T,K,θ ) K 1 _hidden ;

a jigsaw puzzle solving task that attempts to predict a correct ordering of randomly ordered image patches with an individual loss function, JPS , defined as:

JPS =CE( y j , ( T,K,θ ) K 2 _hidden ;

a camouflage prediction task that attempts to predict an image patch that has been replaced with a corresponding image patch of a different image with an individual loss function, CP , defined as:

CP =CE( y c , ( T,K,θ ) K 3 _hidden ;

a grey-to-color modeling task that attempts to reconstruct color patch from a greyed image patch with an individual loss function, G2CM , defined as:

G2CM =ΣKLD( k 4i , ( T,K,θ ) K 4 _hidden ; and

a blank-to-color modeling task that attempts to reconstruct a patch from a blank patch with an individual loss function, B2CM , defined as:

B2CM =ΣKLD( k 5i , ( T,K,θ ) K 5 _hidden ,

where:

CE denotes the cross-entropy loss;

is the cross-modality transformer function;

(⋅) MSK_hidden denotes the hidden output of masked-out tokens;

(⋅) CLS_hidden denotes the hidden output of the CLS token;

(⋅) Kx_hidden denotes the hidden output of the image patches for an image patch level x;

T denotes the masked-out text sequence;

K denotes the masked-out kaleidoscope patch sequence;

θ is a rotation angle;

t i is the text token that has been masked out;

y m denotes the text and image match label

y r denotes the rotation angle

y j denotes the jigsaw permutation;

y c denotes the camouflaged patch;

KLD denotes the KL-divergence; and

k Ni denotes the masked-out patches of the K N kaleidoscope patches.

22. A computer system comprising:

a processor for executing instructions;

a memory storing the instructions, which when executed by the processor configure the computer system to perform a method of pre-training a predictive model in a fashion domain, the method comprising:

receiving an input image of a fashion product and input text describing the fashion product;

tokenizing the input text to generate a plurality of input text tokens;

generating a plurality of patch groups of one or more image patches of the input image, each of the patch groups of one or more image patches covering a common portion of the input image;

masking one or more of the image patches and the input text tokens;

generating a training embedding from the image patches and the input text tokens; and

training a cross modality transformer-based model to predict the masked one or more of the image patches and the input text tokens using the training embedding.

23. The computer system of claim 22 , wherein the method further comprises:

aligning one or more input text tokens with one or more aligned image patches based on a possible relevance of each of the input text tokens to respective ones of the aligned image patches.

24. The computer system of claim 23 , wherein masking the one or more of the image patches and the input text tokens comprises:

for one or more of the input text tokens aligned with one or more aligned image patches:

selecting either a respective input text token or one or more of the aligned image patches aligned with the respective input text token; and

masking the selected one of the respective input text token or the aligned image patches.