Methods and apparatus for multi-modal prediction using a trained statistical model

1. A method for predicting an association between input data in a first modality and data in a second modality, the method comprising:

accessing a statistical model trained to represent links between data having a plurality of modalities including the first modality and the second modality, the statistical model comprising a plurality of encoders and decoders, each of which is trained to process data for one of the plurality of modalities, and a joint-modality representation coupling the plurality of encoders and decoders, the plurality of encoders and decoders including an encoder/decoder pair for each of the plurality of modalities, wherein:

the encoder of the encoder/decoder pair is trained to project data for the modality into a latent representation space associated with the join-modality representation; and

the decoder of the encoder/decoder pair is trained to decode input from the common representation space to generate output in a representation space of the modality;

selecting, based on the first modality and the second modality, an encoder/decoder pair or a pair of encoders, from among the plurality of encoders and decoders; and

predicting the association between the input data in the first modality and the data in the second modality, the predicting comprising:

processing the input data with the joint-modality representation and the selected encoder/decoder pair or pair of encoders to obtain a projection of the input data in a representation space of the model, the representation space comprising a representation space of the second modality or the latent representation space associated with the joint-modality representation; and

predicting the association between the input data in the first modality and the data in the second modality by determining, in the representation space, a measure of similarity between the projection of the input data and the data in the second modality.

2. The method of claim 1 , wherein selecting, based on the first modality and the second modality, an encoder/decoder pair or a pair of encoders, from among the plurality of encoders and decoders comprises:

selecting an encoder trained to process data for the first modality and a decoder trained to process data to project the data into the representation space of the second modality.

3. The method of claim 2 , wherein the representation space is the representation space for the second modality and predicting the association between the input data in the first modality and the data in the second modality comprises:

predicting the association between the input data in the first modality and the data in the second modality in the representation space for the second modality.

4. The method of 3 , wherein predicting the association between the input data in the first modality and the data in the second modality comprises:

processing the input data in the first modality using the selected encoder to obtain a projection of the input data in the common representation of the statistical model;

processing, using the decoder trained to process data to project the data into the representation space of the second modality, to obtain an output vector in the representation space for the second modality;

projecting the data from the second modality into the representation space for the second modality to produce a plurality of projected vectors; and

predicting the association between the input data in the first modality and the data in the second modality based on a comparison of the output vector and the projected vectors in the representation space for the second modality.

5. The method of claim 4 , wherein predicting the association between the input data in the first modality and the data in the second modality comprises:

calculating a distance between the output vector and each of the plurality of projected vectors; and

predicting the association based on the calculated distances.

6. The method of claim 5 , wherein calculating a distance comprises calculating a Euclidean distance.

7. The method of claim 1 , wherein selecting, based on the first modality and the second modality, an encoder/decoder pair or a pair of encoders, from among the plurality of encoders and decoders comprises:

selecting a first encoder trained to process data for the first modality and a second encoder trained to process data for the second modality.

8. The method of claim 7 , wherein the representation space is the latent representation space associated with the joint-modality representation and predicting the association between the input data in the first modality and the data in the second modality comprises:

predicting the association between the input data in the first modality and the data in the second modality in the latent representation space associated with the joint-modality representation.

9. The method of claim 8 , wherein predicting the association between the input data in the first modality and the data in the second modality comprises:

providing as input to the first encoder, the input data to produce a first modality feature vector in the latent representation space;

providing as input to the second encoder, the data for the second modality to produce a plurality of second modality feature vectors in the latent representation space; and

predicting the association between the input data in the first modality and the data in the second modality based on a comparison of the first modality feature vector and the plurality of second modality feature vectors in the latent representation space.

10. The method of claim 9 , wherein predicting the association between the input data in the first modality and the data in the second modality comprises:

calculating a distance between the first modality feature vector and each of the plurality of second modality feature vectors in the latent representation space; and

predicting the association based on the calculated distances.

11. The method of claim 10 , wherein calculating a distance comprises calculating a Euclidean distance.

12. The method of claim 1 , wherein encoder/decoder pairs for the plurality of modalities each comprise parameter values trained using a self-supervised learning technique.

13. A computer system, comprising:

at least one computer processor; and

at least one storage device encoded with a plurality of instructions that, when executed by the at least one computer processor perform a method of predicting an association between input data in a first modality and data in a second modality using a statistical model trained to represent interactions between data having a plurality of modalities including the first modality and the second modality, the statistical model comprising a plurality of encoders and decoders, each of which is trained to process data for one of the plurality of modalities, and a joint-modality representation coupling the plurality of encoders and decoders, the plurality of encoders and decoders including an encoder/decoder pair for each of the plurality of modalities, wherein:

the encoder of the encoder/decoder pair is trained to project data for the modality into a latent representation space associated with the join-modality representation; and

the decoder of the encoder/decoder pair is trained to decode input from the common representation space to generate output in a representation space of the modality;

the method comprising:

selecting, based on the first modality and the second modality, an encoder/decoder pair or a pair of encoders, from among the plurality of encoders and decoders; and

predicting the association between the input data in the first modality and the data in the second modality, the predicting comprising:

processing the input data with the joint-modality representation and the selected encoder/decoder pair or pair of encoders to obtain a projection of the input data in a representation space of the model, the representation space comprising a representation space of the second modality or the latent representation space associated with the joint-modality representation; and

predicting the association between the input data in the first modality and the data in the second modality by determining, in the representation space, a measure of similarity between the projection of the input data and the data in the second modality.

14. The computer system of claim 13 , wherein selecting, based on the first modality and the second modality, an encoder/decoder pair or a pair of encoders, from among the plurality of encoders and decoders comprises selecting an encoder trained to process data for the first modality and a decoder trained to process data to project the data into the representation space of the second modality.

15. The computer system of claim 14 , wherein the representation space comprises the representation space for the second modality and predicting the association between the input data in the first modality and the data in the second modality comprises:

predicting the association between the input data in the first modality and the data in the second modality in the representation space for the second modality.

16. The computer system of 15 , wherein predicting the association between the input data in the first modality and the data in the second modality comprises:

processing the input data in the first modality using the selected encoder to obtain a projection of the input data in the common representation of the statistical model;

processing, using the decoder trained to process data to project the data into a representation space of the second modality, to obtain an output vector in the representation space for the second modality;

projecting the data from the second modality into the representation space for the second modality to produce a plurality of projected vectors; and

predicting the association between the input data in the first modality and the data in the second modality based on a comparison of the output vector and the projected vectors in the representation space for the second modality.

17. The computer system of claim 16 , wherein predicting the association between the input data in the first modality and the data in the second modality comprises:

calculating a distance between the output vector and each of the plurality of projected vectors; and

predicting the association based on the calculated distances.

18. The computer system of claim 17 , wherein calculating a distance comprises calculating a Euclidean distance.

19. The computer system of claim 13 , wherein selecting, based on the first modality and the second modality, an encoder/decoder pair or a pair of encoders, from among the plurality of encoders and decoders comprises selecting a first encoder trained to process data for the first modality and a second encoder trained to process data for the second modality.

20. The computer system of claim 19 , wherein the representation space comprises the latent representation space associated with the joint-modality representation and predicting the association between the input data in the first modality and the data in the second modality comprises:

predicting the association between the input data in the first modality and the data in the second modality in the latent representation space associated with the joint-modality representation.

21. The computer system of claim 20 , wherein predicting the association between the input data in the first modality and the data in the second modality comprises:

providing as input to the first encoder, the input data to produce a first modality feature vector in the latent representation space;

providing as input to the second encoder, the data for the second modality to produce a plurality of second modality feature vectors in the latent representation space; and

predicting the association between the input data in the first modality and the data in the second modality based on a comparison of the first modality feature vector and the plurality of second modality feature vectors in the latent representation space.

22. The computer system of claim 21 , wherein predicting the association between the input data in the first modality and the data in the second modality comprises:

calculating a distance between the first modality feature vector and each of the plurality of second modality feature vectors; and

predicting the association based on the calculated distances.