Machine learning systems for generating multi-modal data archetypes
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating multi-modal data archetypes. In one aspect, a method comprises obtaining a plurality of training examples, wherein each training example corresponds to a respective patient and includes multi-modal data, having a plurality of feature dimensions, that characterizes the patient; jointly training an encoder neural network and a decoder neural network on the plurality of training examples; and generating a plurality of multi-modal data archetypes that each correspond to a respective dimension of a latent space, comprising, for each multi-modal data archetype: processing a predefined embedding that represents the corresponding dimension of the latent space using the decoder neural network to generate multi-modal data, having the plurality of feature dimensions, that defines the multi-modal data archetype.
1. A method comprising:
generating a plurality of multi-modal data archetypes using an encoder neural network that has been jointly trained along with a decoder neural network, wherein:
the encoder neural network is configured to process input multi-modal data characterizing an input patient to generate an embedding of the input multi-modal data in a multi-dimensional latent space;
the decoder neural network is configured to process the embedding of the input multi-modal data to generate a reconstruction of the input multi-modal data; and
generating the plurality of multi-modal data archetypes comprises:
processing, for each patient in a population of patients, multi-modal data characterizing the patient using the encoder neural network to generate an embedding of the multi-modal data in the latent space,
wherein the embeddings of multi-modal data characterizing the patients in the population of patients collectively define a set of embeddings in the latent space;
processing the set of embeddings in the latent space to generate a set of parameters defining a convex hull of the set of embeddings in the latent space;
processing: (i) the set of parameters defining the convex hull of the set of embeddings in the latent space, and (ii) the set of embeddings, to identify a proper subset of the embeddings in the set of embeddings as being archetype embeddings; and
identifying the respective multi-modal data represented by each archetype embedding as a respective multi-modal data archetype.
2. The method of claim 1 , further comprising generating a respective representation of each of the plurality of multi-modal data archetypes, comprising, for each multi-modal data archetype:
generating a respective intensity score for each of a plurality of feature dimensions of the multi-modal data archetype based on: (i) a value of the feature dimension of the multi-modal data archetype, and (ii) a distribution defined by values of the feature dimension of multi-modal data included in the plurality of training examples;
wherein the representation of the multi-modal data archetype comprises the respective intensity score for each of the plurality of feature dimensions of the multi-modal data archetype.
3. The method of claim 2 , wherein for each of the plurality of feature dimensions of the multi-modal data archetype, the intensity score for the feature dimension characterizes a likelihood of the value of the feature dimension of the multi-modal data archetype under the distribution defined by values of the feature dimension of multi-modal data included in the plurality of training examples.
4. The method of claim 3 , wherein for each of the plurality of feature dimensions of the multi-modal data archetype, determining the intensity score for the feature dimension comprises:
determining a mean and a standard deviation of the distribution defined by values of the feature dimension of multi-modal data included in the plurality of training examples; and
determining the intensity score for the feature dimension using the mean and the standard deviation of the distribution defined by values of the feature dimension of multi-modal data included in the plurality of training examples.
5. The method of claim 2 , further comprising providing the representations of the multi-modal data archetypes as explainability data that explains the dimensions of the latent space.
6. The method of claim 1 , wherein jointly training the encoder neural network and the decoder neural network comprises, for each of a plurality of training examples:
processing multi-modal data from the training example using the encoder neural network, in accordance with current values of a set of encoder parameters, to generate an embedding of the multi-modal data from the training example;
processing the embedding of the multi-modal data from the training example using the decoder neural network, in accordance with current values of a set of decoder parameters, to generate a reconstruction of the multi-modal data from the training example; and
updating the current values of the set of encoder parameters and the current values of the set of decoder parameters using gradients of a reconstruction loss function that measures an error in the reconstruction of the multi-modal data from the training example, wherein:
the reconstruction loss function comprises a plurality of scaling factors that each scale a respective term in the reconstruction loss function that measures an error in the reconstruction of a corresponding proper subset of a set of feature dimensions of the multi-modal data from the training example, and
each of the plurality of scaling factors has a respective value that is based on a relevance of the corresponding proper subset of the feature dimensions of the multi-modal data from the training example to a particular medical condition.
7. The method of claim 6 , wherein the respective value of each of the plurality of scaling factors is based on a relevance of the corresponding proper subset of the feature dimensions of the multi-modal data from the training example to diagnosing the particular medical condition.
8. The method of claim 6 , wherein the respective value of each of the plurality of scaling factors is based on a relevance of the corresponding proper subset of the feature dimensions of the multi-modal data from the training example to a treatment for the particular medical condition.
9. The method of claim 6 , wherein scaling factors corresponding to proper subsets of the feature dimensions of the multi-modal data that are more relevant to the particular medical condition have higher values than scaling factors corresponding to proper subsets of the feature dimensions of the multi-modal data that are less relevant to the particular medical condition.
10. The method of claim 1 , wherein jointly training the encoder neural network and the decoder neural network comprises, for each latent dimension in a proper subset of a plurality of latent dimensions of the latent space:
obtaining multi-modal data that defines a target multi-modal data archetype, having a plurality of feature dimensions, that corresponds to the latent dimension;
processing a predefined embedding that represents the latent dimension using the decoder neural network to generate multi-modal data, having the plurality of feature dimensions, that defines a predicted multi-modal data archetype corresponding to the latent dimension; and
updating values of a set of decoder parameters using gradients of a loss function that measures an error between: (i) the predicted multi-modal data archetype corresponding to the latent dimension, and (ii) the target multi-modal data archetype corresponding to the latent dimension.
11. The method of claim 1 , wherein each of the plurality of multi-modal data archetypes includes a feature representation for a functional magnetic resonance imaging (fMRI) modality, and wherein the feature representation for the fMRI modality is derived from a series of fMRI images that each correspond to a respective time point in a sequence of time points and characterize blood flow in a brain of the patient at the time point.
12. The method of claim 1 , wherein each of the plurality of multi-modal data archetypes includes a feature representation for an electroencephalography (EEG) modality, and wherein the feature representation of the EEG modality is derived from a plurality of voltage waveforms that are each measured by a respective electrode placed in proximity to a brain of the patient.
13. The method of claim 1 , wherein each of the plurality of multi-modal data archetypes includes a feature representation for a genomics modality, and wherein the feature representation of the genomics modality is derived from data defining a sequence of nucleotides from a genome of the patient.
14. The method of claim 1 , further comprising:
processing the plurality of multi-modal data archetypes to identify one or more dimensions to be removed from the latent space; and
removing the identified dimensions from the latent space.
15. The method of claim 14 , wherein processing the plurality of multi-modal data archetypes to identify one or more dimensions to be removed from the latent space comprises, for each multi-modal data archetype:
determining whether a value of a feature dimension of the multi-modal data archetype satisfies a threshold; and
determining whether a corresponding dimension of the latent space should be removed based at least in part on the whether the value of the feature dimension of the multi-modal data archetype satisfies the threshold.
16. The method of claim 1 , wherein processing: (i) the set of parameters defining the convex hull of the set of embeddings in the latent space, and (ii) the set of embeddings, to identify a proper subset of the embeddings in the set of embeddings as being archetype embeddings comprises:
determining a set of vertices of the convex hull of the set of embeddings in the latent space; and
identifying the archetype embeddings using the set of vertices of the convex hull of the set of embeddings in the latent space.
17. The method of claim 16 , wherein identifying the archetype embeddings using the set of vertices of the convex hull of the set of embeddings in the latent space comprises, for each vertex:
identifying an embedding in the set of embeddings that has a minimum distance to the vertex from among the embeddings in the set of embeddings as being an archetype embedding corresponding to the vertex.
18. The method of claim 17 , wherein for each vertex, identifying an embedding in the set of embeddings that has a minimum distance to the vertex comprises:
identifying an embedding in the set of embeddings that has a minimum Euclidean distance to the vertex.
19. A system comprising:
one or more computers; and
one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
generating a plurality of multi-modal data archetypes using an encoder neural network that has been jointly trained along with a decoder neural network, wherein:
the encoder neural network is configured to process input multi-modal data characterizing an input patient to generate an embedding of the input multi-modal data in a multi-dimensional latent space;
the decoder neural network is configured to process the embedding of the input multi-modal data to generate a reconstruction of the input multi-modal data; and
generating the plurality of multi-modal data archetypes comprises:
processing, for each patient in a population of patients, multi-modal data characterizing the patient using the encoder neural network to generate an embedding of the multi-modal data in the latent space,
wherein the embeddings of multi-modal data characterizing the patients in the population of patients collectively define a set of embeddings in the latent space;
processing the set of embeddings in the latent space to generate a set of parameters defining a convex hull of the set of embeddings in the latent space;
processing: (i) the set of parameters defining the convex hull of the set of embeddings in the latent space, and (ii) the set of embeddings, to identify a proper subset of the embeddings in the set of embeddings as being archetype embeddings; and
identifying the respective multi-modal data represented by each archetype embedding as a respective multi-modal data archetype.
20. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
generating a plurality of multi-modal data archetypes using an encoder neural network that has been jointly trained along with a decoder neural network, wherein:
the encoder neural network is configured to process input multi-modal data characterizing an input patient to generate an embedding of the input multi-modal data in a multi-dimensional latent space;
the decoder neural network is configured to process the embedding of the input multi-modal data to generate a reconstruction of the input multi-modal data; and
generating the plurality of multi-modal data archetypes comprises:
processing, for each patient in a population of patients, multi-modal data characterizing the patient using the encoder neural network to generate an embedding of the multi-modal data in the latent space,
wherein the embeddings of multi-modal data characterizing the patients in the population of patients collectively define a set of embeddings in the latent space;
processing the set of embeddings in the latent space to generate a set of parameters defining a convex hull of the set of embeddings in the latent space;
processing: (i) the set of parameters defining the convex hull of the set of embeddings in the latent space, and (ii) the set of embeddings, to identify a proper subset of the embeddings in the set of embeddings as being archetype embeddings; and
identifying the respective multi-modal data represented by each archetype embedding as a respective multi-modal data archetype.