Retrieval-augmented fusion language models for AI-based protein and drug design
Methods and apparatus for obtaining representations of proteins and small molecule drugs for synthesis; wherein input queries into trained mixed modality protein and natural language models are augmented with relevant query-related documents. In one embodiment, the relevant query-related documents are obtained by maximum inner product search of an embedding latent vector space into which the query and the documents are projected. The top-k most relevant documents to the query are then combined with the query as input into the trained mixed modality language model. In one embodiment, the mixed modality model is an autoregressive multicapitate transformer whose decoder output heads correspond to the represented modalities. The method returns mixed modality output representations of proteins or small molecule drugs for synthesis or manufacture.
1 . A method, comprising:
a) receiving, at a processor, a trained mixed modality neural network:
i) wherein representation modalities are for representations of features of proteins,
ii) wherein the represented features include one or more of sequence, structure, function, interactions, interactors, binding partners, attributes, and properties,
iii) wherein the respective modalities of the representations include:
(1) natural language representation modality, and
(2) sequence representation modality,
iv) wherein the neural network is configured to accept as input data, a query consisting of one or more of the modalities, and to yield as output data, a response to the query, wherein the response also consists of one or more of the modalities;
b) receiving, at a processor, a retriever function:
i) wherein the retriever is configured to accept as input, queries to the mixed modality neural network,
ii) wherein given as input, a query to the mixed modality neural network, the retriever function's output includes a set of documents related to the input query;
c) using the trained mixed modality neural network to obtain a representation of a protein as output:
i) wherein the output is obtained in response to a combined input of the input query and the set of related documents;
d) synthesizing the protein.
2 . The method of claim 1 , wherein the biological activity of the protein is tested in vitro or in vivo.
3 . The method of claim 1 , wherein retriever function is a neural network trained on a similarity objective.
4 . The method of claim 1 , wherein the trained mixed modality neural network is an autoregressive transformer.
5 . The method of claim 4 , wherein for each respective head of the transformer, final output is a probability distribution over a set of possible values at that head.
6 . The method of claim 5 , wherein the input query specifies a target receptor and requests a peptide ligand of the receptor; and wherein the output is a representation of a peptide ligand of the specified target receptor.
7 . The method of claim 6 , for generating a representation of a peptide ligand of a target receptor, wherein the method is also for generating and synthesizing an effective peptide ligand, the method further comprising:
a) generating the output by randomly sampling the output probability distribution of the active head, at each iteration of the autoregression;
b) using the same input query, repeating the random-sampling based generation process a plurality of times, each yielding a candidate peptide ligand;
c) assessing the interaction, efficacy, and properties of each candidate ligand with the target receptor;
d) selecting the most effective ligand;
e) synthesizing the ligand.
8 . The method of claim 7 , wherein for each of the modalities on this list: (a) natural language, (b) protein sequence, and (c) protein structure; the input embedding used for input data of the respective modality is distinct from the input embedding used for input data of any of the other modalities on the list.
9 . A method comprising
a) receiving a trained mixed modality neural network:
i) wherein the representation modalities are for representations of features of proteins,
ii) wherein the represented features include one or more of sequence, structure, function, interactions, interactors, binding partners, attributes, and properties,
iii) wherein the respective modalities of the representations include:
(1) natural language representation modality,
(2) sequence representation modality,
(3) structure representation modality, and
(4) small molecule drug representation modality,
iv) wherein the neural network is configured to accept as input data, a query consisting of one or more of the modalities, and to yield as output data, a response to the query, wherein the response also consists of one or more of the modalities,
v) wherein the neural network has multiple output heads, each with its own loss function,
vi) wherein the neural network heads include one head for natural language representation output, a different head for protein sequence representation output, a different head for protein structure representation output, and a different head for small molecule drug representation output;
b) receiving a retriever function:
i) wherein the retriever is configured to accept as input, queries to the mixed modality neural network,
ii) wherein given as input, a query to the mixed modality neural network, the retriever function's output includes a set of documents related to the input query;
c) using the trained mixed modality neural network to obtain a representation of a ligand as output:
i) wherein the output is obtained in response to a combined input of the input query and the set of related documents;
d) manufacturing the ligand.
10 . The method of claim 9 , further comprising testing the biological activity of the ligand in vitro or in vivo.
11 . The method of claim 10 , wherein the input query specifies a target receptor and requests a peptide ligand of the receptor; and wherein the manufactured ligand is a peptide.