Multi-headed neural networks for AI-based protein and drug design
Methods and apparatus for protein and drug design using neural networks with two or more output heads, wherein one head, a sequence head, is trained to generate the sequence of a protein, and another head, a structure head, is trained to generate the structure of the protein; and wherein the neural network is configured to accept a representation of a specified condition as input, and output a representation of a protein's sequence and structure. The structure head and sequence head each have their own loss functions, and the weights of the neural network body are shared, and jointly updated during training. Non-limiting examples of specified input conditions include representations of associated proteins and/or sets of properties of the desired output protein. Some embodiments of the invention include for the design and synthesis of effective peptide drug ligands, synthetic biologic antibody drugs, antibody drug conjugates, and monoclonal antibody (mAb) drugs.
1 . A method, comprising:
a) receiving, at a processor, a trained neural network:
i) wherein the neural network is configured to accept a representation of one or more specified conditions as input, and to yield as output, a representation of a protein associated with the one or more specified conditions,
ii) wherein the neural network has at least two output heads, including one output head which generates the output protein's sequence and a different output head which generates the output protein's structure,
iii) wherein the process of training the neural network entailed a loss function computation corresponding to the sequence head output and a different loss function computation corresponding to the structure head output,
iv) wherein some non output head weights of the neural network are shared;
b) using the neural network to generate a representation of a protein or small molecule, given a representation of the specified condition(s) as input,
wherein, based on the input, the sequence head generates a sequence representation of the protein and the structure head generates a structure representation of the protein, and
wherein a sequence and structure representation of the generated protein is returned as output.
2 . The method of claim 1 , wherein the output protein is synthesized.
3 . The method of claim 1 , wherein the training process uses backpropagation; and wherein during training, the backpropagation and weight updates proceed backwards independently from each of the heads through the ancestral nodes of the respective head.
4 . The method of claim 3 , wherein the sequence head's final output is a probability distribution over amino acids and auxiliary tokens; and wherein the structure head's final output is a probability distribution over possible structure parameters associated with each residue.
5 . The method of claim 4 , wherein the sequence and structure generation is via an autoregressive procedure.
6 . The method of claim 5 , wherein the specified condition is a target receptor and the specified condition's representation is a representation of the target receptor's sequence and structure; and wherein the output protein is a peptide ligand drug.
7 . The method of claim 6 , for generating a representation of a peptide ligand drug's sequence and structure given a representation of a target receptor's sequence and structure, wherein the method is also for obtaining and synthesizing an effective peptide ligand drug, the method further comprising:
a) using the trained neural network to obtain a representation of a peptide ligand drug, given a representation of a target receptor:
i) wherein, during autoregression, each residue is determined by randomly sampling the output probability distribution of the sequence head,
ii) wherein, during autoregression, the structure parameters associated with each residue are determined by randomly sampling the output probability distribution of the structure head;
b) repeating the random sampling-based peptide ligand drug representation generation procedure a plurality of times, thereby generating a plurality of representations of candidate peptide ligand drugs;
c) assessing the binding interaction and efficacy of each of the generated candidate peptide ligand drug representations;
d) selecting the most effective candidate peptide ligand drug;
e) synthesizing the peptide ligand drug.
8 . The method of claim 7 , wherein the specified conditions are a set of desired properties of the output protein; wherein the possible values of each property are categorical classes, each numerically encoded:
a) wherein the specified conditions are represented by a vector of length equal to the number of properties, as each entry position holds the value of the respective specified property;
b) wherein each specified property is numerically encoded categorically or continuously.
9 . The method of claim 8 , wherein the output protein is a peptide ligand for a given target protein.
10 . The method of claim 9 , wherein the given target protein is a receptor, and wherein the peptide ligand represented by the output is synthesized.
11 . A method, comprising:
a) receiving, at a processor, a representation of a protein:
i) wherein the representation of the protein was obtained using a neural network trained and configured to return a representation of a protein, given one or more specified conditions on the protein,
ii) wherein the neural network has at least two heads, including one output head which generates the output protein's sequence and a different output head which generates the output protein's structure,
iii) wherein during training of the neural network, there was a loss function computation corresponding to the sequence head output and a different loss function computation corresponding to the structure head output,
iv) wherein some non output head weights of the neural network were shared,
v) wherein, based on the input, the sequence head generated the sequence representation of the protein and the structure head generated the structure representation of the protein;
b) synthesizing the protein.
12 . The method of claim 11 , wherein biological properties of the protein are assessed in silico or in vitro.
13 . The method of claim 11 , wherein biological properties of the protein are assessed in vivo.
14 . The method of claim 11 , wherein the protein is used as a diagnostic or therapeutic agent in a human, animal, or plant.
15 . A method, comprising:
a) receiving a protein:
i) wherein the protein was synthesized from a representation obtained using a neural network trained and configured to return a representation of a protein, given one or more specified conditions on the protein,
ii) wherein the neural network has at least two heads, including one output head which generates the output protein's sequence and a different output head which generates the output protein's structure,
iii) wherein during training of the neural network, there was a loss function computation corresponding to the sequence head output and a different loss function computation corresponding to the structure head output,
iv) wherein some non output head weights of the neural network were shared,
v) wherein, based on the input, the sequence head generated the sequence representation of the protein and the structure head generated the structure representation of the protein;
b) assessing biological properties of the protein in vitro or in vivo.
16 . The method of claim 15 , wherein the protein is used as a diagnostic or therapeutic agent in a human, animal, or plant.
17 . The method of claim 15 , wherein the protein is a ligand of an industrial enzyme.
18 . A method, comprising:
a) receiving a protein:
i) wherein the protein was synthesized from a representation obtained using a neural network trained and configured to return a representation of a protein, given one or more specified conditions on the protein,
ii) wherein the neural network has at least two heads, including one output head which generates the output protein's sequence and a different output head which generates the output protein's structure,
iii) wherein during training of the neural network, there was a loss function computation corresponding to the sequence head output and a different loss function computation corresponding to the structure head output,
iv) wherein some non output head weights of the neural network were shared,
v) wherein, based on the input, the sequence head generated the sequence representation of the protein and the structure head generated the structure representation of the protein;
b) using the protein as a diagnostic or therapeutic agent in a human, animal, or plant.
19 . The method of claim 18 , wherein the specified condition includes a representation of an antigen, and wherein the output protein is an associated antibody.
20 . The method of claim 18 , wherein the specified condition includes a representation of a target receptor, and wherein the output protein is a peptide ligand of that receptor.