Embedding-based generative model for protein design
A system and method for designing protein sequences conditioned on a specific target fold. The system is a transformer-based generative framework for modeling a complex sequence-structure relationship. To mitigate the heterogeneity between the sequence domain and the fold domain, a Fold-to-Sequence model jointly learns a sequence embedding using a transformer and a fold embedding from the density of secondary structural elements in 3D voxels. The joint sequence-fold representation through novel intra-domain and cross-domain losses with an intra-domain loss forces two semantically similar (where the proteins should have the same fold(s)) samples from the same domain to be close to each other in a latent space, while a cross-domain loss forces two semantically similar samples in different domains to be closer. In an embodiment, the Fold-to-Sequence model performs design tasks that include low resolution structures, structures with a region of missing residues, and NMR structural ensembles.
1. A computer-implemented method to design biological sequences using inverse folding of three-dimensional structures, the method comprising:
using a first transformer model encoder as part of a sequence-to-sequence model running on a hardware processor to generate latent sequence representations of corresponding receive an input one-dimensional biological sequences of nucleotides or amino acids represented as text which correspond with respective three-dimensional biological structures, a generated latent sequence representation of the generated latent sequence representations comprising an encoded vector representation of a one-dimensional biological sequence of the one-dimensional biological sequences in a first latent space;
using a second transformer model encoder as part of a fold-to-sequence model running on the hardware processor to generate respective latent fold representations of the respective three-dimensional biological structures corresponding to the respective input one-dimensional biological sequences, a generated latent fold representation of the generated latent fold representations comprising an encoded vector representation of a three-dimensional biological structure of the three-dimensional biological structures in a second latent space, wherein the three-dimensional biological structure comprises a density of structural elements represented in 3D voxels of a 3D voxel grid space, each 3D voxel comprising a feature representation comprising a probabilistic approximation determined from contributions of residues within neighboring voxels;
training, using the hardware processor, a transformer model decoder to learn a joint latent space between said generated latent sequence representations of input one-dimensional biological sequences and said generated respective latent fold representations of the respective three-dimensional biological structures corresponding to the input one-dimensional biological sequences;
receiving, at said second transformer model encoder running at the hardware processor, an encoded representation of a target three-dimensional biological structure;
generating, using said second transformer model encoder, an encoded vector representation of said received target three-dimensional biological structure in the second latent space; and
inputting the encoded vector representation of the target three-dimensional biological structure to said trained transformer model decoder running on said hardware processor, said trained transformer model outputting, in response, an alternative one-dimensional sequence of nucleotides or amino acids which will result in a similar three-dimensional biological structure having the same folds as said target three-dimensional biological structure.
2. The method of claim 1 , wherein said three-dimensional biological structure is a protein, said training the transformer model decoder comprising:
running a supervised learning task for classifying a fold representation of the protein using an encoded vector representation of the input one-dimensional biological sequence of the protein in the first latent space and using an encoded vector representation of the input three-dimensional biological sequence corresponding to the input protein sequence in the second latent space.
3. The method of claim 2 , wherein the sequence-to-sequence model comprises said first transformer encoder and said trained transformer decoder, said method further comprising:
comparing classified protein folds between two samples of input one-dimensional biological sequences of two proteins having similar fold representations and determining a resulting closeness of their encoded vector representations in the first latent space; and
training the sequence-to-sequence model by regularizing by a sequence intra-domain loss function comprising a sequence-to-sequence reconstruction loss and a first intra-domain similarity loss, determined by the resulting closeness of their encoded vector representations in the first latent space.
4. The method of claim 3 , wherein the fold-to-sequence model comprises said second transformer encoder and said trained transformer decoder, said method further comprising:
comparing classified protein folds between two samples of input three-dimensional biological structures of proteins having similar fold representations and determining a resulting closeness of their encoded vector representations in the second latent space; and
subsequently training the fold-to-sequence model by regularizing by a fold-to-sequence reconstruction loss and a second intra-domain similarity loss determined by the resulting closeness of their encoded vector representations in the second latent space.
5. The method of claim 4 , wherein said fold-to-sequence model is further regularized using a cosine similarity cross domain loss, said method further comprising:
determining a cosine similarity cross-domain loss by maximizing a cosine similarity between encoded vector representations in said first latent space and said second latent space, said determining the cosine similarity cross domain loss comprising:
computing a similarity matrix as a product between said encoded vector representations in said first latent space and said second latent space, wherein an i'th row in said similarity matrix represents a similarity between an i'th position in a fold and every position of a sequence;
performing upon said similarity matrix a row-wise average pooling with a defined kernel size; and
performing a row-wise max operation to result in averaging a score of continuous positions in the sequence as defined by the kernel size; and
obtaining a final similarity score by averaging over all positions in the fold.
6. The method of claim 5 , wherein said fold-to-sequence model is further regularized using a cyclic cross domain loss, said method further comprising:
determining a cyclic loss cross-domain loss responsive to a difference between an encoded latent state of an argmax of an alternative one-dimensional sequence output of the fold-to-sequence model and a native encoded latent state of the input one-dimensional biological sequence of the corresponding three-dimensional biological structure using the first transformer model encoder.
7. The method of claim 2 , further comprising:
obtaining a fold representation of a protein by:
re-scaling a three-dimensional (3D) protein structure to fit within the 3D voxel grid space having discretized voxels of predetermined size;
featurizing the protein structure to obtain protein structural element features for individual 3D voxels from the protein structure content.
8. The method of claim 7 , wherein the featurizing the protein structure further comprises:
denoting a position of each residue by its alpha carbon as a 3D coordinate within the 3D voxel grid space; and
assigning a secondary structure label to each residue, said secondary structure labels comprising one selected from: a helix, a beta strand, a loop and a bend/turn.
9. The method of claim 8 , further comprising:
generating an encoded representation of an input three-dimensional biological structure by applying a sinusoidal 3D positional encoding to the featurized protein structure in the 3D voxel grid space.
10. The method of claim 1 , wherein the structural elements represented in 3D voxels comprise secondary structure elements selected from the group comprising: a helix, beta-sheet, beta-strand, loop, bend, and turn, wherein a 3D voxel of the 3D voxels includes a representation of said secondary structure element as an associated label assigned to a residue.
11. A computer program product for designing biological sequences using inverse folding of three-dimensional structures, the computer program product comprising a non-transitory storage medium readable by a processing circuit and storing computer readable instructions run by the processing circuit for performing a method comprising:
using a first transformer model encoder as part of a sequence-to-sequence model to generate latent sequence representations of corresponding input one-dimensional biological sequences of nucleotides or amino acids represented as text which correspond with respective three-dimensional biological structures, a generated latent sequence representation of the generated latent sequence representations comprising an encoded vector representation of a one-dimensional biological sequence of the one-dimensional biological sequences in a first latent space;
using a second transformer model encoder as part of a fold-to-sequence model to generate respective latent fold representations of the respective three-dimensional biological structures corresponding to the respective input one-dimensional biological sequences, a generated latent fold representation of the generated latent fold representations comprising an encoded vector representation of a three-dimensional biological structure of the three-dimensional biological structures in a second latent space, wherein the three-dimensional biological structure comprises a density of structural elements represented in 3D voxels of a 3D voxel grid space, each 3D voxel comprising a feature representation comprising a probabilistic approximation determined from contributions of residues within neighboring voxels;
training a transformer model decoder to learn a joint latent space between said generated latent sequence representations of input one-dimensional biological sequences and said generated respective latent fold representations of the respective three-dimensional biological structures corresponding to the input one-dimensional biological sequences;
receiving, at said second transformer model encoder running at the processing circuit, an encoded representation of a target three-dimensional biological structure;
generating, using said second transformer model encoder, an encoded vector representation of said received target three-dimensional biological structure in the second latent space; and
inputting the encoded vector representation of the target three-dimensional biological structure to said trained transformer model decoder running on said processing circuit, said trained transformer model outputting, in response, an alternative one-dimensional sequence of nucleotides or amino acids which will result in a similar three-dimensional biological structure having the same folds as said target three-dimensional biological structure.
12. The computer program product as claimed in claim 11 , wherein said three-dimensional biological structure is a protein, said training the transformer model decoder comprising:
running a supervised learning task for classifying a fold representation of the protein using an encoded vector representation of the input one-dimensional biological sequence of the protein in the first latent space and using an encoded vector representation of the input three-dimensional biological sequence corresponding to the input protein sequence in the second latent space.
13. The computer program product as claimed in claim 12 , wherein the sequence-to-sequence model comprises said first transformer encoder and said trained transformer decoder, said method further comprising:
comparing classified protein folds between two samples of input one-dimensional biological sequences of two proteins having similar fold representations and determining a resulting closeness of their encoded vector representations in the first latent space; and
training the sequence-to-sequence model by regularizing by a sequence intra-domain loss function comprising a sequence-to-sequence reconstruction loss and a first intra-domain similarity loss determined by the resulting closeness of their encoded vector representations in the first latent space.
14. The computer program product of claim 13 , wherein the fold-to-sequence model comprises said second transformer encoder and said trained transformer decoder, said method further comprising:
comparing classified protein folds between two samples of input three-dimensional biological structures of proteins having similar fold representations and determining a resulting closeness of their encoded vector representations in the second latent space; and
subsequently training the fold-to-sequence model by regularizing by a fold-to-sequence reconstruction loss and a second intra-domain similarity loss determined by the resulting closeness of their encoded vector representations in the second latent space.
15. The computer program product of claim 14 , wherein said fold-to-sequence model is further regularized using a cosine similarity cross domain loss, said method further comprising:
determining a cosine similarity cross-domain loss by maximizing a cosine similarity between encoded vector representations in said first latent space and said second latent space, said determining the cosine similarity cross domain loss comprising:
computing a similarity matrix as a product between said encoded vector representations in said first latent space and said second latent space, wherein an i'th row in said similarity matrix represents a similarity between an i'th position in a fold and every position of a sequence;
performing upon said similarity matrix a row-wise average pooling with a defined kernel size; and
performing a row-wise max operation to result in averaging a score of continuous positions in the sequence as defined by the kernel size; and
obtaining a final similarity score by averaging over all positions in the fold.
16. The computer program product of claim 15 , wherein said fold-to-sequence model is further regularized using a cyclic cross domain loss, said method further comprising:
determining a cyclic loss cross-domain loss responsive to a difference between an encoded latent state of an argmax of an alternative one-dimensional sequence output of the fold-to-sequence model and a native encoded latent state of the input one-dimensional biological sequence of the corresponding three-dimensional biological structure using the first transformer model encoder.
17. The computer program product of claim 12 , wherein said method further comprises:
obtaining a fold representation of a protein by:
re-scaling a three-dimensional (3D) protein structure to fit within a 3D voxel grid space having discretized voxels of predetermined size;
featurizing the protein structure to obtain protein structural element features for individual voxels from the protein structure content.
18. The computer program product of claim 17 , wherein said method further comprises:
generating an encoded representation of an input three-dimensional biological structure by applying a sinusoidal 3D positional encoding to the featurized protein structure in the 3D voxel grid space.
19. The computer program product as claimed in claim 11 , wherein the structural elements represented in 3D voxels comprise secondary structure elements selected from the group comprising: a helix, beta-sheet, beta-strand, loop, bend, and turn, wherein a 3D voxel of the 3D voxels includes a representation of said secondary structure element as an associated label assigned to a residue.
20. A computer-implemented system to design biological sequences using inverse folding of three-dimensional structures, the system comprising:
a memory storage device; and
a hardware processor coupled to said memory storage device and configured to perform a method to:
use a first transformer model encoder as part of a sequence-to-sequence model to generate latent sequence representations of corresponding input one-dimensional biological sequences of nucleotides or amino acids represented as text which correspond with respective three-dimensional biological structures, a generated latent sequence representation of the generated latent sequence representations comprising an encoded vector representation of a one-dimensional biological sequence of the one-dimensional biological sequences in a first latent space;
use a second transformer model encoder as part of a fold-to-sequence model to generate respective latent fold representations of the respective three-dimensional biological structures corresponding to the respective input one-dimensional biological sequences, a generated latent fold representation of the generated latent sequence representations comprising an encoded vector representation of a three-dimensional biological structure of the three-dimensional biological structures in a second latent space, wherein the three-dimensional biological structure comprises a density of structural elements represented in 3D voxels of a 3D voxel grid space, each 3D voxel comprising a feature representation comprising a probabilistic approximation determined from contributions of residues within neighboring voxels;
train a transformer model decoder to learn a joint latent space between said generated latent sequence representations of input one-dimensional biological sequences and said generated respective latent fold representations of the respective three-dimensional biological structures corresponding to the input one-dimensional biological sequences;
receive, at said second transformer model encoder, an encoded representation of a target three-dimensional biological structure;
generate, using said second transformer model encoder, an encoded vector representation of said received target three-dimensional biological structure in the second latent space; and
input the encoded vector representation of the target three-dimensional biological structure to said trained transformer model decoder, said trained transformer model outputting, in response, an alternative one-dimensional sequence of nucleotides or amino acids which will result in a similar three-dimensional biological structure having the same folds as said target three-dimensional biological structure.
21. The computer-implemented system of claim 20 , wherein said three-dimensional biological structure is a protein, wherein to train the transformer model decoder, the hardware processor is further configured to:
run a supervised learning task for classifying a fold representation of the protein using an encoded vector representation of the input one-dimensional biological sequence of the protein in the first latent space and using an encoded vector representation of the input three-dimensional biological sequence corresponding to the input protein sequence in the second latent space.
22. The computer-implemented system of claim 21 , wherein the sequence-to-sequence model comprises said first transformer encoder and said trained transformer decoder, said hardware processor is further configured to:
compare classified protein folds between two samples of input one-dimensional biological sequences of two proteins having similar fold representations and determining a resulting closeness of their encoded vector representations in the first latent space; and
train the sequence-to-sequence model by regularizing by a sequence intra-domain loss function comprising a sequence-to-sequence reconstruction loss and a first intra-domain similarity loss determined by the resulting closeness of their encoded vector representations in the first latent space.
23. The computer-implemented system of claim 22 , wherein the fold-to-sequence model comprises said second transformer encoder and said trained transformer decoder, said hardware processor is further configured to:
compare classified protein folds between two samples of input three-dimensional biological structures of proteins having similar fold representations and determining a resulting closeness of their encoded vector representations in the second latent space; and
subsequently train the fold-to-sequence model by regularizing by a fold-to-sequence reconstruction loss and a second intra-domain similarity loss determined by the resulting closeness of their encoded vector representations in the second latent space.
24. The computer-implemented system of claim 23 , wherein said fold-to-sequence model is further regularized using a cosine similarity cross domain loss and a cyclic cross domain loss, said hardware processor is further configured to:
determine a cosine similarity cross-domain loss by maximizing a cosine similarity between encoded vector representations in said first latent space and said second latent space, said determining the cosine similarity cross domain loss comprising:
computing a similarity matrix as a product between said encoded vector representations in said first latent space and said second latent space, wherein an i'th row in said similarity matrix represents a similarity between an i'th position in a fold and every position of a sequence;
performing upon said similarity matrix a row-wise average pooling with a defined kernel size; and
performing a row-wise max operation to result in averaging a score of continuous positions in the sequence as defined by the kernel size; and
obtain a final similarity score by averaging over all positions in the fold, and
determine the cyclic loss cross-domain loss responsive to a difference between an encoded latent state of an argmax of an alternative one-dimensional sequence output of the fold-to-sequence model and a native encoded latent state of the input one-dimensional biological sequence of the corresponding three-dimensional biological structure using the first transformer model encoder.
25. The computer-implemented system of claim 20 , wherein the structural elements represented in 3D voxels comprise secondary structure elements selected from the group comprising: a helix, beta-sheet, beta-strand, loop, bend, and turn, wherein a 3D voxel of the 3D voxels includes a representation of said secondary structure element as an associated label assigned to a residue.