Method and system for selecting embryos
An Artificial Intelligence (AI) computational system for generating an embryo viability score from a single image of an embryo to aid selection of an embryo for implantation in an In-Vitro Fertilisation (IVF) procedure is described. The AI model uses a deep learning method applied to images in which the Zona Pellucida region in the image is identified using segmentation, and ground truth labels such as detection of a heartbeat at a six week ultrasound scan.
1. A method for computationally generating an Artificial Intelligence (AI) model configured to estimate an embryo viability score from an image, the method comprising:
receiving a plurality of images and associated metadata, wherein each image is captured during a pre-determined time window after In-Vitro Fertilization (IVF) and the pre-determined time window is 24 hours or less, and the metadata associated with the image comprises at least a pregnancy outcome label;
pre-processing each image comprising at least segmenting the image to identify a Zona Pellucida region;
generating an Artificial Intelligence (AI) model configured to generate an embryo viability score from an input image by:
training at least one Zona Deep Learning Model using a deep learning method, comprising training a deep learning model on a set of Zona Pellucida images in which the Zona Pellucida regions are identified, and the associated pregnancy outcome labels are at least used to assess the accuracy of a trained model; and
training one or more additional AI models wherein each additional AI model is either a computer vision model trained using a machine learning method that uses a combination of one or more computer vision descriptors extracted from an image to estimate an embryo viability score, a deep learning model trained on images localized to the embryo comprising both Zona Pellucida and IntraZonal Cavity (IZC) regions, and a deep learning model trained on a set of IntraZonal Cavity (IZC) images in which all regions apart from the IZC are masked, and either using an ensemble method to combine at least two of the at least one Zona deep learning model and the one or more additional AI models to generate the AI model embryo viability score from an input image or using a distillation method to train an AI model to generate the AI model embryo viability score using the at least one Zona deep learning model and the one or more additional AI models to generate the AI model; and
deploying the AI model.
2. The method of claim 1 , wherein the set of Zona Pellucida images comprising images in which regions bounded by the Zona Pellucida region are masked.
3. The method of claim 1 , wherein the AI model is generated using an ensemble model comprising selecting at least two contrasting AI models from the at least one Zona deep learning model and the one or more additional AI models, and selection of AI models is performed to generate a set of contrasting AI models and applying a voting strategy to the at least two contrasting AI models that defines how the selected at least two contrasting AI models are combined to generate an outcome score for an image.
4. The method of claim 1 , wherein selecting at least two contrasting AI models comprises:
generating a distribution of embryo viability scores from a set of images for each of the at least one Zona deep learning model and the one or more additional AI models; and
comparing the distributions and discarding a model if the associated distributions is too similar to another distribution to select AI models with contrasting distributions.
5. The method of claim 1 , wherein the pre-determined time window is a 24 hour timer period beginning 5 days after fertilization.
6. The method of claim 1 , wherein the pregnancy outcome label is a ground-truth pregnancy outcome measurement performed within 12 weeks after embryo transfer.
7. The method of claim 6 , wherein the ground-truth pregnancy outcome measurement is whether a fetal heartbeat is detected.
8. The method of claim 1 , further comprising cleaning the plurality of image comprising identifying images with likely incorrect pregnancy outcome labels, and excluding or re-labelling the identified images.
9. The method of claim 1 , wherein each AI model generates an outcome score wherein the outcome is a n-ary outcome having n states, and training an AI model comprises a plurality of training-validation cycles further comprises randomly allocating the plurality of images to one of a training set, a validation set or a blind validation set, such that the training dataset comprises at least 60% of the images, the validation dataset comprises at least 10% of the images, and the blind validation dataset comprises at least 10% of the images, and after allocating the images to the training set, validation set and blind validation set, calculating the frequency of each of the n-ary outcome states in each of the training set, validation set and blind validation set, and testing that the frequencies are similar, and if the frequencies are not similar then discarding the allocation and repeating the randomization until a randomization is obtained in which the frequencies are similar.
10. The method of claim 1 , wherein training a computer vision model comprising performing a plurality of a training-validation cycles, and during each cycle the images are clustered based on the computer vision descriptors using an unsupervised clustering algorithm to generate a set of clusters, and each image is assigned to a cluster using a distance measure based on the values of the computer vision descriptors of the image, and a supervised learning method is use to determine whether a particular combination of these features corresponds to an outcome measure, and frequency information of the presence of each computer vision descriptor in the plurality of images.
11. The method of claim 1 , wherein the deep learning method uses a loss function configured to modify an optimization protocol is to emphasize global minima.
12. The method of claim 11 , wherein the loss function includes a residual term defined in terms of the network weights, which encodes the collective difference in the predicted value from the model and the target outcome for each image, and includes it as an additional contribution to the normal cross entropy loss function.
13. The method of claim 1 , wherein the method is performed on a cloud based computing system using a webserver, a database, and a plurality of training servers, wherein the webserver receives one or more model training parameters from a user, and the webserver initiates a training process on one or more of the plurality of training servers, comprising uploading training code to one of the plurality the training server, and the training server requests the plurality of images and associated metadata from a data repository, and performs the steps of preparing each image, generating a plurality of computer vision models and generating a plurality of deep learning models, and each training server is configured to periodically save the models to a storage service, and accuracy information to one or more log files to allow a training process to be restarted.
14. The method of claim 1 , wherein the ensemble model is trained to bias residual inaccuracies to minimize false negatives.
15. The method of claim 1 , wherein each image is a phase contrast image.
16. A method for computationally generating an embryo viability score from an image, the method comprising:
generating, in a computational system, an Artificial Intelligence (AI) model configured to generate an embryo viability score from an image according to the method comprising:
receiving a plurality of images and associated metadata, wherein each image is captured during a pre-determined time window after In-Vitro Fertilization (IVF) and the pre-determined time window is 24 hours or less, and the metadata associated with the image comprises at least a pregnancy outcome label;
pre-processing each image comprising at least segmenting the image to identify a Zona Pellucida region; and
generating the Artificial Intelligence (AI) model which is configured to generate an embryo viability score from an input image by:
training at least one Zona Deep Learning Model using a deep learning method, comprising training a deep learning model on a set of Zona Pellucida images in which the Zona Pellucida regions are identified, and the associated pregnancy outcome labels are at least used to assess the accuracy of a trained model; and
training one or more additional AI models wherein each additional AI model is either a computer vision model trained using a machine learning method that uses a combination of one or more computer vision descriptors extracted from an image to estimate an embryo viability score, a deep learning model trained on images localized to the embryo comprising both Zona Pellucida and IntraZonal Cavity (IZC) regions, and a deep learning model trained on a set of IntraZonal Cavity (IZC) images in which all regions apart from the IZC are masked, and either using an ensemble method to combine at least two of the at least one Zona deep learning model and the one or more additional AI models to generate the AI model embryo viability score from an input image or using a distillation method to train an AI model to generate the AI model embryo viability score using the at least one Zona deep learning model and the one or more additional AI models to generate the AI model;
receiving, from a user via a user interface of the computational system, an image captured during a pre-determined time window after In-Vitro Fertilization (IVF);
pre-processing the image according to the pre-processing steps used to generate the AI model;
providing the pre-processed image to the AI model to obtain an estimate of the embryo viability score; and
sending the embryo viability score to the user via the user interface.
17. A method for obtaining an embryo viability score from an image, comprising:
uploading, via a user interface, an image captured during a pre-determined time window after In-Vitro Fertilization (IVF) to a cloud based Artificial Intelligence (AI) model configured to generate an embryo viability score from an image wherein the AI model is generated according to the method comprising:
receiving a plurality of images and associated metadata, wherein each image is captured during a pre-determined time window after In-Vitro Fertilization (IVF) and the pre- determined time window is 24 hours or less, and the metadata associated with the image comprises at least a pregnancy outcome label;
pre-processing each image comprising at least segmenting the image to identify a Zona Pellucida region; and
generating the Artificial Intelligence (AI) model which is configured to generate an embryo viability score from an input image by:
training at least one Zona Deep Learning Model using a deep learning method, comprising training a deep learning model on a set of Zona Pellucida images in which the Zona Pellucida regions are identified, and the associated pregnancy outcome labels are at least used to assess the accuracy of a trained model; and
training one or more additional AI models wherein each additional AI model is either a computer vision model trained using a machine learning method that uses a combination of one or more computer vision descriptors extracted from an image to estimate an embryo viability score, a deep learning model trained on images localized to the embryo comprising both Zona Pellucida and IntraZonal Cavity (IZC) regions, and a deep learning model trained on a set of IntraZonal Cavity (IZC) images in which all regions apart from the IZC are masked, and either using an ensemble method to combine at least two of the at least one Zona deep learning model and the one or more additional AI models to generate the AI model embryo viability score from an input image or using a distillation method to train an AI model to generate the AI model embryo viability score using the at least one Zona deep learning model and the one or more additional AI models to generate the AI model;
receiving an embryo viability score from the cloud based AI model via the user interface.
18. A cloud based computational system comprising one or more computing apparatus, comprising one or more processors and one or more memories, configured to computationally generate an Artificial Intelligence (AI) model configured to estimate an embryo viability score from an image according to the method comprising:
receiving a plurality of images and associated metadata, wherein each image is captured during a pre-determined time window after In-Vitro Fertilization (IVF) and the pre-determined time window is 24 hours or less, and the metadata associated with the image comprises at least a pregnancy outcome label;
pre-processing each image comprising at least segmenting the image to identify a Zona Pellucida region; and
generating the Artificial Intelligence (AI) model which is configured to generate an embryo viability score from an input image by:
training at least one Zona Deep Learning Model using a deep learning method, comprising training a deep learning model on a set of Zona Pellucida images in which the Zona Pellucida regions are identified, and the associated pregnancy outcome labels are at least used to assess the accuracy of a trained model; and
training one or more additional AI models wherein each additional AI model is either a computer vision model trained using a machine learning method that uses a combination of one or more computer vision descriptors extracted from an image to estimate an embryo viability score, a deep learning model trained on images localized to the embryo comprising both Zona Pellucida and IntraZonal Cavity (IZC) regions, and a deep learning model trained on a set of IntraZonal Cavity (IZC) images in which all regions apart from the IZC are masked, and either using an ensemble method to combine at least two of the at least one Zona deep learning model and the one or more additional AI models to generate the AI model embryo viability score from an input image or using a distillation method to train an AI model to generate the AI model embryo viability score using the at least one Zona deep learning model and the one or more additional AI models to generate the AI model.
19. A cloud based computational system comprising one or more computing apparatus, comprising one or more processors and one or more memories, configured to computationally generate an embryo viability score from an image, wherein the computational system comprises:
an Artificial Intelligence (AI) model deployed to the cloud based computational system and configured to generate an embryo viability score from an image wherein the AI model is generated according to the method comprising:
receiving a plurality of images and associated metadata, wherein each image is captured during a pre-determined time window after In-Vitro Fertilization (IVF) and the pre-determined time window is 24 hours or less, and the metadata associated with the image comprises at least a pregnancy outcome label;
pre-processing each image comprising at least segmenting the image to identify a Zona Pellucida region; and
generating the Artificial Intelligence (AI) model which is configured to generate an embryo viability score from an input image by:
training at least one Zona Deep Learning Model using a deep learning method, comprising training a deep learning model on a set of Zona Pellucida images in which the Zona Pellucida regions are identified, and the associated pregnancy outcome labels are at least used to assess the accuracy of a trained model;
training one or more additional AI models wherein each additional AI model is either a computer vision model trained using a machine learning method that uses a combination of one or more computer vision descriptors extracted from an image to estimate an embryo viability score, a deep learning model trained on images localized to the embryo comprising both Zona Pellucida and IntraZonal Cavity (IZC) regions, and a deep learning model trained on a set of IntraZonal Cavity (IZC) images in which all regions apart from the IZC are masked, and either using an ensemble method to combine at least two of the at least one Zona deep learning model and the one or more additional AI models to generate the AI model embryo viability score from an input image or using a distillation method to train an AI model to generate the AI model embryo viability score using the at least one Zona deep learning model and the one or more additional AI models to generate the AI model;
a user interface configured to:
receive, from a user, an image captured during a pre-determined time window after In-Vitro Fertilization (IVF);
provide the image to the AI model to obtain an embryo viability score; and
send the embryo viability score to the user via the user interface.
20. A computational system configured to generate an embryo viability score from an image, wherein the computational system comprises at least one processor, and at least one memory comprising instructions to configure the at least one processor to:
receive an image captured during a pre-determined time window after In-Vitro Fertilization (IVF)
upload, via a user interface, the image captured during a pre-determined time window after In-Vitro Fertilization (IVF) to an Artificial Intelligence (AI) model deployed on a cloud based computational system configured to generate an embryo viability score from an image wherein the AI model is generated according to the method comprising:
receiving a plurality of images and associated metadata, wherein each image is captured during a pre-determined time window after In-Vitro Fertilization (IVF) and the pre-determined time window is 24 hours or less, and the metadata associated with the image comprises at least a pregnancy outcome label;
pre-processing each image comprising at least segmenting the image to identify a Zona Pellucida region; and
generating the Artificial Intelligence (AI) model which is configured to generate an embryo viability score from an input image by:
training at least one Zona Deep Learning Model using a deep learning method, comprising training a deep learning model on a set of Zona Pellucida images in which the Zona Pellucida regions are identified, and the associated pregnancy outcome labels are at least used to assess the accuracy of a trained model;
training one or more additional AI models wherein each additional AI model is either a computer vision model trained using a machine learning method that uses a combination of one or more computer vision descriptors extracted from an image to estimate an embryo viability score, a deep learning model trained on images localized to the embryo comprising both Zona Pellucida and IntraZonal Cavity (IZC) regions, and a deep learning model trained on a set of IntraZonal Cavity (IZC) images in which all regions apart from the IZC are masked, and either using an ensemble method to combine at least two of the at least one Zona deep learning model and the one or more additional AI models to generate the AI model embryo viability score from an input image or using a distillation method to train an AI model to generate the AI model embryo viability score using the at least one Zona deep learning model and the one or more additional AI models to generate the AI model;
receive an embryo viability score from the cloud based AI model; and
display the embryo viability score via the user interface.
21. A method for computationally generating an Artificial Intelligence (AI) model configured to estimate an embryo viability score from an image, the method comprising:
receiving a plurality of images and associated metadata, wherein each image is captured during a pre-determined time window after In-Vitro Fertilization (IVF) and the pre-determined time window is 24 hours or less, and the metadata associated with the image comprises at least a pregnancy outcome label;
pre-processing each image comprising at least segmenting the image to identify a Zona Pellucida region;
generating an Artificial Intelligence (AI) model configured to generate an embryo viability score from an input image by training at least one Zona Deep Learning Model using a deep learning method, comprising training a deep learning model on a set of Zona Pellucida images in which the Zona Pellucida regions are identified, and the associated pregnancy outcome labels are at least used to assess the accuracy of a trained model; and
deploying the AI model,
wherein the deep learning method uses a loss function configured to modify an optimization protocol to emphasize global minima, and
wherein the loss function includes a residual term defined in terms of the network weights, which encodes the collective difference in the predicted value from the model and the target outcome for each image, and includes it as an additional contribution to the normal cross entropy loss function.