Systems and methods for estimating embryo viability
A computer-implemented method, including the steps of: receiving video data of a human embryo, the video data representing a sequence of images of the human embryo in chronological order; applying at least one three-dimensional (3D) artificial neural network (ANN) to the video data to determine a viability score for the human embryo, wherein the viability score represents a likelihood that the human embryo will result in a viable embryo or a viable fetus; and outputting the viability score.
1. A computer-implemented method, including the steps of:
receiving video data of a human embryo, the video data captured by an image sensor of an incubator and including a sequence of images of the human embryo in chronological order, the video data including spatial dimensions and temporal dimensions;
applying at least one three-dimensional (3D) artificial neural network (ANN) including a 3D convolutional neural network (CNN) to the video data to determine a viability score for the human embryo, wherein the viability score represents a likelihood that the human embryo will result in a viable embryo or a viable fetus, the 3D convolutional neural network (CNN) being configured to extract features from both the spatial dimensions and the temporal dimensions of the video data by performing 3D convolutions, and the 3D CNN including a plurality of convolution layers each using a 3D convolution kernel, and a plurality of pooling layers each using a 3D pooling kernel that determine the viability score; and
outputting the viability score.
2. The method of claim 1 , wherein the viability score represents a probability that transferring the human embryo will result in a viable fetus.
3. The method of claim 1 , wherein the viability score represents a probability that transferring the human embryo will result in any one or more of the following:
a viable fetal heart detected within a predetermined period of time after embryo transfer;
a biochemical pregnancy detected based on a urine test or a blood test;
a viable gestational sac or a viable yolk sac detected on ultrasound at a predetermined time after embryo transfer; and
a live birth at the end of pregnancy.
4. The method of claim 1 , wherein the 3D ANN is trained by using:
video data representing a plurality of sequences of images of a plurality of human embryos; and
pregnancy outcome data that indicates whether each of the plurality of human embryos has resulted in a viable embryo or a viable fetus.
5. The method of claim 4 , wherein the training of the 3D ANN is performed without manual selection or extraction of features, or human annotation of key development events.
6. The method of claim 1 , further including one or more of:
standardising the received video data so that all videos span a predetermined time period;
cropping the video data to retain predetermined areas of the data;
adjusting contrast of the images in the video data; and
resizing the images in the video data to a predetermined image size.
7. The method of claim 1 , further including:
processing the video data by adding a visual overlay to at least some images of the video data, the visual overlay indicative of contributions of respective portions of the images to the viability score; and
outputting the images with the visual overlays.
8. The method of claim 7 , wherein the visual overlay is a heat map.
9. The method of claim 7 , wherein the visual overlay is generated by:
determining changes of the viability score output caused by occluding portions of the images or sequence of images in the video data.
10. The method of claim 9 , wherein occluding portions of the images in the video data includes applying a three-dimensional occlusion window to the video data.
11. The method of claim 1 , further including:
determining a ranking of a plurality of human embryos based on their viability scores.
12. The method of claim 11 , further including:
selecting, based on the ranking, one of the plurality of human embryos for a single embryo transfer or the order in which multiple embryos should be transferred.
13. A system, including at least one processer configured to:
receive video data of a human embryo, the video data captured by an image sensor of an incubator and including a sequence of images of the human embryo in chronological order, the video data including spatial dimensions and temporal dimensions;
apply at least one three-dimensional (3D) artificial neural network (ANN) including 3D convolution neural network (CNN) to the video data to determine a viability score for the human embryo, wherein the viability score represents a likelihood that the human embryo will result in a viable embryo or a viable fetus, the 3D convolutional neural network (CNN) being configured to extract features from both the spatial dimensions and the temporal dimensions of the video data by performing 3D convolutions, and the 3D CNN including a plurality of convolution layers each using a 3D convolution kernel, and a plurality of pooling layers each using a 3D pooling kernel that determine the viability score; and
output the viability score.
14. The system of claim 13 , wherein the system includes:
an image sensor for capturing video data of the human embryo;
wherein the processer is configured to receive the captured video data from the image sensor.
15. The system of claim 13 , wherein the system includes a time-lapse incubator.
16. A method including:
generating a viability score for a human embryo by probabilistically classifying data including spatial dimensions and temporal dimensions and representing a video of the human embryo with a three-dimensional (3D) artificial neural network (ANN),
wherein the 3D artificial neural network is a 3D convolutional neural network (CNN), and
wherein the 3D convolutional neural network (CNN) being configured to extract features from both the spatial dimensions and the temporal dimensions of the video data by performing 3D convolutions, and the 3D CNN includes a plurality of convolution layers each using a 3D convolution kernel, and a plurality of pooling layers each using a 3D pooling kernel that determine the viability score; or
training a three-dimensional (3D) artificial neural network (ANN) to probabilistically classify data including spatial dimensions and temporal dimensions and representing a video of a human embryo to generate a viability score for the human embryo,
wherein the 3D artificial neural network is a 3D convolutional neural network (CNN) configured to extract features from both the spatial and the temporal dimensions of the data representing the video by performing 3D convolutions, and
wherein the 3D CNN includes a plurality of convolution layers each using a 3D convolution kernel, and a plurality of pooling layers each using a 3D pooling kernel that determine the viability score.
17. The method of claim 16 , wherein the viability score represents a likelihood that the human embryo will result in a viable embryo or a viable foetus.
18. The method of claim 16 , wherein the video is of the human embryo in an incubator optionally for a preselected period of time; and/or wherein the video includes a series of images optionally including time-lapse images.
19. The method of claim 16 , wherein the ANN is trained using:
data representing a plurality of videos of human embryos; and
data representing whether the human embryos have resulted in respective viable embryos or viable fetuses.
20. A system, including at least one processer configured to:
generate a viability score for a human embryo by probabilistically classifying data representing a video of the human embryo with a three-dimensional (3D) artificial neural network (ANN),
wherein the 3D artificial neural network is a 3D convolutional neural network (CNN) configured to extract features from both the spatial and the temporal dimensions of the data by performing 3D convolutions, and
wherein the 3D CNN includes a plurality of convolution layers each using a 3D convolution kernel, and a plurality of pooling layers each using a 3D pooling kernel; or
train a three-dimensional (3D) artificial neural network (ANN) to probabilistically classify data representing a video of a human embryo to generate a viability score for the human embryo,
wherein the 3D artificial neural network is a 3D convolutional neural network (CNN) configured to extract features from both the spatial and the temporal dimensions of the data by performing 3D convolutions, and
wherein the 3D CNN includes a plurality of convolution layers each using a 3D convolution kernel, and a plurality of pooling layers each using a 3D pooling kernel that determine the viability score.