Automated control of microfluidic devices based on machine learning
A system is provided to automatically monitor and control the operation of a microfluidic device using machine learning technology. The system receives images of a channel of a microfluidic device collected by a camera during operation of the microfluidic device. Upon receiving an image, the system applies a classifier to the image to classify the operation of the microfluidic device as normal, in which no adjustment to the operation is needed, or as abnormal, in which an adjustment to the operation is needed. When an image is classified as normal, the system may make no adjustment to the microfluidic device. If, however, an image is classified as abnormal, the system may output an indication that the operation is abnormal, output an indication of a needed adjustment, or control the microfluidic device to make the needed adjustment.
1. A method performed by a computing system, the method comprising, for each of a plurality of intervals:
receiving an image of a channel of a microfluidic device;
applying a classifier to the image to generate a label for the image; and
when the label indicates that the microfluidic device is operating abnormally, identifying an adjustment for controlling the microfluidic device to transition to operating normally
wherein the labels comprise dripping, jetting, breaking, and wetting.
2. The method of claim 1 further comprising controlling the microfluidic device based on the adjustment.
3. The method of claim 1 wherein the adjustment indicates a change in flow rate of a fluid.
4. The method of claim 1 wherein fluids of the channel are a double emulsion.
5. The method of claim 1 wherein the classifier is pre-trained based on images that are not of the channel.
6. The method of claim 1 wherein the classifier includes a convolution neural network.
7. The method of claim 1 wherein the classifier includes a convolution neural network to generate features of the image and a sub-classifier to generate the label for the image based on the generated features of the image.
8. The method of claim 7 wherein the sub-classifier is a support vector machine.
9. The method of claim 7 wherein the sub-classifier generates the label for the image further based on an additional feature not generated by the convolution neural network.
10. The method of claim 1 wherein the classifier being trained using training data that includes sample images that each has a label, the sample images being of a channel of a training microfluidic device taken during operation of the training microfluidic device.
11. A method performed by a computing system, the method comprising:
for each of a plurality of intervals,
receiving an image of a channel of a microfluidic device during operation of the microfluidic device;
applying a classifier to the image to assign one of a plurality of labels to the image, the classifier including a convolution neural network that was trained using training data that includes sample images that each have a label, the sample images being of a channel of a training microfluidic device taken during operation of the training microfluidic device; and
when the label is associated with an adjustment for the microfluidic device, identifying the adjustment; and
controlling operation of the microfluidic device based on the adjustment
wherein an adjustment is a change in a flow rate of a fluid of a double emulsion and an adjustment is to discard malformed droplets of a double emulsion.
12. The method of claim 11 wherein during initial operation of the microfluidic device, the adjustments control the microfluidic device to enter a normal mode of operation.
13. A method performed by a computing system, the method comprising, for each of a plurality of intervals:
receiving one image for that interval of a channel of a microfluidic device;
applying a classifier to the one image to generate a label for the one image the label indicating whether the microfluidic device is operating normally or abnormally; and
indicating that the microfluidic device is operating as specified by the label
wherein the operation of the microfluidic device for each interval can be determined from content of the one image for that interval.
14. The method of claim 13 further comprising adjusting operation of the microfluidic device based on the generated label.
15. The method of claim 14 wherein the adjusting is changing the flow rate of a fluid.
16. The method of claim 15 wherein fluids of the channel are a double emulsion.
17. The method of claim 13 wherein the labels comprise dripping, jetting, breaking, and wetting.
18. The method of claim 13 wherein the classifier includes a convolution neural network to generate features of the image and a sub-classifier to generate the label for the image based on the generated features of the image.
19. The method of claim 13 wherein the microfluidic device is used to sort cells.
20. The method of claim 13 wherein the microfluidic device is used for microencapsulation.
21. The method of claim 13 wherein the label indicates an abnormal operation of the microfluidic device.
22. The method of claim 21 wherein when the label indicates an abnormal operation, the flow rate of a fluid is adjusted.
23. The method of claim 13 wherein the classifier is trained using training data that includes sample images that each have a label, the sample images being of a channel of a training microfluidic device taken during operation of the training microfluidic device.
24. One or more computing systems comprising:
one or more computer-readable storage media storing computer-executable instructions that, when executed, control the one or more computing systems to, for each of a plurality of intervals:
receive one image for that interval of a channel of a microfluidic device;
apply a classifier to the one image to generate a label for the one image the label indicating whether the microfluidic device is operating normally or abnormally; and
indicate that the microfluidic device is operating as specified by the label
wherein the operation of the microfluidic device for each interval can be determined from content of the one image for that interval and
one or more processors for executing the computer-executable instructions stored in the one or more computer-readable storage media.
25. The one or more computing systems of claim 24 wherein the instructions further control the one or more computing systems to adjust operation of the microfluidic device based on the generated label.
26. The one or more computing systems of claim 25 wherein the adjusting is changing the flow rate of a fluid.
27. The one or more computing systems of claim 26 wherein fluids of the channel are a double emulsion.
28. The one or more computing systems of claim 24 wherein the labels comprise dripping, jetting, breaking, and wetting.
29. The one or more computing systems of claim 24 wherein the classifier includes a convolution neural network to generate features of the image and a sub-classifier to generate the label for the image based on the generated features of the image.
30. The one or more computing systems of claim 24 wherein the microfluidic device is used to sort cells.
31. The one or more computing systems of claim 24 wherein the microfluidic device is used for microencapsulation.
32. The one or more computing systems of claim 24 wherein the label indicates an abnormal operation of the microfluidic device.
33. The one or more computing systems of claim 32 wherein when the label indicates an abnormal operation, the flow rate of a fluid is adjusted.
34. The one or more computing systems of claim 24 wherein the classifier is trained using training data that includes sample images that each have a label, the sample images being of a channel of a training microfluidic device taken during operation of the training microfluidic device.