Micro-object movement monitor to observe controlled micro-object movement
What is disclosed is methods and systems for monitoring micro-object movements. Images depicting micro-objects moving across a backplane is received. Based on processing by a machine learning based image processing model, a first location and orientation of each micro-object in a first image and a second location and orientation of each micro-object in a second image is determined. Based on estimates of forces applied during a time between capture of the first and second images, movement during the time of a micro-object in the first image is estimated. A likely image of the micro-object in the second image is inferred based on the likely image having a location and orientation in the second image that corresponds to a combination of the movement and the location and orientation of the micro-object in the first image. An indication that the micro-object in the first image corresponds to the likely image is output.
1 . An apparatus for monitoring movement of micro-objects on a backplane, the apparatus comprising:
an image receiver configured to, when operating, receive a time sequence of captured images of a micro-assembler backplane being processed within a micro-assembler station, wherein the time sequence of images depicts a progression of a plurality of micro-objects moving across the micro-assembler backplane under control of the micro-assembler station;
an image processor configured to, when operating:
determine, based on processing of a first captured image in the time sequence of captured images by a machine learning based image processing model, a respective first location and a respective first orientation of each respective micro-object in the plurality of micro-objects on the micro-assembler backplane captured in the first captured image; and
determine, for a second captured image in the time sequence of images, a respective second location and respective second orientation of each respective micro-object on the micro-assembler backplane;
a movement processor configured to, when operating:
estimate, based on estimates of forces applied during a time duration between a time of capture of the first captured image and a time of capture of a second captured image in the time sequence of captured images, a calculated movement during the time duration for a tracked micro-object captured with a respective first location and a respective first orientation in the first captured image in the time sequence of images; and
infer a likely image of the tracked micro-object in the second captured image based on the likely image having a respective second location and a respective second orientation in the second captured image that corresponds to a combination of the calculated movement for the tracked micro-object applied to the respective first location and the respective first orientation of the tracked micro-object in the first captured image; and
an indicator output configured to, when operating, indicate, based on an inference of the likely image of the tracked micro-object in the second captured image, that the tracked micro-object in the first captured image corresponds to the likely image of the tracked micro-object in the second captured image.
2 . The apparatus of claim 1 , wherein the movement processor estimates the calculated movement of the tracked micro-object during the time duration by at least:
determining a respective potential of each respective micro-object in the plurality of micro-objects during the time duration; and
determining a respective calculated motion for each respective micro-object during the time duration wherein the respective calculated motion is based on the respective potential of each respective micro-object during the time duration,
wherein the calculated movement of the tracked micro-object during the time duration is based on the respective calculated motion for each respective micro-object.
3 . The apparatus of claim 2 , wherein the micro-assembler backplane comprises a plurality of electrodes distributed across its surface, and wherein the movement processor determines the respective potential of each respective micro-object in the plurality of micro-objects by at least:
determining an electric potential profile across the micro-assembler backplane during the time duration based upon selective activation of a subset of electrodes in the plurality of electrodes during the time duration.
4 . The apparatus of claim 3 , wherein the movement processor is further configured to, when operating:
receive indications of the selective activation of the subset of electrodes during the time duration; and
calculate the electric potential profile based on the indications of the selective activation of the subset of electrodes during the time duration.
5 . The apparatus of claim 4 , wherein the movement processor is configured to, when operating, determine the respective calculated motion for each respective micro-object during the time duration by at least:
determining a respective probability density function for the respective second location of each respective micro-object at the time of capture of the electric potential profile during the time duration, wherein determining the respective probability density function for each respective micro-object limits calculation of the respective probability density function to within a determined radius of a location of the respective micro-object in the first captured image.
6 . The apparatus of claim 1 , further comprising a training data set generator configured to, when operating:
generate a machine learning process training data set, wherein generating the machine learning process training data set comprises:
providing a set of object images of micro-objects, where each respective object image in the set of object images comprises a respective view of a micro-object having a respective orientation that is associated with the respective view and is different from each respective orientation of micro-objects in other object images within the set of object images;
generating a synthesized training image corresponding to a micro-assembler backplane on which micro-objects are placed, each synthesized training image comprising:
a selected plurality of object images selected from within the set of object images of micro-objects, wherein each respective object image in the selected plurality of object images is located at a respective location in the synthesized training image; and
a composite set of label data indicating for each respective object image in the selected plurality of object images:
the respective location of the respective object image in the synthesized training image; and
the respective orientation of the micro-object within the respective object image; and
a machine learning model training processor configured to, when operating, train the machine learning based image processing model based on a generated training data set.
7 . The apparatus of claim 6 , wherein the training data set generator is configured to generate the synthesized training image by at least generating a plurality of synthesized training images, wherein each respective synthesized training image in the plurality of synthesized training images differs from other respective synthesized training images in the plurality of synthesized training images by having at least one respective object image with at least one of: a different location within the synthesized training image; or with a different orientation.
8 . The apparatus of claim 6 , wherein the training data set generator is configured to generate the synthesized training image by at least modifying the synthesized training image to include a background pattern, where the background pattern comprises an image corresponding to a surface of a micro-assembler backplane on which micro-objects are placed for movement into defined locations.
9 . The apparatus of claim 6 , wherein the training data set generator is configured to provide the set of object images by at least:
receiving a first object image of a micro-object; and
generating at least some object images in the set of object images by at least manipulating the first object image to generate a respective object image that presents the micro-object in its respective view with a selected orientation.
10 . The apparatus of claim 6 , the training data set generator is configured to provide the set of object images by at least:
generating at least some object images in the set of object images by at least capturing a plurality of object images of a micro-object where each respective object image in the plurality of object images presents the micro-object with the respective orientation of the respective view.
11 . The apparatus of claim 10 , wherein the set of object images comprises a first number of object images, where the respective view in the object images in the first number of object images have respective orientations that differ from each other by multiple of an offset angle.
12 . A method of controlling placement of micro-objects on a backplane, the method comprising:
receiving a time sequence of captured images of a micro-assembler backplane being processed within a micro-assembler station, wherein the time sequence of images depicts a progression of a plurality of micro-objects moving across the micro-assembler backplane under control of the micro-assembler station;
determining, based on processing of a first captured image in the time sequence of captured images by a machine learning based image processing model, a respective first location and a respective first orientation of each respective micro-object in the plurality of micro-objects on the micro-assembler backplane captured in the first captured image;
estimating, based on estimates of forces applied during a time duration between a time of capture of the first captured image and a time of capture of a second captured image in the time sequence of captured images, a calculated movement during the time duration for a tracked micro-object captured with a first location in the first captured image in the time sequence of images;
determining, for the second captured image in the time sequence of images, a respective second location and respective second orientation of each respective micro-object on the micro-assembler backplane;
inferring a likely image of the tracked micro-object in the second captured image based on the likely image having a respective second location and a respective second orientation in the second captured image that corresponds to a combination of the calculated movement for the tracked micro-object applied to the respective first location and the respective first orientation of the tracked micro-object in the first captured image; and
indicating, based on inferring the likely image of the tracked micro-object in the second captured image, that the tracked micro-object in the first captured image corresponds to the likely image of the tracked micro-object in the second captured image.
13 . The method of claim 12 , wherein estimating the calculated movement of the tracked micro-object during the time duration comprises:
determining a respective potential of each respective micro-object in the plurality of micro-objects during the time duration; and
determining a respective calculated motion for each respective micro-object during the time duration wherein the respective calculated motion is based on the respective potential of each respective micro-object during the time duration,
wherein the calculated movement of the tracked micro-object during the time duration is based on the respective calculated motion for each respective micro-object.
14 . The method of claim 13 , wherein the micro-assembler backplane comprises a plurality of electrodes distributed across its surface, and wherein determining the respective potential of each respective micro-object in the plurality of micro-objects comprises:
determining an electric potential profile across the micro-assembler backplane during the time duration based upon selective activation of a subset of electrodes in the plurality of electrodes during the time duration.
15 . The method of claim 14 , further comprising:
receiving indications of the selective activation of the subset of electrodes during the time duration; and
calculating the electric potential profile based on the indications of the selective activation of the subset of electrodes during the time duration.
16 . The method of claim 15 , wherein determining the respective calculated motion for each respective micro-object during the time duration comprises:
determining a respective probability density function for the respective location of each respective micro-object at the time of capture of the electric potential profile during the time duration, wherein determining the respective probability density function for each respective micro-object limits calculation of the respective probability density function to within a determined radius of a location of the respective micro-object in the first captured image.
17 . The method of claim 12 , further comprising:
generating a machine learning process training data set, wherein generating the machine learning process training data set comprises:
providing a set of object images of micro-objects, where each respective object image in the set of object images comprises a respective view of a micro-object having a respective orientation that is associated with the respective view and is different from each respective orientation of micro-objects in other object images within the set of object images;
generating a synthesized training image corresponding to a micro-assembler backplane on which micro-objects are placed, each synthesized training image comprising:
a selected plurality of object images selected from within the set of object images of micro-objects, wherein each respective object image in the selected plurality of object images is located at a respective location in the synthesized training image; and
a composite set of label data indicating for each respective object image in the selected plurality of object images:
the respective location of the respective object image in the synthesized training image; and
the respective orientation of the micro-object within the respective object image; and
training the machine learning based image processing model based on a generated training data set.
18 . The method of claim 17 , wherein generating the synthesized training image further comprises generating a plurality of synthesized training images, wherein each respective synthesized training image in the plurality of synthesized training images differs from other respective synthesized training images in the plurality of synthesized training images by having at least one respective object image with at least one of: a different location within the synthesized training image; with a different orientation, or both a different location and a different orientation.
19 . The method of claim 17 , wherein generating the synthesized training image further comprises modifying the synthesized training image to include a background pattern, where the background pattern comprises an image corresponding to a surface of a micro-assembler backplane on which micro-objects are placed for movement into defined locations.
20 . The method of claim 17 , wherein providing the set of object images comprises:
receiving a first object image of a micro-object; and
generating at least some object images in the set of object images by at least manipulating the first object image to generate a respective object image that presents the micro-object in its respective view with a selected orientation.
21 . The method of claim 17 , wherein providing the set of object images comprises:
generating at least some object images in the set of object images by at least capturing a plurality of object images of a micro-object where each respective object image in the plurality of object images presents the micro-object with the respective orientation of the respective view.
22 . The method of claim 21 , wherein the set of object images comprises a first number of object images, where the respective view in the object images in the first number of object images have respective orientations that differ from each other by multiple of an offset angle.