Systems and methods for automatically observing and supervising activities in an environment

1 . A computer-implemented method, comprising:

at an experimentation anomaly detection service:

collecting, by one or more processors and at each timepoint of a set of timepoints and via one or more sensors, environment data from an experimentation workspace;

identifying, by the one or more processors and based at least in part on the environment data, a first object and a second object, wherein:

the first object comprises a arrangement of cavities configured to store one or more fluids; and

the second object comprises a tip configured to distribute the one or more fluids among the arrangement of cavities;

detecting, by the one or more processors, a tip-associated error, wherein the tip-associated error comprises a presence of a droplet within the tip of the second object, the presence of the droplet on the tip of the second object, the presence of bubbles within the one or more fluids within the tip, the presence of a dust particle within the tip, a quantity of liquid within the tip that satisfies a threshold liquid quantity, or a combination thereof;

detecting, by the one or more processors, that the tip of the second object is retreating from a cavity of the arrangement of cavities over a same timepoint of the set of timepoints in which the tip-associated error is present; and

generating, by the one or more processors, a message indicating that the droplet is present within the tip of the second object based at least in part on detecting that the tip of the second object is retreating from the cavity over the same timepoint of the set of timepoints in which the tip-associated error is occurring, thereby enabling a removal of the droplet from the tip of the second object.

2 . The computer-implemented method according to claim 1 , further comprising:

generating, by the one or more processors, a second message indicating the presence of the bubbles within the one or more fluids within the tip, the presence of the dust particle within the tip, or the quantity of liquid within the tip that satisfies the threshold liquid quantity based at least in part on detecting that the tip of the second object is retreating from the cavity over the same timepoint of the set of timepoints in which the tip-associated error is occurring.

3 . The computer-implemented method according to claim 1 , further comprising:

detecting, by the one or more processors, a second tip-associated error, wherein the second tip-associated error comprises a presence of a second droplet on the tip of the second object;

detecting, by the one or more processors, that the tip of the second object is retreating from the cavity of the arrangement of cavities over a same timepoint of the set of timepoints in which the second tip-associated error is present; and

generating, by the one or more processors, a second message indicating that the second droplet is present on the tip of the second object based at least in part on detecting that the tip of the second object is retreating from the cavity over the same timepoint of the set of timepoints in which the second tip-associated error is occurring, thereby enabling a removal of the second droplet from the tip of the second object.

4 . The computer-implemented method according to claim 1 , further comprising:

providing, by the one or more processors and to a user interface, a first object annotation and a second object annotation, wherein the first object annotation represents the tip of the second object and the second object annotation represents the tip-associated error; and

displaying, by the one or more processors, the first object annotation and the second object annotation within a display section of the user interface, wherein the display section depicts the environment data collected by the one or more sensors.

5 . The computer-implemented method according to claim 4 , wherein:

a first subset of sensors of the one or more sensors comprises a first camera positioned at a first angle;

a second subset of sensors of the one or more sensors comprises a second camera positioned at a second angle different from the first angle;

the display section depicts the environment data collected by the first camera;

the user interface comprises a second display section that depicts the environment data collected by the second camera; and

the computer-implemented method further comprises, at the experimentation anomaly detection service:

providing, by the one or more processors and to the user interface, a third object annotation that represents the tip of the second object; and

displaying, by the one or more processors, the third object annotation within the second display section of the user interface.

6 . The computer-implemented method according to claim 5 , wherein the second camera has a higher magnification than the first camera or wherein the first camera or the second camera are configured to adjust a respective zoom.

7 . The computer-implemented method according to claim 4 , wherein the first object annotation comprises a set of connected keypoints that track a shape and orientation of the tip of the second object between timepoints.

8 . The computer-implemented method according to claim 1 , further comprising:

identifying a pattern for visiting the arrangement of cavities for the second object, wherein the pattern for visiting the arrangement indicates to visit a second cavity of the arrangement of cavities subsequent to visiting a first cavity of the arrangement of cavities;

detecting, by the one or more processors, that the tip of the second object has visited the first cavity of the arrangement of cavities;

detecting, by the one or more processors and after the tip of the second object has visited the first cavity of the arrangement of cavities, that the tip of the second object is moving towards or visiting a third cavity of the arrangement of cavities without detecting, after detecting that the tip of the second object has visited the first cavity, that the tip of the second object has visited the second cavity of the arrangement of cavities; and

generating a second message indicating that the tip of the second object has failed to visit the second cavity of the arrangement of cavities.

9 . The computer-implemented method according to claim 8 , wherein:

the first object is overlaid with a third object,

the third object comprises a set of light sources,

each of the set of light sources is configured to display a light onto a respective cavity of the arrangement of cavities,

each cavity of the arrangement of cavities comprises a transparent material, and the second message comprises the respective light source for the third cavity being toggled.

10 . The computer-implemented method according to claim 1 , further comprising:

at the experimentation anomaly detection service:

providing, by the one or more processors and to a user interface, a representation of a path between the tip of the second object and the surface of the first object, wherein the user interface comprises a display section that depicts the environment data collected by the one or more sensors; and

displaying, by the one or more processors, the representation of the path within the display section of the user interface.

11 . The computer-implemented method according to claim 1 , further comprising:

at the experimentation anomaly detection service:

providing, by the one or more processors and to a user interface, a first object annotation that represents a form of the first object, wherein the user interface comprises a display section that depicts the environment data collected by the one or more sensors; and

displaying, by the one or more processors, the first object annotation within the display section of the user interface.

12 . The computer-implemented method according to claim 1 , further comprising:

at the experimentation anomaly detection service:

providing, by the one or more processors and to a user interface, an indication of a first subset of the cavities of the arrangement of cavities that have been visited by the tip of the second object, wherein the user interface comprises a display section that depicts an arrangement of the arrangement of cavities of the first object; and

updating, by the one or more processors, the display section to display the first subset of the cavities of the arrangement of cavities that have been visited by the tip of the second object.

13 . The computer-implemented method according to claim 1 , wherein:

the first object is a microplate;

the arrangement of cavities of the first object are a grid of wells;

the second object is a pipette or a micropipette; and

the tip of the second object is a pipette tip or a micropipette tip.

14 . The computer-implemented method according to claim 1 , wherein the second object comprises a plurality of tips, wherein each of the plurality of tips is configured to distribute the one or more liquids to a respective cavity of the arrangement of cavities.

15 . The computer-implemented method according to claim 1 , further comprising:

tracking, by the one or more processors and based at least in part on the collected environment data, the first object and the second object, wherein detecting the tip-associated error is based at least in part on tracking.

16 . The computer-implemented method according to claim 1 , further comprising:

at the experimentation anomaly detection service:

providing, by the one or more processors and to a language model, an indication that the tip-associated error has occurred; and

generating, by the language model, an explanation for the tip-associated error, wherein the message comprises the explanation for the tip-associated error.

17 . The computer-implemented method according to claim 1 , further comprising:

at the experimentation anomaly detection service:

generating, by the one or more processors and based at least in part on detecting the tip-associated error, a second message indicating a task for resolving the tip-associated error; and

providing, by the one or more processors, the generated second message to a user interface.

18 . The computer-implemented method according to claim 1 , wherein:

the experimentation workspace is a training environment;

the tip-associated error is based at least in part on one or more actions of a trainee within the training environment; and

the one or more actions comprise an erroneous action performed by a trainee within the training environment, wherein the tip-associated error is detected based at least in part on the erroneous action.

19 . A computer-implemented system, comprising:

one or more processors;

a memory; and

a computer-readable medium operably coupled to the one or more processors, the computer-readable medium having computer-readable instructions stored thereon that, when executed by the one or more processors, cause a computing device to perform operations comprising:

at an experimentation anomaly detection service:

collecting, by the one or more processors and at each timepoint of a set of timepoints and via one or more sensors, environment data from an experimentation workspace;

identifying, by the one or more processors and based at least in part on the environment data, a first object and a second object, wherein:

the first object comprises an arrangement of cavities configured to store one or more fluids; and

the second object comprises a tip configured to distribute the one or more fluids among the arrangement of cavities;

detecting, by the one or more processors, a tip-associated error, wherein the tip-associated error comprises a presence of a droplet within the tip of the second object, the presence of the droplet on the tip of the second object, the presence of bubbles within the one or more fluids within the tip, the presence of a dust particle within the tip, a quantity of liquid within the tip that satisfies a threshold liquid quantity, or a combination thereof;

detecting, by the one or more processors, that the tip of the second object is retreating from a cavity of the arrangement of cavities over a same timepoint of the set of timepoints in which the tip-associated error is present; and

generating, by the one or more processors, a message indicating that the droplet is present within the tip of the second object based at least in part on detecting that the tip of the second object is retreating from the cavity over the same timepoint of the set of timepoints in which the tip-associated error is occurring, thereby enabling a removal of the droplet from the tip of the second object.

20 . A computer-program product comprising a non-transitory machine-readable storage medium storing computer instructions that, when executed by processing circuitry, perform operations comprising:

at an experimentation anomaly detection service:

collecting, by the processing circuitry and at each timepoint of a set of timepoints and via one or more sensors, environment data from an experimentation workspace;

identifying, by the processing circuitry and based at least in part on the environment data, a first object and a second object, wherein:

the first object comprises an arrangement of cavities configured to store one or more fluids; and

the second object comprises a tip configured to distribute the one or more fluids among the arrangement of cavities;

detecting, by the processing circuitry, a tip-associated error, wherein the tip-associated error comprises a presence of a droplet within the tip of the second object, the presence of the droplet on the tip of the second object, the presence of bubbles within the one or more fluids within the tip, the presence of a dust particle within the tip, a quantity of liquid within the tip that satisfies a threshold liquid quantity, or a combination thereof;

detecting, by the processing circuitry, that the tip of the second object is retreating from a cavity of the arrangement of cavities over a same timepoint of the set of timepoints in which the tip-associated error is present; and

generating, by the processing circuitry, a message indicating that the droplet is present within the tip of the second object based at least in part on detecting that the tip of the second object is retreating from the cavity over the same timepoint of the set of timepoints in which the tip-associated error is occurring, thereby enabling a removal of the droplet from the tip of the second object.