Adaptive tutoring system for machine tasks in augmented reality
A machine task tutorial system is disclosed that utilizes augmented reality to enable an expert user to record a tutorial for a machine task that can be learned by different trainee users in an adaptive manner. The machine task tutorial system advantageously utilizes an adaptation model that focuses on spatial and bodily visual presence for machine task tutoring. The machine task tutorial system advantageously enables adaptive tutoring in the recorded-tutorial environment based on machine state and user activity recognition. The machine task tutorial system advantageously utilizes AR to provide tutorial recording, adaptive visualization, and state recognition. In this way, the machine task tutorial system supports more effective apprenticeship and training for machine tasks in workshops or factories.
1. A method for providing tutorial guidance for performing a machine task, the method comprising:
storing, in a memory, tutorial data defining a plurality of steps of a machine task, the plurality of steps including interactions with a machine in an environment;
displaying, on a display, an augmented reality graphical user interface including graphical tutorial elements that convey information regarding the plurality of steps of the machine task, the graphical tutorial elements being superimposed on at least one of (i) the machine and (ii) the environment;
monitoring, with at least one sensor, motions of a first person and states of the machine during a performance of the machine task by the first person by (i) detecting a position of a respective component of the machine in the environment, (ii) determining a bounding box in an image that encompasses the respective component of the machine within the image based on the position of the respective component of the machine, and (iii) determining a state of the respective component using a machine learning model based on the image cropped by the bounding box;
determining, with a processor, during a performance of a respective step of the plurality of steps involving a respective component of the machine, that the first person is stuck in the performance of the respective step based on the monitored motions of the first person and the monitored states of the machine; and
increasing, with the processor, a level of detail of the graphical tutorial elements in the graphical user interface that are displayed with respect to the respective component in response to determining that the first person is stuck in the performance of the machine task.
2. The method of claim 1 , wherein the graphical tutorial elements include a virtual representation of a human that is animated to show a human motion required to perform at least one of the plurality of steps of the machine task.
3. The method of claim 1 , wherein the graphical tutorial elements include a virtual representation of a component of the machine that is animated to show a manipulation of the component required to perform at least one of the plurality of steps of the machine task.
4. The method of claim 3 , wherein the graphical tutorial elements include a virtual arrow superimposed in the environment to indicate a direction of the manipulation of the component required to perform the at least one of the plurality of steps of the machine task.
5. The method of claim 1 , wherein the graphical tutorial elements include at least one of (i) a text description and (ii) a graphical representation of an expected outcome of a respective step in the plurality of steps of the machine task.
6. The method of claim 1 , wherein the graphical tutorial elements include at least one of (i) a text description and (ii) a graphical representation of an expected outcome of a respective group of consecutive steps in the plurality of steps of the machine task.
7. The method according to claim 1 , the monitoring further comprising:
determining whether the first person is looking at a region of interest for a particular step in the plurality of steps of the machine task, the region of interest being defined in the tutorial data.
8. The method according to claim 7 , the determining that the first person is stuck in the performance respective step further comprising:
determining that the first person is stuck in response to the first person looking away from the region of interest for longer than a first threshold amount of time, without interacting with any components of the machine.
9. The method according to claim 7 , the determining that the first person is stuck in the performance respective step further comprising:
determining that the first person is stuck in response to the first person looking at the region of interest for longer than a second threshold amount of time, without interacting with any components of the machine.
10. The method of claim 1 , the monitoring the motions of the first person further comprising:
classifying a current state of the first person based on the monitored motions of the first person.
11. The method of claim 10 , the classifying the current state of the first person further comprising at least one of:
determining whether the first person is changing perspective of the environment; and
determining whether the first person is statically observing the environment.
12. The method of claim 10 , the classifying the current state of the first person further comprising at least one of:
determining whether the first person is interacting with a component of the machine.
13. The method according to claim 12 , the determining that the first person is stuck in the performance respective step further comprising:
determining that the first person is stuck in response to the first person interacting with a component that is not the respective component of the machine for longer than a third threshold amount of time.
14. The method according to claim 12 , the determining that the first person is stuck in the performance respective step further comprising:
determining that the first person is stuck in response to the first person interacting with the respective component of the machine for longer than a fourth threshold amount of time, without setting a state of the respective component to a target state.
15. The method according to claim 1 , the determining that the first person is stuck in the performance respective step further comprising:
operating a finite state machine having a plurality of states based on the monitored motions of the first person and the monitored states of the machine; and
determining that the first person is stuck in response to the finite state machine remaining at a respective state in the plurality of states of the state machine for longer than a respective threshold amount of time.
16. The method according to claim 1 , the increasing the level of detail of the graphical tutorial elements further comprising:
increasing, in the graphical user interface, a number of graphical tutorial elements that are displayed with respect to the respective component of the machine.
17. The method according to claim 1 further comprising:
decreasing, in the graphical user interface, a number of graphical tutorial elements that are displayed with respect to respective component of the machine in response to the first person completing the respective step in the plurality of steps of the machine task that involved the respective component.
18. The method of claim 1 , wherein the tutorial data was previously recorded and generated by a second person.
19. An augmented reality device for providing tutorial guidance for performing a machine task, augmented reality device comprising:
a memory configured to store tutorial data defining a plurality of steps of a machine task, the plurality of steps including interactions with a machine in an environment;
a display screen configured to display an augmented reality graphical user interface including graphical tutorial elements that convey information regarding the plurality of steps of the machine task, the graphical tutorial elements being superimposed on at least one of (i) the machine and (ii) the environment;
at least one sensor configured to measure sensor data; and
a processor operably connected to the memory, the display screen, and the at least one sensor, the processor being configured to:
monitor, based on the sensor data, motions of a first person and states of the machine during a performance of the machine task by the first person by (i) detecting a position of a respective component of the machine in the environment, (ii) determining a bounding box in an image that encompasses the respective component of the machine within the image based on the position of the respective component of the machine, and (iii) determining a state of the respective component using a machine learning model based on the image cropped by the bounding box;
determine, during a performance of a respective step of the plurality of steps involving a respective component of the machine, that the first person is stuck in the performance of the respective step based on the monitored motions of the first person and the monitored states of the machine; and
operate the display screen to increase a level of detail of the graphical tutorial elements in the graphical user interface that are displayed with respect to the respective component in response to determining that the first person is stuck in the performance of the machine task.