Systems and methods for customizing playback of digital tutorials
Systems and methods are disclosed herein for continuing playback of a digital tutorial until a user interrupts the playback by signaling to the system that there is an issue or that the user needs help. The system, through detecting a recording that the user captured or a person's utterance (e.g., through passive voice monitoring) determines that the user's needs assistance with the digital tutorial. The system determines, based on the recording, that the user needs help to get to a specific step and play supplemental instructions to the user to get to the specific step.
1 . A method comprising:
capturing, via a camera of a device, image data of a current state of a task;
storing the captured image data on a memory of the device;
detecting a user utterance;
determining a target state of the task based on the user utterance, wherein determining the target state of the task comprises:
detecting that the user utterance matches metadata associated with a particular state of the task; and
determining that the particular state of the task is the target state of the task;
determining, using a trained neural network, based on the image data, whether the current state of the task matches the target state of the task that was determined based on the user utterance; and
in response to determining that the current state of the task does not match a target state of the task, causing to be output a recommendation to bring the current state of the task to the target state of the task.
2 . The method of claim 1 , further comprising, generating the trained neural network by:
inputting a plurality of images to a neural network, each image of the plurality of images corresponding to a respective state of a plurality of states of the task; and
iteratively updating weights associated with nodes in the neural network.
3 . The method of claim 1 , wherein the task comprises a plurality of steps, and wherein determining, using the trained neural network, based on the image data, whether the current state of the task matches a target state of the task, comprises:
determining, using the trained neural network, based on the image data, a step, of the plurality of steps, corresponding to the current state of the task; and
determining whether the step corresponding to the current state of the task matches a step, of the plurality of steps, corresponding to the target state of the task.
4 . The method of claim 3 , wherein causing to be output the recommendation to bring the current state of the task to the target state of the task comprises:
identifying an instruction associated with the step corresponding to the current state of the task; and
causing to be output the instruction.
5 . The method of claim 1 , wherein the recommendation comprises at least one of an audio based output and a visual based output.
6 . The method of claim 1 , further comprising prompting a user to enable video capture, prior to the capturing, via the camera of the device, image data of the current state of the task.
7 . The method of claim 1 , further comprising, in response to determining that the current state of the task matches the target state of the task:
identifying a next state associated with the task, wherein the next state follows the current state in a sequence of states associated with the task; and
outputting instructions corresponding to the next state.
8 . The method of claim 1 , further comprising accessing the trained neural network via a network connection of the device.
9 . A system comprising:
a camera;
a memory; and
control circuitry configured to:
cause to be captured, via the camera, image data of a current state of a task;
cause to be stored the captured image data on the memory;
detect a user utterance;
determine a target state of the task based on the user utterance, wherein determining the target state of the task comprises:
detecting that the user utterance matches metadata associated with a particular state of the task; and
determining that the particular state of the task is the target state of the task;
determine, using a trained neural network, based on the image data, whether the current state of the task matches the target state of the task that was determined based on the user utterance; and
in response to determining that the current state of the task does not match a target state of the task, cause to be output a recommendation to bring the current state of the task to the target state of the task.
10 . The system of claim 9 , wherein the control circuitry is further configured to generate the trained neural network by:
inputting a plurality of images to a neural network, each image of the plurality of images corresponding to a respective state of a plurality of states of the task; and
iteratively updating weights associated with nodes in the neural network.
11 . The system of claim 9 , wherein the task comprises a plurality of steps, and wherein the control circuitry is further configured, when determining, using the trained neural network, based on the image data, whether the current state of the task matches a target state of the task, to:
determine, using the trained neural network, based on the image data, a step, of the plurality of steps, corresponding to the current state of the task; and
determine whether the step corresponding to the current state of the task matches a step, of the plurality of steps, corresponding to the target state of the task.
12 . The system of claim 11 , wherein the control circuitry is further configured, when causing to be output the recommendation to bring the current state of the task to the target state of the task to:
identify an instruction associated with the step corresponding to the current state of the task; and
cause to be output the instruction.
13 . The system of claim 9 , further comprising at least one of a display and a speaker, wherein the recommendation comprises at least one of an audio based output and a visual based output.
14 . The system of claim 9 , wherein the control circuitry is further configured to prompt a user to enable video capture, prior to the capturing, via the camera, image data of the current state of the task.
15 . The system of claim 9 , wherein the control circuitry is further configured, in response to determining that the current state of the task matches the target state of the task, to:
identify a next state associated with the task, wherein the next state follows the current state in a sequence of states associated with the task; and
output instructions corresponding to the next state.
16 . The system of claim 9 , further comprising a network connection, wherein the control circuitry is further configured to access the trained neural network via the network connection.