Multi-modal learning system
An instruction is presented to a user for making a target gesture. The target gesture may be a portion of an exemplary or model symbol. The instruction may be presented in various ways, such as being printed on a writing surface or being played in audio format through a speaker of a smart pen device. A writing gesture made on a writing surface by the user is digitally captured using a smart pen device. The captured writing gesture is compared to the target gesture and feedback is determined based on this comparison. This feedback may indicate the correctness of the user's writing gesture. The feedback may be presented to the user through various means, including through the smart pen device. The comparison may also be used to determine a next instruction for the user to follow.
1. A method for instructing a student to create a plurality of target symbols, the method comprising:
capturing, by a smart pen device, a plurality of instructional modules, each instructional module comprising a sequence of target gestures made by an instructor on a dot-enabled surface, wherein a target symbol is created by the sequence of target gestures;
presenting to the student on a display of the smart pen device, a representation of the sequence of target gestures of a first instructional module;
capturing, by the smart pen device, a second sequence of gestures made by the student on the dot-enabled surface, wherein a second symbol is created by the second sequence of gestures;
comparing, by the smart pen device, the second symbol to the target symbol associated with the first instructional module and comparing the second sequence of gestures to the sequence of target gestures associated with the first instructional module;
determining, by the smart pen device, feedback based on the comparisons, wherein the feedback indicates a similarity between the target symbol associated with the first instructional module and the second symbol made by the student, and a similarity between the sequence of target gestures associated with the first instructional module and the second sequence of gestures;
presenting, by the smart pen device, the feedback to the student;
selecting, by the smart pen device, a second instructional module based on the comparisons; and
presenting to the student on the display of the smart pen device, a representation of the sequence of target gestures of the second instructional module.
2. The method of claim 1 , wherein the feedback is presented on a display of the smart pen device.
3. The method of claim 1 , wherein the feedback is presented through a speaker of the smart pen device.
4. The method of claim 1 , wherein the feedback is presented haptically through the smart pen device.
5. The method of claim 1 , wherein the feedback is presented substantially in real time as the second sequence of gestures is captured.
6. The method of claim 1 , wherein the sequence of target gestures associated with the first instructional module and the second sequence of gestures are digitally captured using an imaging system in the smart pen device.
7. The method of claim 1 , wherein the dot-enabled surface comprises an encoded coordinate system that is readable by the smart pen device.
8. The method of claim 1 , wherein selecting the second instructional module comprises:
determining a skill level of the student based on the comparisons;
determining a difficulty of each of the instructional modules; and
selecting the second instructional module based on the determined skill level of the user and the difficulty of the second instructional module.
9. A non-transitory computer-readable storage medium storing computer program instructions for instructing a student to create a plurality of target symbols, the instructions when executed by a processor cause the processor to perform steps comprising:
capturing a plurality of instructional modules, each instructional module comprising a sequence of target gestures made by an instructor on a dot-enabled surface, wherein a target symbol is created by the sequence of target gestures;
presenting to the student on a display of the smart pen device, a representation of the sequence of target gestures of a first instructional module;
capturing a second sequence of gestures made on by the student on the dot-enabled surface, wherein a second symbol is created by the second sequence of gestures;
comparing the second symbol to the target symbol associated with the first instructional module and comparing the second sequence of gestures to the sequence of target gestures associated with the first instructional module;
determining feedback based on the comparisons, wherein the feedback indicates a similarity between the target symbol associated with the first instructional module and the second symbol made by the student, and a similarity between the sequence of target gestures associated with the first instructional module and the second sequence of gestures;
presenting the feedback to the student;
selecting a second instructional module based on the comparisons; and
presenting to the student on the display of the smart pen device, a representation of the sequence of target gestures of the second instructional module.
10. The non-transitory computer-readable storage medium of claim 9 , wherein presenting with the smart pen device, the representation of the sequence of target gestures associated with the first instructional module includes presenting to the student, textual instructions for making the sequence of target gestures associated with the first instructional module.
11. The non-transitory computer-readable storage medium of claim 9 , wherein the feedback is presented on a display of the smart pen device.
12. The non-transitory computer-readable storage medium of claim 9 , wherein the feedback is presented substantially in real time as the second sequence of gestures is captured.
13. The non-transitory computer-readable storage medium of claim 9 , wherein the sequence of target gestures associated with the first instructional module and the second sequence of gestures are digitally captured using an imaging system in the smart pen device.
14. The non-transitory computer-readable storage medium of claim 9 , wherein the dot-enabled surface comprises an encoded coordinate system that is readable by the smart pen device.
15. The non-transitory computer-readable storage medium of claim 9 , wherein the instructions causing the processor to select the second instructional module comprise instructions that when executed by the processor cause the processor to:
determine a skill level of the student based on the comparisons;
determine a difficulty of each of the instructional modules; and
select the second instructional module based on the determined skill level of the user and the difficulty of the second instructional module.
16. A smart pen device for instructing a student to create a target symbol, the smart pen device comprising:
a processor; and
a non-transitory computer-readable storage medium storing computer-executable program instructions, the instructions when executed by the processor to perform steps including:
capturing a plurality of instructional modules, each instructional module comprising a sequence of target gestures made by an instructor on a dot-enabled surface, wherein a target symbol is created by the sequence of target gestures;
presenting to the student on a display of the smart pen device, a representation of the sequence of target gestures of a first instructional module;
capturing a second sequence of gestures made by the student on the dot-enabled surface, wherein a second symbol is creating by the second sequence of gestures;
comparing the second symbol to the target symbol associated with the first instructional module and comparing the second sequence of gestures to the sequence of target gestures associated with the first instructional module;
determining feedback based on the comparisons, wherein the feedback indicates a similarity between the target symbol associated with the first instructional module and the second symbol made by the student, and a similarity between the sequence of target gestures associated with the first instructional module and the second sequence of gestures;
presenting the feedback to the student;
selecting a second instructional module based on the comparisons; and
presenting to the student on the display of the smart pen device, a representation of the sequence of target gestures of the second instructional module.
17. The smart pen device of claim 16 , wherein the feedback is presented on a display of the smart pen device.
18. The smart pen device of claim 16 , wherein the feedback is presented through a speaker of the smart pen device.
19. The smart pen device of claim 16 , wherein the feedback is presented haptically through the smart pen device.
20. The smart pen device of claim 16 , wherein the instructions causing the processor to select the second instructional module comprise instructions that when executed by the processor cause the processor to:
determine a skill level of the student based on the comparisons;
determine a difficulty of each of the instructional modules; and
select the second instructional module based on the determined skill level of the user and the difficulty of the second instructional module.