Passive haptic training system and methods
Exemplary systems and wearable haptic device systems are disclosed for training a user to improve perception or discrimination. An exemplary system can be used to train a user to play a musical instrument, such as a piano, in passive training sessions via a wearable haptic device. The exemplary wearable haptic device can be integrated with sensors and in operative communication with a passive haptic learning system (e.g., cloud-based infrastructure) that is configured to generate, update, and/or modify tactile training data for generating tactile outputs at the wearable haptic device.
1 . A computer-implemented method for generating tactile training data for training a user to learn to play a musical instrument, for use in a therapy or treatment session, and/or to improve perception or discrimination comprising:
receiving, by one or more processors, a musical notation file or data object having musical symbols that indicate pitches, rhythms, or chords of a song or instrumental musical piece;
generating, by the one or more processors, fingering position data for a musical instrument by:
identifying, by the one or more processors, repeated segments of musical symbols in the musical notation file or data object; and
determining at least one repeated segment having a pattern that meets one or more predefined static key constraints and/or one or more predefined dynamic key constraints;
generating, by the one or more processors, tactile training data for at least one of the repeated segments by assigning a predefined tactile value associated with tactile outputs at a wearable haptic device, wherein the tactile training data includes a replication of at least one of the repeated segments; and
storing, by the one or more processors, tactile training data in a database or local storage device, wherein the tactile training data is subsequently used to generate the tactile outputs at the wearable haptic device.
2 . The computer-implemented method of claim 1 , wherein generating, by the one or more processors, fingering position data for the musical instrument further comprises:
dividing the remaining content in the musical notation file or data object into segments surrounding the identified repeated segments.
3 . The computer-implemented method of claim 1 , wherein identifying, by the one or more processors, repeated segments of musical symbols comprises:
processing, by the one or more processors, the musical notation file or data object using a trained machine learning model configured to classify the musical symbols as groups of symbols.
4 . The computer-implemented method of claim 1 , wherein the one or more predefined static key constraints comprises fingering position or hand size and the one or more predefined dynamic key constraints comprises hand positioning, hand spread, and/or hand movement information.
5 . The computer-implemented method of claim 1 , wherein the wearable haptic device is configured to adjust the tactile outputs to one of the predefined tactile value for the one or more predefined static key constraints, one or more predefined dynamic key constraints, and a plurality of global rules.
6 . The computer-implemented method of claim 1 , wherein identifying, by the one or more processors, repeated segments of musical segments comprises:
processing, by the one or more processors, the musical notation file or data object using a trained machine learning model that evaluates the musical symbols in the musical notation file or data object using a sliding window operation.
7 . The computer-implemented method of claim 1 , further comprising:
receiving, by the one or more processors, feedback data comprising at least one of audio sensor data, video sensor data, motion sensor data or bend sensor data obtained during an active training session;
evaluating, by the one or more processors, a difference between (i) the pitches, rhythms, or chords of the musical notation file or data object and (ii) corresponding pitches, rhythms, or chords of the audio sensor data or video sensor data,
wherein the determined difference is used to (i) modify, by the one or more processors, the stored tactile training data for a future lesson or (ii) generate additional tactile training data for a new lesson.
8 . The computer-implemented method of claim 7 , wherein evaluating the difference comprises:
converting, by the one or more processors, at least a portion of the feedback data into musical symbol data, pitch data, rhythm data, and/or chord data that is compared to the musical symbol data, pitch data, rhythm data, and/or chord data of the musical notation file or data object.
9 . The computer-implemented method of claim 7 , wherein evaluating the difference comprises:
performing, by the one or more processors, a sequence matching operation on at least a portion of the musical notation file or data object and the feedback data.
10 . The computer-implemented method of claim 1 , wherein the tactile training data includes 10-25 actions for a passive training sessions.
11 . The computer-implemented method of claim 1 , wherein the wearable haptic device comprises one or more gloves that each comprise a set of actuators that are associated with a target location of the user's hand.
12 . The computer-implemented method of claim 1 , wherein the wearable haptic device comprises one or more wearable devices that each comprise a set of one or more actuators.
13 . A system for training a user to learn how to play a musical instrument, for use in a therapy or treatment session, and/or to improve perception or discrimination comprising:
at least one wearable haptic device comprising a plurality of actuators; and
a controller operatively coupled to the at least one wearable haptic device that is configured to:
obtain tactile training data for a user of the at least one wearable haptic device, wherein the tactile training data comprises:
(i) fingering position data for a musical instrument,
(ii) a replication of at least one repeated segment of musical symbols in a musical notation file or data object having symbols that indicate pitches, rhythms, or chords of a song or instrumental musical piece, and
(iii) a predefined tactile value associated with tactile outputs at the wearable haptic device.
14 . The system of claim 13 , wherein the fingering position data is determined by:
identifying repeated segments of musical symbols in the musical notation file or data object, and
determining at least one repeated segment having a pattern that meets one or more predefined static key constraints and/or one or more predefined dynamic key constraints.
15 . The system of claim 14 , wherein the one or more predefined static key constraints comprises fingering position or hand size and the one or more predefined dynamic key constraints comprises hand positioning, hand spread, or hand movement information.
16 . The system of claim 14 , wherein the wearable haptic device is configured to adjust the tactile outputs to one of the predefined tactile values for the one or more predefined static key constraints, one or more predefined dynamic key constraints, and a plurality of global rules.
17 . The system of claim 13 , wherein the controller is further configured to:
receive feedback data comprising audio sensor data or video sensor data obtained during an active training session;
evaluate a difference between (i) the pitches, rhythms, or chords of the musical notation file or data object and (ii) corresponding pitches, rhythms, or chords of the audio sensor data or video sensor data,
wherein the determined difference is used to (i) modify the stored tactile training data for a future lesson or (ii) generate additional tactile training data for a new lesson.
18 . The system of claim 17 , wherein evaluating the difference comprises performing a sequence matching operation on at least a portion of the musical notation file or data object and the feedback data.
19 . The system of claim 17 , wherein the tactile training data includes 10-25 actions.
20 . The system of claim 13 , wherein the at least one wearable haptic device comprises a first wearable glove comprising a first set of actuators, and a second wearable glove comprising a second set of actuators.
21 . The system of claim 20 , wherein each actuator is configured to stimulate a target area of the user's hands.