Apparatus, method, and system of insight-based cognitive assistant for enhancing user's expertise in learning, review, rehearsal, and memorization

1. A personal emotion-based learning assistant system comprising:

at least one data capture apparatus which captures data comprising synchronized visual and audio information and which captures corresponding cognitive signals generated from a user; and

at least one display apparatus configured to:

output multi-component information of the visual and audio information in different display areas of the at least one display apparatus, wherein the different display areas comprise:

at least one first video playing area in which a video of the visual and audio information is played, and

at least one second additional information area in which a plurality of synchronized text and symbol scrolling segments are displayed, wherein the plurality of synchronized text and symbol scrolling segments comprise at least one of the visual and audio information converted to text and symbols and are displayed synchronously with the played video;

generate and display, on at least one of the first video playing area and the second additional information area, a plurality of indicators synchronized with the displayed text and symbol segments;

divide the data captured by the at least one data capture apparatus into a plurality of segments;

for each of the plurality of segments, determine a first cognitive state and a second cognitive state of the user, from among a plurality of types of cognitive states and a first level of the determined first cognitive state and a second level of the determined second cognitive state, from among a plurality of levels of the cognitive states,

wherein the first cognitive state is determined based on the cognitive signals generated from the user during the capturing of the data by the data capture apparatus,

wherein the second cognitive state is determined based on the cognitive signals generated from the user during replay of the data in the at least one first video playing area,

wherein the plurality of indicators indicate the first cognitive state and the second cognitive state for a corresponding segment from among the plurality of segments.

2. The personal emotion-based learning assistant system according to claim 1 , wherein at least one timeline is displayed in at least one of the at least one first video playing area and the at least one second additional information area.

3. The personal emotion-based learning assistant system according to claim 2 , further comprising:

a user interface configured to receive input from the user,

wherein, in response to receiving the input from the user via the user interface to move to another text and symbol segment in the at least one second additional information area, the at least one display apparatus displays the another text and symbol segment, plays the video from a different location corresponding to the another text and symbol segment, and moves a current position indicator on the timeline to another location corresponding to the another text and symbol segment, and

wherein, in response to receiving the input from the user via the user interface to move the current position indicator on the timeline to the another location, the at least one display apparatus displays the current position indicator on the timeline at the another location, plays the video from a point corresponding to the another location and displays the text and symbol segment corresponding to the another location.

4. The personal emotion-based learning assistant system according to claim 2 , wherein the at least one timeline is color coded based on at least one of the first cognitive state of the user and the second cognitive state of the user and at least one of the first level of the cognitive state and the second level of the cognitive state, for said each of the plurality of segments.

5. The personal emotion-based learning assistance system according to claim 2 , wherein the at least one timeline comprises:

a first timeline displayed in the at least one first video playing area, which has the plurality of indicators synchronized with the video, wherein the plurality of indicators indicate the first cognitive state and the first level of the cognitive state, of the user during an original presentation of the captured data, and

a second timeline displayed in the at least one second additional information area, which has the plurality of indicators synchronized with the displayed text and symbol segments, wherein the plurality of indicators indicate the second cognitive state and the second level of the cognitive state, of the user during review in which the video is played in the at least one first video playing area.

6. The personal emotion-based learning assistant system according to claim 1 , wherein the at least one display apparatus comprises:

a first display apparatus which displays the at least one first video playing area; and

a second display apparatus which displays the at least one second additional information area.

7. The personal emotion-based learning assistant system according to claim 1 , wherein the at least one display apparatus further displays a third area comprising at least one of images, graphs, texts, and sketches related to a corresponding respective segment currently being displayed.

8. The personal emotion-based learning assistant system according to claim 1 , wherein the at least one display apparatus further comprises a motion to sketch converter configured to convert motion information of an object depicted in the plurality of segments into synchronized sketch and symbols.

9. A personal emotion-based learning assistant method comprising:

capturing, by at least one data capture apparatus, data comprising synchronized visual and audio information;

capturing, by the at least one data capture apparatus, cognitive signals generated from a user corresponding to the captured data;

outputting multi-component information of the captured synchronized visual and audio information in different display areas, wherein the different display areas comprise:

at least one first video playing area in which a video of the captured synchronized visual and audio information is played, and

at least one second additional information area in which a plurality of synchronized text and symbol scrolling segments are displayed, wherein the plurality of synchronized text and symbol scrolling segments comprise at least one of the captured synchronized visual and audio information converted to text and symbols and are displayed synchronously with the played video;

generating and displaying, by at least one display apparatus, on at least one of the first video playing area and the second additional information area, a plurality of indicators synchronized with the displayed text and symbol segments;

dividing the data captured by the at least one data capture apparatus into a plurality of segments; and

for each of the plurality of segments, determining a first cognitive state and a second cognitive state of the user, from among a plurality of types of cognitive states and determining a first level of the determined first cognitive state and a second level of the determined second cognitive state, from among a plurality of levels of the cognitive states,

wherein the first cognitive state is determined based on the cognitive signals generated from the user during the capturing of the data by the data capture apparatus,

wherein the second cognitive state is determined based on the cognitive signals generated from the user during a replay of the data in the at least one first video playing area, and

wherein the plurality of indicators indicate the first cognitive state and the second cognitive state for a corresponding segment from among the plurality of segments.

10. The personal emotion-based learning assistant method according to claim 9 , wherein at least one timeline is displayed in at least one of the at least one first video playing area and the at least one second additional information area.

11. The personal emotion-based learning assistant method according to claim 10 , further comprising:

receiving, via a user interface, input from the user;

in response to the receiving the input from the user to move to another text and symbol segment in the at least one second additional information area, displaying the another text and symbol segment, playing the video from a different location corresponding to the another text and symbol segment, and moving a current position indicator on the timeline to another location corresponding to the another text and symbol segment; and

in response to the receiving the input from the user to move the current position indicator on the timeline to the another location, displaying the current position indicator on the timeline at the another location, playing the video from a point corresponding to the another location, and displaying the text and symbol segment corresponding to the another location.

12. The personal emotion-based learning assistant method according to claim 10 , wherein the at least one timeline is color coded based on at least one of the determined first cognitive state of the user and the determined second cognitive state of the user and at least one of the determined first level of the cognitive state of the user and the determined second level of the cognitive state, for said each of the plurality of segments.

13. The personal emotion-based learning assistant method according to claim 9 , wherein the at least one first video playing area is displayed on a first display of a first display apparatus and the at least one second additional information area is displayed on a second display of a second display apparatus.

14. The personal emotion-based learning assistant method according to claim 9 , further comprising:

displaying a third area comprising at least one of images, graphs, texts, and sketches related to a corresponding respective segment currently being displayed.

15. The personal emotion-based learning assistant method according to claim 9 , further comprising:

converting motion information of an object depicted in the plurality of segments into synchronized sketch and symbols.

16. A non-transitory computer readable recording medium storing therein a personal emotion-based cognitive assistant method, which when executed by a computer causes the computer to:

control at least one data capture apparatus to capture data comprising synchronized visual and audio information;

control the at least one data capture apparatus to capture cognitive signals generated from a user corresponding to the captured data;

output multi-component information of the visual and audio information in different display areas,

wherein the different display areas comprise:

at least one first video playing area in which a video of the visual and audio information is played, and

at least one second additional information area in which synchronized text and symbol scrolling segments are displayed, wherein the synchronized text and symbol scrolling segments comprise at least one of the visual and audio information converted to text and symbols and are displayed synchronously with the played video;

generate and display on at least one of the first video playing area and the second additional information area, a plurality of indicators synchronized with the displayed synchronized text and symbol segments;

divide the data captured by the at least one data capture apparatus into a plurality of segments; and

for each of the plurality of segments, determine a first cognitive state of the user and a second cognitive state of the user, from among a plurality of types of cognitive states and determine a first level of the cognitive state and a second level of the cognitive state, from among a plurality of levels of the cognitive state,

wherein the first cognitive state is determined based on the cognitive signals generated from the user during the capturing of the data by the data capture apparatus,

wherein the second cognitive state is determined based on the cognitive signals generated from the user during a replay of the data in the at least one first video playing area, and

wherein the plurality of indicators indicate the first cognitive state and the second cognitive state for a corresponding segment from among the plurality of segments.