System for multimodal machine-aided comprehension analysis and assistance
Described is a system for multimodal machine-aided comprehension analysis. The system can be implemented in an augmented reality headset that, in conjunction with a processor, generates an initial scene graph of a scene proximate the user. Items and labels are presented, with the headset tracking eye movements of the user as the user gazes upon the subject labels, item labels, and relationship labels. A resulting scene graph (having relationship triplets) is generated based on the eye movements of the user and an amount of time the user spends gazing upon each of the display components. A comprehension model is generated by estimating a user's comprehension of the relationship triplets, with a knowledge model being generated based on a known knowledge graph and the comprehension model. Cues are then presented to the user based on the comprehension and knowledge models to assist the user in their comprehension of the scene.
1. A system for multimodal machine-aided comprehension analysis, the system comprising:
one or more processors and associated memory, the memory being a non-transitory computer-readable medium having executable instructions encoded thereon, such that upon execution of the instructions, the one or more processors perform operations of:
generating an initial scene graph of a scene proximate a user based on an image of the scene, the initial scene graph having one or more subjects and objects, with subject labels, item labels, and relationship labels;
tracking eye movements of the user as the user gazes upon the subject labels, item labels, and relationship labels;
generating a resulting scene graph based on the eye movements of the user and an amount of time the user spends gazing upon each of the subject labels, item labels and relationship labels, the resulting scene graph connecting the subject labels, item labels and relationship labels as relationship triplets;
generating a comprehension model by estimating a user's comprehension of the relationship triplets in the image based on the user's gaze data;
generating a knowledge model based on a known knowledge graph and the comprehension model, the knowledge model specifying the user's background knowledge level and comprehension level; and
generating a cue and presenting the cue to the user.
2. The system as set forth in claim 1 , wherein the cue is generated and presented to the user via an augmented reality headset.
3. The system as set forth in claim 2 , wherein if the user has comprehension below a predetermined comprehension threshold and background knowledge above a predetermined background knowledge threshold, then the cue is a visual cue.
4. The system as set forth in claim 3 , wherein if the user has comprehension below a predetermined comprehension threshold and background knowledge below a predetermined background knowledge threshold, then the cue is a dialog-based cue.
5. The system as set forth in claim 4 , wherein the comprehension model is generated by determining a comprehension value for each possible triplet in the scene graph, based on
Comprehension
Relationship
=
T
1
+
T
2
C
1
+
T
T
,
where T 1 denotes a gaze time looking at an object in the relationship triplet, T 2 denotes a gaze time looking at a subject in the relationship triplet, C 1 denotes a constant calibration value, and T T denotes a shortest time between looking at one item before another item.
6. The system as set forth in claim 1 , wherein if the user has comprehension below a predetermined comprehension threshold and background knowledge above a predetermined background knowledge threshold, then the cue is a visual cue.
7. The system as set forth in claim 1 , wherein if the user has comprehension below a predetermined comprehension threshold and background knowledge below a predetermined background knowledge threshold, then the cue is a dialog-based cue.
8. The system as set forth in claim 1 , wherein the comprehension model is generated by determining a comprehension value for each possible triplet in the scene graph, based on
Comprehension
Relationshi
=
T
1
+
T
2
C
1
+
T
T
,
where T 1 denotes a gaze time looking at an object in the relationship triplet, T 2 denotes a gaze time looking at a subject in the relationship triplet, C 1 denotes a constant calibration value, and T T denotes a shortest time between looking at one item before another item.
9. A computer implemented method for multimodal machine-aided comprehension analysis, the method comprising an act of:
causing one or more processers to execute instructions encoded on a non-transitory computer-readable medium, such that upon execution, the one or more processors perform operations of:
generating an initial scene graph of a scene proximate a user based on an image of the scene, the initial scene graph having one or more subjects and objects, with subject labels, item labels, and relationship labels;
tracking eye movements of the user as the user gazes upon the subject labels, item labels, and relationship labels;
generating a resulting scene graph based on the eye movements of the user and an amount of time the user spends gazing upon each of the subject labels, item labels and relationship labels, the resulting scene graph connecting the subject labels, item labels and relationship labels as relationship triplets;
generating a comprehension model by estimating a user's comprehension of the relationship triplets in the image based on the user's gaze data;
generating a knowledge model based on a known knowledge graph and the comprehension model, the knowledge model specifying the user's background knowledge level and comprehension level; and
generating a cue and presenting the cue to the user.
10. The method as set forth in claim 9 , wherein the cue is generated and presented to the user via an augmented reality headset.
11. The method as set forth in claim 10 , wherein if the user has comprehension below a predetermined comprehension threshold and background knowledge above a predetermined background knowledge threshold, then the cue is a visual cue.
12. The method as set forth in claim 11 , wherein if the user has comprehension below a predetermined comprehension threshold and background knowledge below a predetermined background knowledge threshold, then the cue is a dialog-based cue.
13. The method as set forth in claim 12 , wherein the comprehension model is generated by determining a comprehension value for each possible triplet in the scene graph, based on
Comprehension
Relationship
=
T
1
+
T
2
C
1
+
T
T
,
where T 1 denotes a gaze time looking at an object in the relationship triplet, T 2 denotes a gaze time looking at a subject in the relationship triplet, C 1 denotes a constant calibration value, and T T denotes a shortest time between looking at one item before another item.
14. The method as set forth in claim 9 , wherein if the user has comprehension below a predetermined comprehension threshold and background knowledge above a predetermined background knowledge threshold, then the cue is a visual cue.
15. The method as set forth in claim 9 , wherein if the user has comprehension below a predetermined comprehension threshold and background knowledge below a predetermined background knowledge threshold, then the cue is a dialog-based cue.
16. The method as set forth in claim 9 , wherein the comprehension model is generated by determining a comprehension value for each possible triplet in the scene graph, based on
Comprehension
Relationship
=
T
1
+
T
2
C
1
+
T
T
,
where T 1 denotes a gaze time looking at an object in the relationship triplet, T 2 denotes a gaze time looking at a subject in the relationship triplet, C 1 denotes a constant calibration value, and T T denotes a shortest time between looking at one item before another item.
17. A computer program product for multimodal machine-aided comprehension analysis, the system comprising:
a non-transitory computer-readable medium having executable instructions encoded thereon, such that upon execution of the instructions by one or more processors, the one or more processors perform operations of:
generating an initial scene graph of a scene proximate a user based on an image of the scene, the initial scene graph having one or more subjects and objects, with subject labels, item labels, and relationship labels;
tracking eye movements of the user as the user gazes upon the subject labels, item labels, and relationship labels;
generating a resulting scene graph based on the eye movements of the user and an amount of time the user spends gazing upon each of the subject labels, item labels and relationship labels, the resulting scene graph connecting the subject labels, item labels and relationship labels as relationship triplets;
generating a comprehension model by estimating a user's comprehension of the relationship triplets in the image based on the user's gaze data;
generating a knowledge model based on a known knowledge graph and the comprehension model, the knowledge model specifying the user's background knowledge level and comprehension level; and
generating a cue and presenting the cue to the user.
18. The computer program product as set forth in claim 17 , wherein the cue is generated and presented to the user via an augmented reality headset.
19. The computer program product as set forth in claim 18 , wherein if the user has comprehension below a predetermined comprehension threshold and background knowledge above a predetermined background knowledge threshold, then the cue is a visual cue.
20. The computer program product as set forth in claim 19 , wherein if the user has comprehension below a predetermined comprehension threshold and background knowledge below a predetermined background knowledge threshold, then the cue is a dialog-based cue.
21. The computer program product as set forth in claim 20 , wherein the comprehension model is generated by determining a comprehension value for each possible triplet in the scene graph, based on
Comprehension
Relationship
=
T
1
+
T
2
C
1
+
T
T
,
where T 1 denotes a gaze time looking at an object in the relationship triplet, T 2 denotes a gaze time looking at a subject in the relationship triplet, C 1 denotes a constant calibration value, and T T denotes a shortest time between looking at one item before another item.
22. The computer program product as set forth in claim 17 , wherein if the user has comprehension below a predetermined comprehension threshold and background knowledge above a predetermined background knowledge threshold, then the cue is a visual cue.
23. The computer program product as set forth in claim 17 , wherein if the user has comprehension below a predetermined comprehension threshold and background knowledge below a predetermined background knowledge threshold, then the cue is a dialog-based cue.
24. The computer program product as set forth in claim 17 , wherein the comprehension model is generated by determining a comprehension value for each possible triplet in the scene graph, based on
Comprehension
Relationship
=
T
1
+
T
2
C
1
+
T
T
,
where T 1 denotes a gaze time looking at an object in the relationship triplet, T 2 denotes a gaze time looking at a subject in the relationship triplet, C 1 denotes a constant calibration value, and T T denotes a shortest time between looking at one item before another item.