LEARNING CURVE DISAGGREGATION BY STUDENT MASTERY
Techniques are described for disaggregating learning curves by student mastery for refining and accurately evaluating automated tutoring models. A method comprises receiving performance data for users logging whether a correct response was provided for each opportunity to use a particular skill in a tutoring system, determining a plurality of subpopulations from the users by using the performance data to group by number of opportunities needed for the particular skill to reach a mastery threshold, creating disaggregated learning curves for each of the plurality of subpopulations that map performance opportunities to percentages correct, and evaluating the disaggregated learning curves to identify a suitable adaptation for the tutoring system. The suitable adaptation may then be carried out and may include sending a notification of portions of the tutoring system that need attention and/or adjusting parameters of the tutoring system for a projected learning progression of a particular user.
1 . A method comprising:
receiving performance data for a plurality of users, the performance data logging whether a correct response or an incorrect response was provided for each opportunity to use a particular skill in a tutoring system;
determining a plurality of subpopulations from the plurality of users by using the performance data to assign the plurality of users to groups, wherein each user of the plurality of users is assigned to a group based, at least in part, on number of opportunities needed for the user to reach a mastery threshold for the particular skill in the tutoring system;
creating disaggregated learning curves for each of the plurality of subpopulations by mapping said each opportunity to use the particular skill to a value based, at least in part, on a number of users in each subpopulation that provided the correct response for said each opportunity;
evaluating the disaggregated learning curves to identify a suitable adaptation for the tutoring system;
wherein the method is performed by one or more computing devices.
2 . The method of claim 1 , wherein the method further comprises:
causing the suitable adaptation to be carried out.
3 . The method of claim 1 , wherein the evaluating determines whether each of the disaggregated learning curves fits a power function meeting a minimum exponent, the fitting demonstrating a learning of the particular skill by an associated subpopulation.
4 . The method of claim 3 , wherein the suitable adaptation comprises weighing an attention metric to identify a portion of the tutoring system for attention, wherein the attention metric is based on a percentage of the plurality of users that demonstrated the learning of the particular skill.
5 . The method of claim 4 , further comprising sending a notification concerning the portion of the tutoring system for attention.
6 . The method of claim 1 , wherein the evaluating determines a membership of a particular user within a particular subpopulation from the plurality of subpopulations.
7 . The method of claim 6 , wherein the suitable adaptation comprises adjusting the tutoring system for the particular user based on the determined membership of the particular user.
8 . The method of claim 7 , wherein the adjusting of the tutoring system is for an in-progress tutoring session.
9 . The method of claim 1 , wherein the determining of the plurality of subpopulations uses Bayesian Knowledge Tracing.
10 . A tutoring system comprising one or more computing devices configured to:
receive performance data for a plurality of users, the performance data logging whether a correct response or an incorrect response was provided for each opportunity to use a particular skill in the tutoring system;
determine a plurality of subpopulations from the plurality of users by using the performance data to assign the plurality of users to groups, wherein each user of the plurality of users is assigned to a group based, at least in part, on number of opportunities needed for the user to reach a mastery threshold for the particular skill in the tutoring system;
create disaggregated learning curves for each of the plurality of subpopulations by mapping said each opportunity to use the particular skill to a value based, at least in part, on a number of users in each subpopulation that provided the correct response for said each opportunity;
evaluate the disaggregated learning curves to identify a suitable adaptation for the tutoring system.
11 . The tutoring system of claim 10 , wherein the tutoring system is configured to evaluate by determining whether each of the disaggregated learning curves fits a power function meeting a minimum exponent, the fitting demonstrating a learning of the particular skill by an associated subpopulation.
12 . The tutoring system of claim 11 , wherein the suitable adaptation comprises calculating an attention metric using a population percentage to identify a portion of the tutoring system for attention, wherein the population percentage corresponds to a percentage of the plurality of users that demonstrated the learning of the particular skill.
13 . The tutoring system of claim 11 , wherein the tutoring system is configured to evaluate by determining a membership of a particular user within a particular subpopulation from the plurality of subpopulations, and wherein the suitable adaptation comprises adjusting the tutoring system for the particular user based on the determined membership of the particular user.
14 . A non-transitory computer-readable medium storing one or more sequences of instructions which, when executed by one or more processors, cause performing of:
receiving performance data for a plurality of users, the performance data logging whether a correct response or an incorrect response was provided for each opportunity to use a particular skill in a tutoring system;
determining a plurality of subpopulations from the plurality of users by using the performance data to assign the plurality of users to groups, wherein each user of the plurality of users is assigned to a group based, at least in part, on number of opportunities needed for the user to reach a mastery threshold for the particular skill in the tutoring system;
creating disaggregated learning curves for each of the plurality of subpopulations by mapping said each opportunity to use the particular skill to a value based, at least in part, on a number of users in each subpopulation that provided the correct response for said each opportunity;
evaluating the disaggregated learning curves to identify a suitable adaptation for the tutoring system.
15 . The non-transitory computer-readable medium of claim 14 , wherein the one or more sequences of instructions further cause performing of:
causing the suitable adaptation to be carried out.
16 . The non-transitory computer-readable medium of claim 14 , wherein the evaluating determines whether each of the disaggregated learning curves fits a power function meeting a minimum exponent, the fitting demonstrating a learning of the particular skill by an associated subpopulation.
17 . The non-transitory computer-readable medium of claim 16 , wherein the suitable adaptation comprises weighing an attention metric to identify a portion of the tutoring system for attention, wherein the attention metric is based on a percentage of the plurality of users that demonstrated the learning of the particular skill.
18 . The non-transitory computer-readable medium of claim 17 , wherein the one or more sequences of instructions further cause:
sending a notification concerning the portion of the tutoring system for attention.
19 . The non-transitory computer-readable medium of claim 14 , wherein the evaluating determines a membership of a particular user within a particular subpopulation from the plurality of subpopulations.
20 . The non-transitory computer-readable medium of claim 19 , wherein the suitable adaptation comprises adjusting the tutoring system for the particular user based on the determined membership of the particular user.