AUTOMATED GRADING FOR INTERACTIVE LEARNING APPLICATIONS

1 . An automated grading method for student annotations in an interactive learning application, the method comprising:

(a) distributing an interactive educational resource over a network to a plurality of student devices;

(b) receiving, at a server, an initial set of annotations generated at the student devices in response to the educational resource;

(c) receiving, at the server, a plurality of grades for each of the annotations in the initial set of annotations, each of the grades being provided by a different human grader;

(d) averaging the plurality of grades to produce an average grade for each of the annotations in the initial set of annotations, at least a portion of the initial set of annotations constituting a training set;

(e) extracting portions of annotations within the training set, thereby producing a plurality of seed features;

(f) computationally deriving, from the seed features, one or more grading features predictive of the average grades associated with the training set; and

(g) assigning a grade to a new annotation based on the one or more grading features.

2 . The method of claim 1 , wherein step (g) comprises using a machine-learning model to predict the grade assigned to the new annotation based on the one or more grading features, the model being predictive in accordance with a prediction algorithm and generated by steps comprising:

dividing the initial set of annotations into the training set and a testing set, each of the training set and testing set comprising a plurality of annotations and average grades associated therewith; and

identifying the one or more grading features based on predictive reliability in accordance with the prediction algorithm.

3 . The method of claim 2 , further comprising the steps of:

computationally predicting, based on the one or more grading features, grades for one or more annotations within the testing set; and

adjusting parameters of the model prior to assigning the grade to the new annotation.

4 . The method of claim 2 , wherein the prediction algorithm is a classification tree.

5 . The method of claim 4 , wherein the prediction algorithm is a random forest comprising a plurality of regression trees.

6 . The method of claim 1 , further comprising, before step (c), distributing the initial set of annotations over the network to a plurality of human graders.

7 . The method of claim 1 , further comprising controlling access to the educational resource by the student device at which the new annotation was generated based at least in part on the grade assigned to the new annotation.

8 . The method of claim 1 , further comprising displaying the grade assigned to the new annotation on the student device at which the new annotation was generated.

9 . The method of claim 1 , wherein step (e) comprises applying natural-language processing to annotations within the training set.

10 . An educational system comprising:

a plurality of student devices for executing an interactive educational resource received over a network, the student devices being configured to receive student annotations of the educational resource and transmit the annotations to a server; and

a server in electronic communication with the student devices, the server comprising:

a communication module configured to (i) receive annotations from the student devices, and (ii) receive grades associated with annotations from a plurality of human graders, and

an analysis module configured to (i) associate an average of a plurality of grades received from different human graders with each of an initial set of annotations, at least a portion of the initial set of annotations constituting a training set, (ii) computationally derive one or more grading features predictive of the average grades associated with the training set, and (iii) assign grades to ungraded annotations based on the one or more grading features.

11 . The system of claim 10 , wherein the analysis module is configured to extract portions of annotations within the training set, thereby producing a plurality of seed features, wherein the one or more grading features are computationally derived from the seed features.

12 . The system of claim 10 , wherein the analysis module uses a machine-learning model to predict the grades assigned to the ungraded annotations based on the one or more grading features, the model being predictive in accordance with a prediction algorithm and generated by steps comprising:

dividing the initial set of annotations into the training set and a testing set, each of the training set and testing set comprising a plurality of annotations and average grades associated therewith; and

identifying the one or more grading features based on predictive reliability in accordance with the prediction algorithm.

13 . The system of claim 12 , wherein the analysis module is configured to:

computationally predict, based on the one or more grading features, grades for one or more annotations within the testing set; and

adjust parameters of the model based on the predictions prior to assigning grades to ungraded annotations.

14 . The system of claim 12 , wherein the prediction algorithm is a classification tree.

15 . The system of claim 14 , wherein the prediction algorithm is a random forest comprising a plurality of regression trees.

16 . The system of claim 10 , wherein at least one of the student devices comprises a computer or a handheld device.

17 . The system of claim 10 , wherein the communication module is configured to transmit grades assigned by the analysis module to the student devices.

18 . The system of claim 10 , further comprising a plurality of grading devices configured to display student annotations of the educational resource, receive grades associated with the student annotations from a human grader, and transmit the grades to the server.

19 . The system of claim 18 , wherein at least one of the grading devices comprises a computer or a handheld device.