Application of high-dimensional linguistic and semantic feature vectors in automated scoring of examination responses

1. A computer implemented method for evaluating examination responses, the method comprising:

receiving, with an examination evaluation server, an examination response;

tokenizing the examination response into one or more n-word segments;

extracting, from the examination response, a set of construct-relevant features indicating one or more scoring metrics, wherein construct relevant features comprise complexity level, a vocabulary difficulty level, a lexical diversity level, and a semantic similarity score, and wherein extracting the semantic similarity score comprises:

generating one or more semantic vectors for each sentence in the examination response, wherein each semantic vector is an average of the semantic vectors of words comprising the sentence,

generating a set of cosine similarity values, wherein each cosine similarity value is a function of the comparing the one or more semantic vectors for each sentence in the examination response and a set of similarly derived sentence semantic vectors, and

generating the semantic similarity score, as an average of the set of cosine similarity values; and

using the one or more scoring metrics as input for a trained machine-learning based examination scoring engine to score the examination response,

wherein the trained machine-learning based examination scoring engine is trained by:

generating, via a prediction and learning engine, a score estimate of a training examination response,

receiving via a reviewer interface, a user-assigned score for the training examination response,

determining an error between the score estimate by comparing the score prediction to the user-assigned score, and

adjusting the machine-learning based examination scoring engine based upon the determined error between the score prediction and user-assigned score.

2. The computer implemented method of claim 1 , wherein generating the set of scoring rules comprises analyzing a training set of user-assigned scores and a set of response feature vectors.

3. The computer implemented method of claim 1 , further comprising:

deriving a feature vector for each response, wherein the set of feature vectors comprise one or more n-grams, and wherein each n-gram is extracted from the examination response.

4. The computer implemented method of claim 3 , wherein each n-gram comprises a string of words or a string of parts-of-speech.

5. The computer implemented method of claim 3 , wherein deriving the set of feature vectors comprises receiving a randomization parameter and the set of scoring rules.

6. The computer implemented method of claim 3 , further comprising:

applying a stochastic gradient boosting tree (GBT) algorithm to the set of feature vectors.

7. The computer implemented method of claim 6 , wherein applying the GBT algorithm comprises:

dividing the examination responses into a plurality of response groups;

generating a group score estimate for each response group; and

adjusting each group score estimate to minimize a scoring discrepancy between the group score estimate and a user-assigned group score.

8. The computer implemented method of claim 1 , further comprising:

receiving, from the reviewer interface, a user-assigned score corresponding to the examination response;

storing the user-assigned score in a data store together with the score estimate;

receiving, from the data store, a plurality of historic user-assigned scores and a plurality of corresponding score estimates; and

generating a conditional agreement value for each score point by determining a set of agreement rates and distribution spreads using a cross-tab matrix of the historic user-assigned scores crossed with the plurality of corresponding score estimates.

9. The computer implemented method of claim 8 , further comprising generating an equally weighted conditional agreement as a second function of an average of a plurality of conditional agreement values across different score points.

10. A system for evaluating examination responses comprising:

an examination evaluation server;

a reviewer interface; and

and a data store,

wherein the examination evaluation server comprises a processor and a non-transitory computer readable medium with computer executable instructions embedded thereon, the computer executable instructions configured to cause the processor to:

receive, with the examination evaluation server, an examination response;

tokenize the examination response into one or more n-word segments;

extract, from the examination response, a set of construct-relevant features indicating one or more scoring metrics, wherein construct-relevant features comprise complexity level, a vocabulary difficulty level, a lexical diversity level, and a semantic similarity score, and wherein extracting the semantic similarity score is a process to:

generate one or more semantic vectors for each sentence in the examination response, wherein each semantic vector is an average of the semantic vectors of words comprising the sentence,

generate a set of cosine similarity values, wherein each cosine similarity value is a function of the comparing the one or more semantic vectors for each sentence in the examination response and a set of similarly derived sentence semantic vectors, and

generate the semantic similarity score, as an average of the set of cosine similarity values; and

use the one or more scoring metrics as input for a trained machine-learning based examination scoring engine to score the examination response,

wherein the trained machine-learning based examination scoring engine is trained by a process to:

generate, via a prediction and learning engine, a score estimate of a training examination response,

receive via the reviewer interface, a user-assigned score for the training examination response,

determine an error between the score estimate by comparing the score prediction to the user-assigned score, and

adjust the machine-learning based examination scoring engine based upon the determined error between the score prediction and user-assigned score.

11. The system of claim 10 , wherein the computer executable instructions are further configured to:

cause the processor to analyze a training set of user-assigned scores and a set of response feature vectors.

12. The system of claim 11 , wherein the computer readable instructions are further configured to cause the processor to:

test scoring accuracy by comparing predicted scores to a test set of responses and user-assigned score.

13. The system of claim 12 , wherein the computer readable instructions are further configured to cause the processor to:

derive a feature vector for each response, wherein the set off feature vectors comprise one or more n-grams, and wherein each n-gram is extracted from the examination response.

14. The system of claim 13 , wherein each n-gram comprises a string of words or a string of parts-of-speech.

15. The system of claim 13 , wherein the computer readable instructions are further configured to cause the processor to:

receive a randomization parameter and the set of scoring rules.

16. The system of claim 13 , wherein the computer readable instructions are further configured to cause the processor to:

apply a gradient boosting tree (GBT) algorithm to the set of feature vectors.

17. The system of claim 16 , wherein the GBT algorithm causes the processor to:

divide the examination responses into a plurality of response groups; generate a group score estimate for each response group; and

adjust each group score estimate to minimize a scoring discrepancy between the group score estimate and a user-assigned group score.

18. The system of claim 10 , wherein the computer readable instructions are further configured to cause the processor to:

receive, from the reviewer interface, a user-assigned score corresponding to the examination response;

store the user-assigned score in the data store together with the score estimate;

receive, from the data store, a plurality of historic user-assigned scores and a plurality of corresponding score estimates; and

generate a conditional agreement value for each score point by determining a set of agreement rates and distribution spreads using a cross-tab matrix of the historic user-assigned scores crossed with the plurality of corresponding score estimate.

19. The system of claim 18 , wherein the computer readable instructions are further configured to generate an equally weighted condition agreement as a second function of an average of a plurality of conditional agreement values across different score points.