System and method for performing automated short constructed response evaluation
A metric, such as a cosine similarity coefficient, is used to determine the degree of similarity between two responses. The assumption is that similar responses contain similar words with similar frequency counts (i.e., word lists). Word lists of the responses are used as a vector to assess the similarity between the word list for each student response and the word lists of responses having known scores. An iterative modeling process called semantic content similarity scoring, implemented by a content scoring module, augments differences among the various score-point responses by extending word lists to include, for example, synonyms of key words, key phrases (two or more words), key rubric scoring rule components, likelihood that a response would be given a specific score by a human rater. The word lists are also augmented by removing stop words.
1. An automated method of scoring a constructed response implemented by a computer, said computer comprising a content scoring module, said content scoring module comprising tagging or parsing software and a machine learning engine, said method comprising:
(a) with said tagging or parsing software, computing the frequencies with which entries of a master word list file occur in each of a plurality of training responses, wherein the training responses comprise a plurality of known scores comprising all possible scores achievable for the response;
(b) with said content scoring module, computing one or more coefficients for each of the training responses based on the frequencies computed in step (a);
(c) with said content scoring module, composing a score-specific word list for each possible score achievable on the response by, for each possible score:
(i) removing one or more entries from the word list to create a modified word list;
(ii) computing the coefficients for each response at that possible score based on the modified word list;
(iii) compare the coefficients computed in step (e) for each response at that possible score;
(iv) computing the coefficients for each response at one or more other possible scores based on the modified word list;
(v) compare the coefficients computed in step (iv) with the coefficients computed in step (ii);
(vi) determining, based on the comparison of step (v), whether to keep the entry out of the word list or put the entry back into the word list;
(vii) repeating steps (i) through (vi) one or more times to develop the score-specific word list for that possible score;
(d) with said content scoring module, repeating step (c) rubric scoring rule for each possible score achievable on the response;
(e) with said content scoring module, computing coefficients for each training response based on the score-specific word lists developed in steps (c) and (d), wherein the number of computed coefficients is reduced by collapsing adjacent response sets;
(f) with said content scoring module, computing coefficients for an unscored response based on each of the score-specific word lists;
(g) with said content scoring module, comparing the coefficients computed for the unscored response in step (f) with the coefficients computed for each training response in step (e), wherein said comparing comprises said content scoring module submitting the coefficients computed for the unscored response in step (f) and the coefficients computed for each training response in step (e) to said machine learning engine; and
(h) with said machine learning engine, predicting a score for the unscored response based on the comparison of step (f).
2. The method of claim 1 , wherein step (c)(iv) comprises computing coefficients for each response at the highest score achievable for the response.
3. The method of claim 1 , wherein step (c)(iv) comprises computing coefficients for each response at the next higher score achievable for the response from the score for which the score-specific word list is being developed.
4. The method of claim 1 , wherein step (c)(iv) comprises computing coefficients for each response at the next higher score and the next lower achievable for the response from the score for which the score-specific word list is being developed.
5. The method of claim 1 , wherein step (c)(iv) comprises computing coefficients for each response at the next two higher scores achievable for the response from the score for which the score-specific word list is being developed.
6. The method of claim 1 , further comprising augmenting a word list used in step (a) by adding one or more non-word variables.
7. The method of claim 1 , further comprising augmenting a word list used in step (a) by one or more of:
reducing words to root words;
combining words that are synonyms into a single word;
incorporating one or more phrases of at least two words;
incorporating values based on a rubric for scoring a response; and
incorporating values based on a likelihood that a score of the response based on a rubric is accurate.
8. An automated method of scoring a constructed response implemented by a computer, said computer comprising a content scoring module, said content scoring module comprising a text-reading program and a machine learning engine, said method comprising:
(a) with said computerized text-reading program, computing the frequencies with which entries of a master word list file occur in each of a plurality of training responses, wherein the training responses comprise a plurality of known scores comprising all possible scores achievable for the response;
(b) with said content scoring module, computing one or more coefficients for each of the training responses based on the frequencies computed in step (a);
(c) with said content scoring module, composing a score-specific word list for each possible score achievable on the response by modifying the master word list by adding and/or removing entries from the master word list file to derive a word list that increases the computed discrimination between responses with dissimilar scores or increases the computed similarities between responses with similar scores;
(d) with said content scoring module, computing coefficients for each training response based on the score-specific word lists developed in steps (c), wherein the number of computed coefficients is reduced by collapsing adjacent response sets;
(e) with said content scoring module, computing coefficients for an unscored response based on each of the score-specific word lists;
(f) with said content scoring module, comparing the coefficients computed for the unscored response in step (e) with the coefficients computed for each training response in step (d), wherein said comparing comprises said content scoring module submitting the coefficients computed for the unscored response in step (e) and the coefficients computed for each training response in step (d) to said machine learning engine; and
(g) with said machine learning engine, predicting a score for the unscored response based on the comparison of step (f).
9. The method of claim 1 , wherein said machine learning engine comprises a neural network and one or more genetic algorithms.
10. The method of claim 8 , wherein said machine learning engine comprises a neural network and one or more genetic algorithms.