System and method for scoring constructed responses

1. A system for improving scoring consistency for computer-scored constructed responses generated by one or more students in response to a prompt, said system comprising:

a computer with a processor and a memory device storing a set of digital instructions executable by said processor, wherein the digital instructions comprise a parser module, a list-based module, and a manual scoring interface, wherein:

the parser module executable by said processor to:

receive a first plurality of constructed responses in an electronic font-based format from a database,

determine, for each constructed response in said first plurality of constructed responses, whether the constructed response is blank, a refusal to respond, an off-topic phrase, or a copy of the prompt,

assign one of a plurality of parser flags to each of said first plurality of constructed responses based on the determining, and

store the assigned parser flag with a corresponding constructed response in the database;

the list-based module executable by said processor to:

receive a second plurality of constructed responses in the electronic font-based format from the database,

match each of the constructed responses in said second plurality of constructed responses with one of a plurality of pre-scored constructed responses,

assign, in the database, a score to each constructed response in said second plurality of constructed responses based on the matching, and

assign, in the database, a list-based processor flag indicating that each of said constructed responses in said second plurality of constructed responses is scored; and

a manual scoring interface executable by said processor to:

receive, over a communications network, a third plurality of constructed responses that have not been assigned a parser flag or a list-based processor flag for display on an interface, and

upon determining that the third plurality of constructed responses have not been assigned the parser flag or the list-based processor flag, transmit, over the communications network to a hand-scoring system, the third plurality of constructed responses, wherein the hand-scoring system is associated with rubrics or anchor papers, and wherein the hand-scoring system is enabled to determine a set of manual scores corresponding to the third plurality of constructed responses.

2. The system of claim 1 wherein the parser module is further executable by said processor to:

strip spaces, commas, and semi-colons in the constructed response,

search the constructed response for substrings that match a first term in a replacement term list,

replace the searched substrings in the constructed response with the corresponding matching term in the replacement term list, and

determine if the constructed response matches a second term in a refusal term list or has a length of zero.

3. The system of claim 1 wherein the parser module is further executable by said processor:

replace numeric, punctuation, and space symbols in the constructed response with a predetermined character of the constructed response,

add the predetermined character to an end of the constructed response,

divide the constructed response into one or more trigraphs, and

determine whether each of the trigraphs are of one of a plurality of valid forms.

4. The system of claim 1 wherein the parser module is further executable by said processor to:

count a number of exact matches between the constructed response and a word list, and

determine that the counted number is less than a predetermined threshold.

5. The system of claim 4 wherein the parser module is further executable by said processor to:

modify the word list based on a user specification, wherein the modifying comprises reducing words of a pre-specified length by a pre-specified percentage.

6. The system of claim 1 wherein the parser module is further executable by said processor to:

remove white space and vowels from the constructed response,

generate a list of n-graphs of the constructed response,

count a number of generated n-graphs of the constructed response,

determine that the number of counted n-graphs of the constructed response is greater than a first predetermined threshold,

remove white space and vowels from the prompt,

generate a list of n-graphs of the prompt,

count a number of matches between the generated constructed response n-graphs and the prompt n-graphs,

count a number of non-matches between the generated constructed response n-graphs and the prompt n-graphs, and

determine that a proportion of the counted matches exceeds a second predetermined threshold.

7. The system of claim 1 , wherein the parser module is further executable by said processor to:

separate the constructed response into a plurality of parts based on one or more items characteristics,

pre-process the plurality of parts according to a user specification,

recombine the pre-processed plurality of parts into a pre-processed student response, and

determine that the pre-processed student response matched one of the plurality of pre-scored constructed responses.

8. The system of claim 7 , wherein the parser module is further executable by said processor to:

replace terms in the pre-processed student response with pre-processed terms from a replacement list.

9. The system of claim 1 , wherein the digital instructions are executable by said processor to further:

generate a report of score distributions, and

display the report of score distributions on a graphical user interface.

10. A computer-implemented method for improving scoring consistency for computer-scored constructed responses generated by one or more students in response to a prompt, the method comprising:

receiving a first plurality of constructed responses in an electronic font-based format from a database;

determining for each constructed response in said first plurality of constructed responses, whether the constructed response is blank, a refusal to respond, a non-English phrase, an off-topic phrase, or a copy of the prompt;

assigning one of a plurality of parser flags to each of said first plurality of constructed responses based on the determining;

storing the assigned parser flag with a corresponding constructed response in the database;

receiving a second plurality of constructed response in the electronic font-based format from the database;

matching each of the constructed responses in said second plurality of constructed responses with one of a plurality of pre-scored constructed responses;

assigning, in the database a score to each constructed response in said second plurality of constructed responses based on the matching;

assigning, in the database, a list-based processor flag indicating that each of said constructed responses in said second plurality of constructed responses in scored;

receiving, over a communications network, a third plurality of constructed responses that have not been assigned a parser flag or a list-based processor flag for display on an interface; and

upon determining that the third plurality of constructed responses have not been assigned the parser flag or the list-based processor flag, transmitting, over the communications network to a hand-scoring system, the third plurality of constructed responses, wherein the hand-scoring system is associated with rubrics or anchor papers, and wherein the hand-scoring system is enabled to determine a set of manual scores corresponding to the third plurality of constructed responses.

11. The method of claim 10 wherein determining for each constructed response in said first plurality of constructed responses, whether the constructed response is the refusal to respond further comprises:

stripping spaces, commas, and semi-colons in the constructed response;

searching the constructed response for substrings that match a first term in a replacement term list;

replacing the searched substrings in the constructed response with the corresponding matching term in the replacement term list; and

determining if the constructed response matches a second term in a refusal term list or has a length of zero.

12. The method of claim 10 wherein determining, for each constructed response in said first plurality of constructed responses, whether the constructed response is a non-English phrase further comprises:

replacing numeric, punctuations, and space symbols in the constructed response with a predetermined character;

adding the predetermined character to an end of the constructed response;

dividing the constructed response into one or more trigraphs; and determining whether each of the trigraphs are one of a plurality of valid forms.

13. The method of claim 10 wherein determining, for each constructed response in said first plurality of constructed responses, whether the constructed response is the off-topic phrase further comprises:

counting a number of exact matches between the constructed response and a word list; and

determining that the counted number is less than a predetermined threshold.

14. The method of claim 13 , further comprising:

modifying the word list based on a user specification, wherein said modifying comprises reducing words of a pre-specified length by a pre-specified percentage.

15. The method of claim 10 wherein determining, for each constructed response is said first plurality of constructed responses, whether the constructed response is the copy of the prompt further comprises:

removing white space and vowels from the constructed response;

generating a list of n-graphs of the constructed response;

counting a number of generated n-graphs of the constructed response;

determining that the number of counted n-graphs of the constructed response is greater than a first predetermined threshold;

removing white space and vowels from the prompt;

generating a list of n-graphs of the prompt;

counting a number of matches between the generated constructed response n-graphs and the prompt n-graphs; and

determining that a proportion of the counted matches exceeds a second predetermined threshold.

16. The method of claim 10 , wherein matching each of the constructed responses in said second plurality of constructed responses with one of a plurality of pre-scored constructed responses further comprises:

separating the constructed response into a plurality of parts based on one or more item characteristics;

pre-processing the plurality of parts according to a user specification;

recombining the pre-processed plurality of parts into a pre-processed student response; and

determining that the pre-processed student responses matches one of the plurality of pre-scored constructed responses.

17. The method of claim 16 , further comprising:

replacing terms in the pre-processed student response with pre-processed terms from a replacement list.

18. The method of claim 10 , further comprising:

generating a report of score distributions.

19. A computer-program product for improving scoring consistency for computer-scored constructed responses generated by one or more students in response to a prompt comprising computer program code, when run on a computer coupled to a processor and a non-transitory memory, causes the processor to perform the steps of:

receiving a first plurality of constructed responses in an electronic font-based format from a database;

determining, for each constructed response in said first plurality of constructed responses, whether the constructed response is: blank, a refusal to respond, a non-English phrase, an off-topic phrase, or a copy of the prompt;

assigning one of a plurality of parser flags to each of said first plurality of constructed responses based on the determining;

storing the assigned parser flag with a corresponding constructed response in the database;

receiving a second plurality of constructed responses in the electronic font-based format from the database;

matching each of the constructed responses in said second plurality of constructed responses with one of a plurality of pre-scored constructed responses;

assigning, in the database, a core to each constructed response in said second plurality of constructed responses based on the matching;

assigning, in the database, a list-based processor flag indicating that each of said constructed responses in said second plurality of constructed responses is scored;

receiving, over a communications network, a third plurality of constructed responses that have not been assigned a parser flag or a list-based processor flag for display on an interface; and

upon determining that the third plurality of constructed responses have not been assigned the parser flag or the list-based processor flag, transmitting, over the communications network to a hand-scoring system, the third plurality of constructed responses, wherein the hand-scoring system is associated with rubrics or anchor papers, and wherein the hand-scoring system is enabled to determine a set of manual scores corresponding to the third plurality of constructed responses.

20. The computer-program product of claim 19 , wherein said step of matching each of the constructed responses in said second plurality of constructed responses with one of a plurality of pre-scored constructed responses further comprises:

separating the constructed response into a plurality of parts based on one or more item characteristics;

pre-processing the plurality of parts according to a user specification;

recombining the pre-processed plurality of parts into a pre-processed student response; and

determining that the pre-processed student response matches one of the plurality of pre-scored constructed responses.