System and method for automated assessment of constrained constructed responses
View Patent ↗A system and method for automated assessment of constrained constructed responses provides for automatic, e.g., programmatic, assessing of a test subject response to a constrained constructed response item or question. In one embodiment, paper test materials with test subject markings thereon are converted to electronic form. Test subject markings are further isolated, for example, by registration, sizing and removal of markings other than a graphic response. The subject markings are further compared with a baseline response, correctness deviation criteria and incorrectness deviation criteria (e.g., correctness, incorrectness, substantial correctness, substantial incorrectness, and so on) from which one or more of a score, further learning or other assessment results may be determined. Cluster analysis or other post-evaluation or post-assessment processing may also be conducted, or re-evaluation or re-assessment may also be conducted.
1. A computerized system for assessing electronically-provided constrained constructed responses, said system comprising:
a computer with a memory storing constrained constructed response assessment program code executable by said computer to assess electronically-provided constrained constructed responses, said program code including a set of instructions defining a constrained constructed response analyzer, a constrained constructed response assessment engine, and a constrained constructed response action engine;
wherein said constrained constructed response analyzer includes a first subset of instructions executable by said computer to receive electronically-provided constrained constructed responses and to retrieve absolute correctness and deviation assessment criteria against which the electronically-provided constrained constructed responses can be compared;
wherein said constrained constructed response assessment engine includes a second subset of instructions executable by said computer to determine at least one evaluation indicator based upon a comparison by said assessment engine of the electronically-provided constrained constructed responses received by said analyzer and the absolute correctness and deviation assessment criteria retrieved by said analyzer;
wherein said constrained constructed response action engine includes a third subset of instructions executable by said computer to produce an assessment system output based on at least one evaluation indicator determined by said assessment engine; and
wherein said constrained constructed response action engine further includes a subset of instructions executable by said computer to compare evaluation indicators from said constrained constructed response assessment engine with a learning map and to determine from said comparison at least one of learning deficiencies and opportunities for further learning.
2. The system of claim 1 , wherein said evaluation indicator is selected by said constrained constructed response assessment engine from a group of indicators including correctness, substantiality of correctness, substantiality of incorrectness, and incorrectness indicators.
3. The system of claim 2 , wherein said group of indicators further includes an indeterminate indicator.
4. The system of claim 2 wherein said constrained constructed response assessment engine is configured such that an evaluation indicator representing substantial correctness yields partial credit.
5. The system of claim 2 , wherein said constrained constructed response assessment engine is configured such that an evaluation indicator representing substantial incorrectness yields partial credit.
6. The system of claim 2 , wherein said constrained constructed response action engine is configured to determine respondent learning needs based on at least one of substantial correctness and substantial incorrectness of a response to a constrained constructed response item.
7. The system of claim 2 , wherein said constrained constructed response action engine is configured to determine a grade based on at least one of substantial correctness and substantial incorrectness of a response to a constrained constructed response item.
8. The system of claim 1 , wherein said assessment criteria include at least one of registration criteria, identification criteria, sizing criteria, item border criteria, response region criteria, response super-region criteria, response format criteria and base response criteria.
9. The system of claim 8 , wherein said sizing criteria include indicators for reversing a size increase.
10. The system of claim 8 , wherein said response region criteria describes an item response area in which a respondent is charged with providing a constrained constructed response.
11. The system of claim 8 , wherein said response super-region criteria describe an item response area in which a response to a constrained constructed item may be found.
12. The system of claim 11 , wherein said response super-region for a constrained constructed response item is larger than a response area called for by the constrained constructed response item.
13. The system of claim 12 , wherein said response super-region corresponds with allowable errors in responding to the constrained constructed response item.
14. The system of claim 11 , wherein said response super-region criteria applied by said system are variable.
15. The system of claim 11 , wherein said constrained constructed response analyzer engine uses super-region criteria to determine more than one super-region for a single constrained constructed response item.
16. The system of claim 8 , wherein said base response criteria for a constrained constructed response item consists of a single correctness criteria.
17. The system of claim 8 , wherein said base response criteria for a constrained constructed response item includes a plurality of correctness criteria.
18. The computerized system according to claim 1 , wherein the electronically-provided constrained constructed responses include at least one of a graphing item response, a markup item response, a matching item response, and a short answer response.
19. The system of claim 18 , wherein said response format criteria include respondent marking detection facilitating parameters.
20. The system of claim 18 , wherein said assessment criteria include both super-region criteria and response format criteria.
21. The system of claim 18 , wherein only those assessment criteria corresponding to a current constrained constructed response item are transferred to said constrained constructed response analyzer.
22. The system of claim 1 , further comprising at least one constrained constructed response reader.
23. The system of claim 22 , wherein said constrained constructed response reader includes at least one of a scanner, a braille reader, and a digitizing tablet.
24. The system of claim 1 , further comprising a printer.
25. The system of claim 24 , wherein said program code further includes a set of instructions executable by said computer to generate an assessment report using said printer.
26. The system of claim 24 , wherein said program code further includes a set of instructions executable by said computer to generate test materials using said printer.
27. The system of claim 1 , wherein said constrained constructed response analyzer further includes a subset of instructions executable by said computer to retrieve format criteria from memory, compare said format criteria with an electronically-provided constrained constructed response, determine a suitably formatted portion of a constrained constructed response and provide a suitable formatted response to said constrained constructed response assessment engine.
28. The system of claim 27 , wherein said constrained constructed response analyzer further includes a subset of instructions executable by said computer to reformat an electronically-provided constrained constructed response for assessment by said constrained constructed response assessment engine.
29. The system of claim 27 , wherein said constrained constructed response analyzer further includes a subset of instructions executable by said computer to determine at least one of registration and sizing of an electronically-provided constrained constructed response and to provide a corresponding layout indicator to said constrained constructed response assessment engine.
30. The system of claim 1 , wherein said constrained constructed response action engine further includes a subset of instructions executable by said computer to perform cluster analysis of constrained constructed responses from multiple respondents.
31. The system of claim 30 , wherein said cluster analysis is performed by said computer using raw constrained constructed response information from multiple respondents.
32. The system of claim 30 , wherein said cluster analysis includes image processing constrained constructed responses from multiple respondents.
33. The system of claim 32 , wherein said image processing of constrained constructed responses includes finding common latent patterns that are similar among multiple respondents.
34. The system of claim 30 , wherein said cluster analysis is performed using evaluation indicators for multiple respondents.
35. The system of claim 34 , wherein clusters detected on the basis of raw constrained constructed response information are compared with clusters detected on the basis of evaluation indicators.
36. The system of claim 30 , further comprising a set of instructions executable by said computer to modify evaluation criteria based on identification of clusters.
37. The system of claim 36 , further comprising a set of instructions executable by said computer to initiate reassessment of a constrained constructed response by said constrained constructed response assessment engine using said modified evaluation criteria.
38. The system of claim 1 , wherein said action engine saves assessment results in said memory.
39. The system of claim 1 , wherein said computer is at least intermittently communicatingly coupleable with an assessment provider system via a network.
40. The system of claim 39 above, wherein said assessment provider system is configured to deliver constrained constructed response items electronically to remote readers via said network, and wherein said readers are configured to accept constrained constructed responses from respondents and transmit said constrained constructed responses to said system.
41. A computerized method for assessing electronically provided constrained constructed responses, said method comprising the steps of:
using a computer to receive an item response portion corresponding to an assessment item, wherein the item response portion has a response region and the assessment item has an associated constrained constructed response item type;
using a constrained constructed response analyzer of the computer to isolate at least a part of the responsive markings within a super-response region corresponding to the response region;
using a constrained constructed response assessment engine of the computer to compare the responsive markings to absolute correctness and deviation assessment criteria and using the comparison to select an evaluation indicator from a group of indicators including correctness, substantiality of correctness, incorrectness and substantiality of incorrectness of the responsive marking;
using a constrained constructed response action engine to determine at least one of performance assessment and learning assessment based on the evaluation indicator selected;
determining a learning deficiency from the evaluation indicator of at least one item; and
determining a relationship of a learning deficiency to a learning map.
42. The method of claim 41 , further comprising the step of conducting a recursive assessment in which at least one of the evaluation indicators and the assessment parameters are compared.
43. The method of claim 42 , wherein said step of conducting a recursive assessment includes comparing the evaluation indicators of a same assessment subject and determining therefrom clusters of errors of the same type.
44. The method of claim 42 , further comprising the step of associating at least one of the deviation assessment criteria of a subtype thereof a deviation type indicator of one of a definite and an indefinite type.
45. The method of claim 42 , further comprising at least one of the steps of regrading and ignoring an item on the basis of said step of conducting a recursive assessment.
46. The method of claim 43 , further comprising the step of evaluating an item based on the size of clusters.
47. The method of claim 43 , further comprising the steps of comparing cluster sizes and determining optimal forms of selected response items that target sub-populations of subjects.
48. The method of claim 41 , further comprising the step of comparing the evaluation indicators of different assessment subjects and determining therefrom clusters of errors of a same type.
49. The method of claim 48 , further comprising at least one of the steps of regrading and ignoring an item on the basis of said step of conducting a recursive assessment.
50. The method of claim 48 , further comprising the step of evaluating an item based on the size of clusters.
51. The method of claim 48 , further comprising the step of comparing cluster sizes and determining optimal forms of selected response items that target sub-populations of subjects.
52. The method of claim 48 , further comprising the step of modifying evaluation criteria based on said recursive assessment.
53. The method of claim 52 , further comprising the step of reassessing constrained constructed responses using said modified evaluation criteria to generate new evaluation indicators.
54. The method of claim 41 , wherein the set of evaluation indicators that can be derived from said step of comparing the responsive markings to evaluation criteria includes an indicator that the responsive marking does not correspond with the evaluation criteria.
55. The method of claim 54 , further comprising the step of manually reviewing a responsive marking if said comparing step results in a evaluation indicator that the responsive marking does not correspond with the evaluation criteria.
56. The method of claim 41 , wherein constrained constructed response portions that are evaluated in said comparing step as corresponding to correctness criteria are not compared to substantial correctness, substantial incorrectness, or incorrectness criteria.
57. The method of claim 41 , wherein constrained constructed response portions that are evaluated in said comparing step as corresponding to correctness criteria are further compared to substantial correctness, substantial incorrectness, and incorrectness criteria.
58. The method of claim 41 , further comprising the step of assigning individual weights to respective evaluation criteria, and wherein said step of determining evaluation indicators uses the individual weights to determine the correctness, substantial correctness, substantial incorrectness, or incorrectness of the responsive marking.
59. The method of claim 41 , wherein said step of determining evaluation indicators includes taking into account membership of the subject in a subpopulation.