Methods for improving certainty of test-taker performance determinations for assessments with open-ended items
The current invention is designed to improve the certainty of test-taker classifications in testing programs utilizing open-ended items. The current invention discloses methods to improve certainty in estimates of test-taker performance for all test-takers while using a minimum of scoring resources. The invention selectively allocates resources to scoring by ceasing to allocate scoring resources to test-takers whose performance has been determined with sufficient certainty, and by having an additional reader or readers score responses from test-takers whose scores are close to cutoff levels. The invention further increases precision of scores for test-takers near a cutoff by selectively allocating more reliable and accurate readers to score responses most likely to influence a test-taker's performance classification. Other scoring resources are also shown.
1. A method of allocating resources to score a test-taker's response to an open-ended item of an assessment comprising:
a. estimating the test-taker's performance on the assessment, wherein estimating the test-taker's performance on the assessment comprises:
1. estimating the test-taker's overall score for all items of the assessment;
2. utilizing one or more cutoff scores which separate the set of all possible overall scores into discrete non-overlapping ranges;
3. treating each of said ranges of scores as an indicator of a performance level on the assessment;
4. assigning to the test-taker the performance level based on the range of scores which includes the test-taker's estimated overall score; and
5. using the assigned performance level as the estimate of the test-taker's performance on the assessment;
b. establishing a specified level of certainty required for a final determination of the test-taker's performance;
c. estimating a level of certainty for the estimated performance of the test-taker;
d. allocating an additional scoring resource to the test-taker's response to at least one open-ended item when the estimated level of certainty fails to meet the specified level of certainty and
e. accepting the assigned performance level as the final determination of the test-taker's performance when the estimated level of certainty meets the specified level of certainty.
2. The method of claim 1 , further comprising repeating steps (a) through (d) until the estimated level of certainty meets the specified level of certainty.
3. The method of claim 1 , wherein the estimated level of certainty is deemed to fail to meet the specified level of certainty if the test-taker's estimated assessment score is within a predefined range of scores around a cutoff score.
4. The method of claim 1 , wherein estimating the level of certainty for the estimated performance of the test-taker comprises:
a. estimating a precision of the estimate of the test-taker's assessment score; and
b. estimating the certainty of the estimate of the test-taker's performance as a function of the estimated precision and the difference between the estimate of the test-taker's assessment score and the cutoff score that is closest to the test-taker's estimated assessment score.
5. The method of claim 1 , wherein allocating a scoring resource comprises:
a. presenting the test-taker's response to the open-ended item to a reader; and
b. recording a score assigned by the reader to the test-taker's response to the open-ended item.
6. The method of claim 5 , wherein allocating a scoring resource further comprises:
a. presenting the test-taker's response to the open-ended item to one or more additional readers; and
b. recording a score assigned by the one or more additional readers to the test-taker's response to the open-ended item.
7. The method of claim 6 , wherein allocating a scoring resource further comprises:
a. identifying unresolved discrepant scores selected for the open-ended item response by the reader and the one or more additional readers; and
b. presenting the open-ended item response for which there are unresolved discrepant scores to a resolving reader in order to resolve the discrepancy.
8. The method of claim 7 , wherein allocating a scoring resource further comprises:
a. determining characteristics of a set of available readers;
b. determining specific characteristics of a reader necessary to improve the estimated level of certainty for the estimated performance of the test-taker associated with the open-ended item response for which there are unresolved discrepant scores; and
c. selecting as the resolving reader a reader satisfying one or more of the specific characteristics to score the open-ended item response for which there are unresolved discrepant scores.
b. recording a score assigned by the one or more additional readers to the test-taker's response to the open-ended item.
9. The method of claim 5 , wherein selection of a reader for presentation of a particular open-ended item is based on scoring characteristics of the reader.
10. The method of claim 8 , wherein said scoring characteristics include one or more characteristics selected from the group comprising: (a) scoring accuracy, (b) scoring patterns, (c) the distribution of the quality of open-ended item responses assigned to a reader, and (d) the extent to which a reader's scores have been reviewed by a lead reader.
11. The method of claim 10 , wherein scoring accuracy is determined by one or more of monitoring items, lead reader review, and inter-reader agreement.
12. The method of claim 9 , wherein selection of a reader further comprises:
a. determining scoring characteristics of a set of available readers;
b. determining specific scoring characteristics of a reader necessary to improve the estimated level of certainty for the estimated performance of the test-taker; and
c. selecting a reader satisfying one or more of the specific scoring characteristics determined in step (b) to score the test-taker's response to the open-ended item.
13. The method of claim 5 , wherein allocating a scoring resource further comprises:
a. estimating an overall distribution of the quality of open-ended item responses to be scored;
b. estimating a distribution of the quality of open-ended item responses presented to each reader to whom a specific open-ended item response may be assigned for scoring;
c. estimating the quality of the specific open-ended item response to be assigned to a reader; and
d. assigning the specific open-ended item response for scoring by a particular reader in order to lessen a difference between the estimated overall distribution of the quality of open-ended item responses to be scored and the estimated distribution of the quality of open-ended item responses presented to the particular reader.
14. The method of claim 5 , wherein allocating a scoring resource further comprises:
a. identifying one or more scores having a precision to which the failure of the estimated level of certainty to meet the specified level of certainty may be attributed;
b. presenting to a lead reader the one or more open-ended item responses identified in step (a); and
c. recording a score selected by the lead reader for the one or more open-ended item responses presented to the lead reader.
15. The method of claim 14 , wherein identifying the one or more scores comprises considering, as a characteristic of a previous reader, the extent to which scores selected by the previous reader have been reviewed by a lead reader.
16. The method of claim 1 , wherein allocating a scoring resource comprises:
a. determining, for at least one of the test-taker's open-ended item responses, possible scores that could result from presenting that open-ended item response for scoring to a reader;
b. estimating the probability of each possible score that could result by presenting that open-ended item response to the reader;
c. determining the effect that each possible score on that open-ended item response would have on the estimate of the test-taker's performance; and d. presenting the test-taker's response to the open-ended item to the reader if the probability of a possible score that would affect the estimated performance is within a pre-defined threshold.
17. The method of claim 1 , wherein said assessment includes multiple open-ended items, and allocating a scoring resource comprises:
a. estimating an increase in the level of certainty for the estimated performance of the test-taker that would be realized from scoring each of the open-ended item responses; and
b. allocating the scoring resource to the open-ended item response estimated to yield the largest increase in the level of certainty for the estimated performance of the test-taker.
18. The method of claim 1 , wherein said assessment includes multiple-choice items, and estimating the test-taker's performance on the assessment comprises determining the test-taker's score on the multiple-choice items.
19. The method of claim 1 , wherein estimating the test-taker's performance on the assessment is based on one or more items of information selected from the items of information comprising: (a) previously recorded scores to open-ended assessment items; (b) teacher ratings of the test-taker; and the (c) performance exhibited by the test-taker on one or more prior assessments.
20. The method of claim 1 , wherein estimating the level of certainty comprises:
a. determining, for at least one open-ended item response, possible scores that could result from the allocation of one or more scoring resources to that open-ended item response;
b. estimating the probability of each possible score that could result on the open-ended item response; and
c. determining the effect that each possible score on that open-ended item response would have on the test-taker's estimated performance level.
21. The method of claim 20 , further comprising allocating a scoring resource to an open-ended item response if the probability of a possible score that would affect the estimated performance level is within a pre-defined threshold.
22. The method of claim 20 , wherein allocating a scoring resource comprises:
a. presenting the test-taker's response to the open-ended item to a reader; and
b. recording a score assigned by the reader to the test-taker's response to the open-ended item, wherein selection of the reader for presentation of a particular open-ended item is based on scoring characteristics of the reader and the probability of a score that would affect the test-taker's estimated performance.
23. The method of claim 1 , wherein said scoring resource comprises allocating computational resources to at least one of estimating the test-taker's performance on the assessment and estimating the level of certainty for the estimated performance of the test-taker.
24. The method of claim 23 , wherein said computational resource for estimating the test-taker's performance on the assessment comprises:
a. calculating the test-taker's total score including scores achieved by the test-taker on the open-ended items;
b. estimating the standard error of the scores achieved by the test-taker on the open-ended items; and
c. estimating the likely range of the test-taker's total score based on the standard error and the total score determined in step (a).
25. The method of claim 23 , wherein said computational resource for estimating the level of certainty for the estimated performance comprises:
a. calculating the test-taker's total score including scores achieved by the test-taker on the open-ended items;
b. estimating the standard error of the scores achieved by the test-taker on the open-ended items;
c. estimating the likely range of the test-taker's total score based on the standard error and the total score determined in step (a); and
d. determining if the test-taker's total score determined in step (a) is near a cutoff score separating one performance level from a next performance level relative to the likely range of the test-taker's total score.
26. A method of allocating resources to score a test-taker's response to an open-ended item of an assessment including a plurality of open-ended items and multiple choice items comprising:
a. scoring the test-taker's responses to the multiple choice items;
b. scoring the test-taker's responses to the open-ended items a first time;
c. estimating a performance level of the test-taker based on the test-taker's score on the multiple choice items and the test-taker's first time score on the open-ended items;
d. determining whether an adiustment to the first time score of one or more of the open-ended item responses could chanae the estimated performance level of the test taker;
e. allocating scoring resources to only those of the test-taker's responses to open-ended items, which if the first time score thereof were adjusted, could change the test-taker's estimated performance level; and
f. accepting the estimated performance level for the test-taker determined in the estimating step as a final performance level if it is determined in the determining step that adiusting the first time score of one or more of the open-ended item responses would not change the estimated performance level.
27. The method of claim 26 , wherein estimating the performance level of the test-taker comprises:
a. utilizing one or more cutoff scores which separate the set of all possible scores into discrete non-overlapping ranges;
b. treating each of said ranges of scores as an indicator of a performance level on the assessment;
c. assigning to the test-taker the performance level based on the range of scores which includes the test-taker's overall score as determined by the test-taker's score on the multiple choice items and the test-taker's first time score on the open-ended items; and
d. using the assigned performance level as the estimate of the test-taker's performance on the assessment.
28. The method of claim 26 , wherein allocating scoring resources comprises:
a. presenting the test-taker's response to the open-ended item to a reader; and
b. recording a score assigned by the reader to the test-taker's response to the open-ended item.
29. The method of claim 28 , wherein selection of a reader for presentation of a particular open-ended item is based on scoring characteristics of the reader.
30. The method of claim 29 , wherein selection of a reader further comprises:
a. determining scoring characteristics of a set of available readers;
b. determining specific scoring characteristics of a reader necessary to improve the precision of the score assigned to the open-ended item response; and
c. selecting a reader satisfying one or more of the specific scoring characteristics determined in step (b) to score the test-taker's response to the open-ended item.
31. The method of claim 28 , wherein allocating a scoring resource further comprises:
a. identifying unresolved discrepancies between scores selected for the open-ended item response by the reader and the first time score of the open-ended item response; and
b. presenting the open-ended item response for which there are unresolved discrepant scores to a resolving reader in order to resolve the discrepancy.
32. The method of claim 26 , wherein said scoring resources comprise allocating computational resources to estimating the performance level of the test-taker on the assessment.
33. The method of claim 32 , wherein said computational resource for estimating the performance level of the test-taker on the assessment comprises:
a. calculating the test-taker's total score including first time scores achieved by the test-taker on the open-ended items;
b. estimating the standard error of the first time scores achieved by the test-taker on the open-ended items; and
c. estimating the likely range of the test-taker's total score based on the standard error and the total score determined in step (a).
34. The method of claim 26 , wherein allocating scoring resources comprises:
a. determining, for at least one of the test-taker's open-ended item responses, possible scores that could result from presenting that open-ended item response for scoring to a reader;
b. estimating the probability of each possible score that could result by presenting that open-ended item response to the reader;
c. determining the effect that each possible score on that open-ended item response would have on the estimate of the test-taker's performance; and
d. presenting the test-taker's response to the open-ended item to the reader if the probability of a possible score that would affect the estimated performance is within a pre-defined threshold.
35. The method of claim 26 , further comprising:
a. calculating the test-taker's total score from the score achieved by the test-taker on the multiple choice items and the first time score achieved by the test-taker on the open-ended items;
b. estimating the standard error of the first time scores achieved by the test-taker on the open-ended items;
c. estimating the likely range of the test-taker's total score based on the standard error and the total score determined in step (a); and
d. allocating scoring resources if the test-taker's total score determined in step (a) is near a cutoff score separating one performance level from a next performance level relative to the likely range of the test-taker's total score.