Methods for improving certainty of test-taker performance determinations for assessments with open-ended items
The current invention is designed to improve the certainty of test-taker classifications in testing programs utilizing open-ended items. The current invention discloses methods to improve certainty in estimates of test-taker performance for all test-takers while using a minimum of scoring resources. The invention selectively allocates resources to scoring by ceasing to allocate scoring resources to test-takers whose performance has been determined with sufficient certainty, and by having an additional reader or readers score responses from test-takers whose scores are close to cutoff levels. The invention further increases precision of scores for test-takers near a cutoff by selectively allocating more reliable and accurate readers to score responses most likely to influence a test-taker's performance classification. Other scoring resources are also shown.
1. A method of allocating resources in conjunction with scoring a test-taker's response to an open-ended item of an assessment including a plurality of items that contribute to a specific performance level classification, wherein the method is implemented using a computer system programmed to perform steps comprising:
a. collecting test-taker information and adding the collected information to a database of the computer system;
b. estimating a test-taker performance classification for a portion of the assessment comprising the plurality of items based on the collected test-taker information, wherein estimating the performance classification comprises classifying the test-taker into one of two or more performance levels;
c. estimating the certainty of the estimated test-taker performance classification; and
d. allocating a scoring resource to score the test-taker's response to the open-ended item as a function of the estimated certainty that the test-taker falls within one performance level and not a different performance level.
2. The method of claim 1 , wherein the test-taker information includes at least one score on at least one multiple choice item of the plurality of items.
3. The method of claim 1 , wherein the test-taker information includes at least one score on at least one additional open-ended item of the plurality of items.
4. The method of claim 1 , wherein the test-taker information includes at least one score on the open ended item.
5. The method of claim 1 , wherein the test-taker information includes at least one score from at least one other assessment.
6. The method of claim 1 , wherein the test-taker information includes at least one teacher rating.
7. The method of claim 1 , wherein the estimated performance classification corresponds to an estimated score on the plurality of items.
8. The method of claim 1 , wherein the estimated performance classification corresponds to an estimated performance level on the plurality of items.
9. The method of claim 1 , wherein the estimated performance classification corresponds to an estimated score on the open-ended item.
10. The method of claim 9 , further comprising:
a. determining reader accuracy based on estimated test taker performance and a score assigned by a reader; and
b. allocating scoring resources based on that determination.
11. The method of claim 1 , further comprising assigning a first score to the open ended item.
12. The method of claim 11 , further comprising assigning, by the scoring resource, an additional score to the open ended item.
13. The method of claim 1 , further comprising determining that a stopping criteria has been reached.
14. The method of claim 13 , wherein the stopping criteria corresponds to applying all available scoring resources.
15. The method of claim 13 , wherein the stopping criteria corresponds to applying all allocated scoring resources.
16. The method of claim 13 , wherein the stopping criteria corresponds to a lack of additional scoring resources that would substantially improve a certainty of the estimated test-taker performance classification.
17. The method of claim 13 , wherein the stopping criteria corresponds to a determination that the estimated test-taker performance classification is determined with sufficient certainty.
18. The method of claim 1 , wherein the estimate of the precision corresponds to reader characteristics of at least one of an individual reader and a set of readers.
19. The method of claim 18 , wherein the reader characteristics include reader accuracy.
20. The method of claim 18 , wherein the reader characteristics include a distribution of item scores assigned by a reader.
21. The method of claim 18 , wherein the reader characteristics include reader agreement statistics.
22. The method of claim 1 , wherein step d. is initially performed before two readers have scored the test-taker's response to the open-ended item.
23. The method of claim 1 , wherein step d. is initially performed after only a single reader has scored the test-taker's response to the open-ended question.
24. The method of claim 23 , wherein no further scoring resources are allocated to score the test-taker's response to the open-ended question if the test-taker's performance classification is determined to be estimated with sufficient certainty after only the single reader has scored the test-taker's response to the open-ended question.
25. A method of allocating scoring resources in conjunction with scoring a test-taker's response to an open-ended item of an assessment, wherein the method is implemented using a computer system programmed to perform steps comprising:
(a) collecting test-taker information and adding the collected information to a database of the computer system;
(b) assigning the open-ended item of the assessment to only a first reader to assign a score to the open-ended item of assessment;
(c) storing the score assigned by the first reader to the database as part of the collected information;
(d) estimating a performance classification for the test-taker using collected information including the score of the open-ended item of the assessment assigned by the first reader;
(e) estimating the certainty of the estimated performance classification; and
(f) only if it is determined that the test-taker's estimated performance classification has not been estimated with sufficient certainty, assigning the open-ended question to a second reader.
26. A method of allocating scoring resources in conjunction with determining a test-taker's performance level classification, wherein the method is implemented using a computer system programmed to perform steps comprising:
(a) collecting test-taker information including responses to each of a plurality of open-ended items of the assessment that contribute to a specific performance level classification and adding the collected information to a database of the computer system;
(b) selecting an un-scored open-ended item response of the plurality of open-ended item responses;
(c) assigning the selected open-ended item response to a reader to assign a score to the selected open-ended item response;
(d) storing the reader's score of the selected open-ended item response in the database as part of the collected information;
(e) repeating steps (b) through (d) for each un-scored open-ended item response of the plurality of open-ended item responses;
(f) estimating the test-taker's performance level classification based on the collected test-taker information, wherein the estimating the test-taker's performance level classification comprises classifying the test-taker into one of two or more performance levels;
(g) estimating the certainty of the estimated test-taker performance level classification; and
(h) if the test-taker's performance level classification has been estimated to fall within one performance level and not a different performance level with sufficient certainty, stopping the scoring of the plurality of open-ended item responses; and
(i) if the test-taker's performance level classification has been estimated to fall within one performance level and not a different performance level with insufficient certainty:
(1) choosing a response from the plurality of open-ended item responses;
(2) assigning the chosen open-ended item response to a reader that has not already scored the chosen open-ended item response to assign an additional score to the item;
(3) storing the additional score of the chosen open-ended item response in the database as part of the collected information; and
(4) repeating steps (f) through (i).
27. The method of claim 26 , wherein the choosing step of step (i)(1) comprises choosing one or more item responses deemed likely to achieve sufficient certainty in the estimation of the test-taker's performance level classification, thereby deferring the allocation of additional scoring resources from other item responses.