Accessibility of testing within a validity framework
Methods for designing an assessment and providing accessibility features for a test taker profile are disclosed. A test taker profile, including multiple knowledge, skill and other attribute(s) (KSAs), is defined in which each KSA has a specific level. Each KSA is designated as either a focal KSA (an essential part of the targeted proficiency to be measured) or an ancillary KSA (not an essential part). For each focal KSA, a focal value (the value that the test taker must have to possess a high level in the targeted proficiency) is defined. For a particular test design alternative (which may include an accessibility feature), a requirement is determined for each KSA. An accessibility feature is selected which avoids indications of invalidity, including: a mismatch between the targeted proficiency and the expected proficiency; the test taker profile does not satisfy all ancillary requirements; and focal requirements are either excessive or insufficient.
1. A method of examining the validity of an assessment design for a test taker profile, the method comprising:
determining a set of knowledge, skills and other attributes (KSAs) for an assessment design;
for each KSA, defining the KSA to be one of a focal KSA and an ancillary KSA;
defining a focal value for each focal KSA;
defining a test taker profile, wherein the test taker profile includes a profile value for each KSA;
assigning a targeted proficiency value to the test taker profile based on the profile value and the focal value for each focal KSA;
determining, for each KSA, a requirement imposed by the assessment design;
assigning an effective proficiency value to the test taker profile based on the profile value and the requirement for each KSA;
if the targeted proficiency value matches the effective proficiency value:
determining whether an indication of invalidity exists,
if an indication of invalidity exists, assigning a problematic value to a validity variable, and
if an indication of invalidity does not exist, selecting the assessment design as an assessment design alternative for the test taker profile;
if the targeted proficiency value does not match the effective proficiency value, assigning a problematic value to the validity variable; and
if the validity variable has a problematic value, modifying one or more of the assessment design and the test taker profile.
2. The method of claim 1 wherein assigning a targeted proficiency value comprises:
if, for each focal KSA, the profile value is at least as high as the focal value, assigning a high targeted proficiency value to the test taker profile; and
if not, assigning a low targeted proficiency value to the test taker profile.
3. The method of claim 1 wherein assigning an effective proficiency value comprises:
if, for each KSA, the profile value is a least as high as the requirement, assigning a high effective proficiency value to the test taker profile; and
if not, assigning a low effective proficiency value to the test taker profile.
4. The method of claim 1 wherein determining whether an indication of invalidity exists comprises:
determining that an indication of invalidity exists if, for an ancillary KSA, the profile value is less than the requirement.
5. The method of claim 1 wherein determining whether an indication of invalidity exists comprises:
determining that an indication of invalidity exists if, for a focal KSA, the requirement exceeds the focal value.
6. The method of claim 1 wherein determining whether an indication of invalidity exists comprises:
determining that an indication of invalidity exists if, for a focal KSA, the requirement is less than the focal value.
7. The method of claim 1 wherein modifying one or more of the assessment design and the test taker profile comprises one or more of the following:
using, in the assessment design, a changed set of familiarization materials to increase the profile value for at least one ancillary KSA;
using, in the assessment design, a changed set of practice questions to increase the profile value for at least one KSA;
changing, in the assessment design, an ancillary KSA to be a focal KSA;
changing, in the assessment design, a focal KSA to be an ancillary KSA;
changing, in the assessment design, the focal value for a focal KSA;
using, in the assessment design, a changed accessibility feature to perform one or more of decreasing an ancillary requirement, decreasing a focal requirement, increasing a focal requirement;
using, in the assessment design, a changed set of eligibility rules for taking the assessment;
using, in the assessment design, a changed set of eligibility rules for using an accessibility feature;
using, in the assessment design, language having one or more of a lower requirement for an ancillary KSA, a higher requirement for a focal KSA, and a lower requirement for a focal KSA;
using, in the assessment design, questions having one or more of a lower requirement for an ancillary KSA, a higher requirement for a focal KSA, and a lower requirement for a focal KSA;
redefining the test taker profile by changing one or more of the set of KSAs for the test taker profile, a number of values of a KSA that is used in the test taker profile, a definition of values of a KSA that is used in the test taker profile, and a definition of the permissible KSA values;
redefining the test taker profile by realigning one or more values for a KSA to broaden the set of test takers represented at a value for a KSA;
redefining the test taker profile by realigning one or more values for a KSA to narrow the set of test takers represented at a value for a KSA;
using, in the assessment design, a changed set of provisions for score reports; and
using, in the assessment design, changed guidance materials for the interpretation of score reports.
8. The method of claim 1 , further comprising:
if the targeted proficiency value does not match the effective proficiency value, determining whether an indication of invalidity exists.
9. The method of claim 8 wherein determining whether an indication of invalidity exists comprises:
determining that an indication of invalidity exists if, for an ancillary KSA, the profile value is less than the requirement.
10. The method of claim 8 wherein determining whether an indication of invalidity exists comprises:
determining that an indication of invalidity exists if, for a focal KSA, the requirement exceeds the focal value.
11. The method of claim 8 wherein determining whether an indication of invalidity exists comprises:
determining that an indication of invalidity exists if, for a focal KSA, the requirement is less than the focal value.
12. The method of claim 1 wherein the test taker profile comprises one or more profile values characteristic of a language learner.
13. The method of claim 1 wherein the test taker profile comprises one or more profile values characteristic of a test taker having a disability.
14. The method of claim 13 wherein the disability comprises one or more of deafness, blindness, dyslexia, low vision, and neuropathy.