Method and system for creating, administering and automating scoring of dimensional modeling constructed response items
The methodology of the present invention is implemented by a system which defines concepts about which a respondent is required to demonstrate knowledge of a relationship. The concepts are represented by respondent-manipulable devices that are uniquely associated with the concepts, and the respondent is able to manipulate attributes of the devices to create an arrangement of the devices representative of the respondent's understanding of the relationship of the concepts with which the devices are associated. The attributes of the manipulated devices are recorded and converted into a response pattern based on the cognitive relationships of the devices as manipulated by the respondent. One or more predefined rubric patterns are associated with the constructed response item and define possible cognitive patterns of the concepts as represented by the devices. The response pattern is compared to the rubric pattern to identify matches between the response pattern and the rubric pattern. Implications, or outcomes, are associated with each rubric pattern defined, and a scoring result is created based on the implications of the rubric patterns that match the response patterns.
1. A computer-readable storage media having stored thereon computer-executable instructions for causing a computer to perform, either alone or in conjunction with one or more users, a method for administering a constructed response assessment item to a respondent, wherein the computer performs steps comprising:
presenting a stimulus response stem requiring the respondent to demonstrate knowledge of a relationship between two or more concepts;
providing the respondent at least one device uniquely associated with each of the two or more concepts, at least one device having at least one attribute that can be selectively manipulated by the respondent to change a relationship of that device with respect to at least one other device to construct an arrangement of devices representing a relationship between the concepts with which the devices in the arrangement are associated;
storing a final value of the attributes of each of the devices in the arrangement constructed by the respondent;
converting the stored attributes into a conceptual pattern associated with the arrangement constructed by the respondent by comparing the final values of the attributes of the devices of the arrangement so that the conceptual pattern represents at least one relationship between the devices in the arrangement as determined by a comparison of the final values of the attributes of the devices and not by the final values of the attributes of the devices; and
comparing the conceptual pattern of the arrangement constructed by the respondent with a conceptual pattern of a pre-defined scoring rubric representing a pre-defined relationship between devices to determine whether or not the conceptual pattern associated with the arrangement constructed by the respondent matches the conceptual pattern of the scoring rubri;
wherein the scoring rubric is a set of relationships between the two or more concepts and is independent of the devices associated with each concept.
2. The media of claim 1 , wherein said pre-defined rubric includes a conceptual pattern corresponding to a correct relationship between the two or more concepts.
3. The media of claim 1 , wherein the devices comprise virtual items displayed on the computer that the respondent can electronically manipulate and/or position with respect to each other to create a virtual arrangement of the devices demonstrating a relationship between the concepts with which the devices are associated.
4. The media of claim 1 , wherein the attribute of each device that can be manipulated includes one or more of size, length, height, spatial, orientation, color, direction of movement, speed of movement, sound, sound volume, pattern, shape, and time.
5. The media of claim 1 , wherein said computer further associates an implication to the conceptual pattern of the arrangement constructed by the respondent based on said comparing step, wherein the implication corresponds to a pattern of the pre-defined rubric and wherein each implication comprises information relating to an outcome that occurs as the result of a pattern being found in the conceptual pattern of the arrangement constructed by the respondent.
6. The media of claim 1 , wherein the conceptual pattern of the pre-defined scoring rubric is reusable across multiple testing systems and with devices other than the at least one device and the at least one other device.
7. The media of claim 1 , wherein the rubric is written in a formalized language that eliminates the need for an item writer to interact directly with a programming environment or understand the mathematical construction of a solution set.
8. The media of claim 1 , wherein each relationship represented by said conceptual pattern consists of one or more predefined cognitive, physical relationship between concepts that are defined using a formal language.
9. The media of claim 8 , wherein said formal language comprises a mathematical expression of a spatial relationship between the concepts.
10. A computer-readable storage media having stored thereon computer-executable instructions for causing a computer to perform, either alone or in conjunction with one or more users, a method of creating a constructed response assessment item, wherein the computer performs steps comprising:
presenting a stimulus response stem requiring a respondent to demonstrate knowledge of a relationship between two or more concepts;
providing at least one device uniquely associated with each of the two or more concepts, at least one device having at least one attribute that can be selectively manipulated by the respondent to change a relationship of that device with respect to at least one other device to construct an arrangement of devices representing a relationship between the concepts with which the devices in the arrangement are associated;
converting final values of the attributes of each of the devices in the arrangement constructed by the respondent into a conceptual pattern by comparing the final values of the attributes of the devices of the arrangement so that the conceptual pattern represents at least one relationship between the devices in the arrangement as determined by a comparison of the final values of the attributes of the devices and not by the final values of the attributes of the devices; and
applying a pre-defined scoring rubric for the constructed response including a conceptual pattern representing a pre-defined relationship between devices and corresponding to a relationship between the two or more concepts to compare a conceptual pattern of the arrangement constructed by the respondent with the conceptual pattern of the scoring rubric;
wherein the scoring rubric is a set of relationships between the two or more concepts and is independent of the devices associated with each concept.
11. The media of claim 10 , wherein said rubric includes a conceptual pattern corresponding to a correct relationship between the two or more concepts.
12. The media of claim 10 , wherein the devices comprise virtual items displayed on the computer that the respondent can electronically manipulate and/or position with respect to each other to create a virtual arrangement of the devices demonstrating a relationship between the concepts with which the devices are associated.
13. The media of claim 10 , wherein the attribute of each device that can be manipulated includes one or more of size, length, height, spatial, orientation, color, direction of movement, speed of movement, sound, sound volume, pattern, shape, and time.
14. A system for administering a constructed response assessment item to a respondent, said system comprising:
electronic storage media;
an output device;
a user input device; and
a computer operatively connected with said storage media, said output device, and said user input device and configured to:
retrieve a stimulus response stem from the storage media, wherein the stimulus response stem requires the respondent to demonstrate knowledge of a relationship between two or more concepts;
present the stimulus response stem to a user with said output device;
present to the respondent with said output device at least one device uniquely associated with each of the two or more concepts;
enable the respondent to use said user input device to selectively manipulate at least one attribute of the at least one device to change a relationship of that device with respect to at least one other device to construct an arrangement of devices representing a relationship between the concepts with which the devices in the arrangement are associated;
store in said storage media a final value of the attributes of each of the devices in the arrangement constructed by the respondent;
convert the stored attributes into a conceptual pattern associated with the arrangement constructed by the respondent by comparing the final values of the attributes of the devices of the arrangement so that the conceptual pattern represents at least one relationship between the devices in the arrangement as determined by a comparison of the final values of the attributes of the devices and not by the final values of the attributes of the devices;
retrieve a pre-defined scoring rubric from said storage media, the pre-defined scoring rubric including a conceptual pattern representing a pre-defined relationship between devices and corresponding to a relationship between two or more concepts; and
compare the conceptual pattern of the arrangement constructed by the respondent with the conceptual pattern of the scoring rubric to determine whether or not the conceptual pattern associated with the arrangement constructed by the respondent matches the conceptual pattern of the scoring rubric;
wherein the scoring rubric is a set of relationships between the two or more concepts and is independent of the devices associated with each concept.
15. The system of claim 14 , wherein said user input device comprises one or more of a mouse, keyboard, joystick, microphone, remote control unit, camera, tactile sensor, or biometric sensor.
16. The system of claim 14 , wherein said output device comprises one or more of a display, joystick feedback components, speakers, or actuators.
17. A method for administering a constructed response assessment item to a respondent, wherein the method is implemented using a computer system programmed to perform steps comprising:
presenting a stimulus response stem requiring the respondent to demonstrate knowledge of a relationship between two or more concepts;
providing the respondent at least one device uniquely associated with each of the two or more concepts, at least one device having at least one attribute that can be selectively manipulated by the respondent to change a relationship of that device with respect to at least one other device to construct an arrangement of devices representing a relationship between the concepts with which the devices in the arrangement are associated;
storing a final value of the attributes of each of the devices in the arrangement constructed by the respondent;
converting the stored attributes into a conceptual pattern associated with the arrangement constructed by the respondent by comparing the final values of the attributes of the devices of the arrangement so that the conceptual pattern represents at least one relationship between the devices in the arrangement as determined by a comparison of the final values of the attributes of the devices and not by the final values of the attributes of the devices; and
comparing the conceptual pattern of the arrangement constructed by the respondent with a conceptual pattern of a pre-defined scoring rubric representing a pre-defined relationship between devices to determine whether or not the conceptual pattern associated with the arrangement constructed by the respondent matches the conceptual pattern of the scoring rubric;
wherein the scoring rubric is a set of relationships between the two or more concepts and is independent of the devices associated with each concept.
18. The method of claim 17 , wherein said pre-defined rubric includes a conceptual pattern corresponding to a correct relationship between the two or more concepts.
19. The method of claim 17 , wherein the devices comprise virtual items displayed on the computer that the respondent can electronically manipulate and/or position with respect to each other to create a virtual arrangement of the devices demonstrating a relationship between the concepts with which the devices are associated.
20. The method of claim 17 , wherein the attribute of each device that can be manipulated includes one or more of size, length, height, spatial, orientation, color, direction of movement, speed of movement, sound, sound volume, pattern, shape, and time.
21. The method of claim 17 , wherein said method further associates an implication to the conceptual pattern of the arrangement constructed by the respondent based on said comparing step, wherein the implication corresponds to a pattern of the pre-defined rubric and wherein each implication comprises information relating to an outcome that occurs as the result of a pattern being found in the conceptual pattern of the arrangement constructed by the respondent.