Non-natural pattern identification for cognitive assessment
Methods, systems, and apparatus, including medium-encoded computer program products, for detection of cheating on a cognitive test. In one aspect, a method includes receiving first information concerning a person, the first information specifying the person's responses, and lack thereof, for items of a cognitive test administered to the person, wherein the cognitive test includes multiple item-recall trials used to assess cognitive impairment; analyzing the first information using a classification algorithm trained on second information concerning a group of people to whom the cognitive test has been administered, the classification algorithm generated in accordance with a first part and a second part, the first part distinguishing between cheaters and non-cheaters, and the second part distinguishing between impaired cheaters and non-impaired cheaters; and identifying, based on the analyzing, the person as a cheater requiring a verification test to confirm an initial result of the cognitive test.
1. A computer-implemented method comprising:
receiving first information concerning a person, the first information specifying the person's responses, and lack thereof, for items of a cognitive test administered to the person, wherein the cognitive test comprises multiple item-recall trials used to assess cognitive impairment;
analyzing the first information using a classification algorithm trained on second information concerning a group of people to whom the cognitive test has been administered, the classification algorithm generated in accordance with a first part and a second part, the first part distinguishing between cheaters and non-cheaters, and the second part distinguishing between impaired cheaters and non-impaired cheaters; and
identifying, based on the analyzing, the person as a cheater requiring a verification test to confirm an initial result of the cognitive test.
2. The method of claim 1 , wherein the classification algorithm is configured to check for cheating strategies characteristic of persons with Alzheimer's disease or a related disorder (ADRD).
3. The method of claim 1 , wherein the classification algorithm is selected to maximize sensitivity while minimizing reduction in specificity, which preserves a high negative predictive value while maintaining a low misclassification rate for impaired cheaters, and the group of people comprising a first sample and a second sample, the method further comprising:
analyzing data in the first part for the first sample to identify a subset of variables that discriminate between cheaters and non-cheaters, and validating results in the first part using the second sample; and
analyzing, in the second part, data of persons identified as cheaters in the first part to identify a subset of variables that discriminate between impaired cheaters and non-impaired cheaters.
4. The method of claim 3 , wherein the subset of variables that discriminate between cheaters and non-cheaters comprises education and multiple numbers corresponding to items recalled on two or more of the multiple item-recall trials including a delayed free recall trial, and the subset of variables that discriminate between impaired cheaters and non-impaired cheaters comprises age, gender, education and a number corresponding to items recalled on at least one of the multiple item-recall trials.
5. The method of claim 1 , wherein the multiple item-recall trials comprise word recall tests of memory, and the analyzing comprises distinguishing between impaired and non-impaired individuals based on a total number of words recalled across the trials.
6. The method of claim 1 , wherein the analyzing comprises evaluating a probability of an order of items recalled by the person given probabilities of recall patterns for the group of people.
7. A non-transitory computer-readable medium encoding a computer program product operable to cause data processing apparatus to perform operations comprising:
receiving first information concerning a person, the first information specifying the person's responses, and lack thereof, for items of a cognitive test administered to the person, wherein the cognitive test comprises multiple item-recall trials used to assess cognitive impairment;
analyzing the first information using a classification algorithm trained on second information concerning a group of people to whom the cognitive test has been administered, the classification algorithm generated in accordance with a first part and a second part, the first part distinguishing between cheaters and non-cheaters, and the second part distinguishing between impaired cheaters and non-impaired cheaters; and
identifying, based on the analyzing, the person as a cheater requiring a verification test to confirm an initial result of the cognitive test.
8. The non-transitory computer-readable medium of claim 7 , wherein the classification algorithm is configured to check for cheating strategies characteristic of persons with Alzheimer's disease or a related disorder (ADRD).
9. The non-transitory computer-readable medium of claim 7 , wherein the classification algorithm is selected to maximize sensitivity while minimizing reduction in specificity, which preserves a high negative predictive value while maintaining a low misclassification rate for impaired cheaters, and the group of people comprising a first sample and a second sample, the operations further comprising:
analyzing data in the first part for the first sample to identify a subset of variables that discriminate between cheaters and non-cheaters, and validating results in the first part using the second sample; and
analyzing, in the second part, data of persons identified as cheaters in the first part to identify a subset of variables that discriminate between impaired cheaters and non-impaired cheaters.
10. The non-transitory computer-readable medium of claim 9 , wherein the subset of variables that discriminate between cheaters and non-cheaters comprises education and multiple numbers corresponding to items recalled on two or more of the multiple item-recall trials including a delayed free recall trial, and the subset of variables that discriminate between impaired cheaters and non-impaired cheaters comprises age, gender, education and a number corresponding to items recalled on at least one of the multiple item-recall trials.
11. The non-transitory computer-readable medium of claim 7 , wherein the multiple item-recall trials comprise word recall tests of memory, and the analyzing comprises distinguishing between impaired and non-impaired individuals based on a total number of words recalled across the trials.
12. The non-transitory computer-readable medium of claim 7 , wherein the analyzing comprises evaluating a probability of an order of items recalled by the person given probabilities of recall patterns for the group of people.
13. A system comprising:
a user device; and
one or more computers operable to interact with the device and to perform operations comprising:
receiving first information concerning a person, the first information specifying the person's responses, and lack thereof, for items of a cognitive test administered to the person, wherein the cognitive test comprises multiple item-recall trials used to assess cognitive impairment;
analyzing the first information using a classification algorithm trained on second information concerning a group of people to whom the cognitive test has been administered, the classification algorithm generated in accordance with a first part and a second part, the first part distinguishing between cheaters and non-cheaters, and the second part distinguishing between impaired cheaters and non-impaired cheaters; and
identifying, based on the analyzing, the person as a cheater requiring a verification test to confirm an initial result of the cognitive test.
14. The system of claim 13 , wherein the classification algorithm is configured to check for cheating strategies characteristic of persons with Alzheimer's disease or a related disorder (ADRD).
15. The system of claim 13 , wherein the classification algorithm is selected to maximize sensitivity while minimizing reduction in specificity, which preserves a high negative predictive value while maintaining a low misclassification rate for impaired cheaters, and the group of people comprising a first sample and a second sample, the operations further comprising:
analyzing data in the first part for the first sample to identify a subset of variables that discriminate between cheaters and non-cheaters, and validating results in the first part using the second sample; and
analyzing, in the second part, data of persons identified as cheaters in the first part to identify a subset of variables that discriminate between impaired cheaters and non-impaired cheaters.
16. The system of claim 15 , wherein the subset of variables that discriminate between cheaters and non-cheaters comprises education and multiple numbers corresponding to items recalled on two or more of the multiple item-recall trials including a delayed free recall trial, and the subset of variables that discriminate between impaired cheaters and non-impaired cheaters comprises age, gender, education and a number corresponding to items recalled on at least one of the multiple item-recall trials.
17. The system of claim 13 , wherein the multiple item-recall trials comprise word recall tests of memory, and the analyzing comprises distinguishing between impaired and non-impaired individuals based on a total number of words recalled across the trials.
18. The system of claim 13 , wherein the analyzing comprises evaluating a probability of an order of items recalled by the person given probabilities of recall patterns for the group of people.
19. The system of claim 13 , wherein the one or more computers comprise a server system operable to interact with the device through a data communication network, and the device is operable to interact with the server as a client.
20. The system of claim 13 , wherein the device comprises a user interface device, the one or more computers comprise the user interface device, and the operations further comprise outputting an indication of the identified person to a device comprising a computer-readable medium.