Computer system and method for training certifying or monitoring human clinical raters
A method and system are disclosed for computerized training, monitoring, certification or re-certification of human raters in clinical trials. The invention provides an interactive computerized interview that can be compared scores obtained by human raters to monitor and train clinical raters on a continuing basis in order to reduce rater drift and variance during clinical trials. Remediation can be automatically flagged or provided to improve a deviating rater's performance. Computerized assessment of symptom severity is provided without the need for human clinical raters. The system and method enable screening of prospective patients for inclusion or exclusion from a clinical trial by automatically obtaining computerized ratings of symptom severity.
1. A method for recertifying or monitoring performance of a clinical rater participating in a clinical trial comprising:
receiving one or more rater inputs reflecting the rater's clinical evaluation of a severity of a previously diagnosed condition in a subject;
performing a computerized assessment of the subject to quantify severity of the previously diagnosed condition in the subject through a computerized interview that comprises:
presenting a first question to the subject and receiving a first input from the subject in response thereto;
based on the first input, selecting a second question from a plurality of questions;
presenting the selected second question to the subject and receiving a second input from the subject in response thereto; and
based on one or more inputs received from the subject, determining in accordance with a clinical rating scale a computerized severity score for the previously diagnosed condition in the subject; and
determining, via a processor, whether the one or more rater inputs are in accordance with the computerized severity score.
2. The method according to claim 1 , wherein the previously diagnosed condition is a psychiatric illness.
3. The method according to claim 1 , wherein the clinical rating scale is in accordance with one of the following scales: the Young Mania Rating Scale (YMRS); the Montgomery-Asberg Depression Rating Scale (MADRS); the Hamilton Depression Scale (HAMD); the Schedule for Affective Disorders and Schizophrenia, Current symptom version mania Rating Scale (SADS-C); an Additional Psychiatry Scale; a Geriatric/Cognitive Scale; an Other Cognitive Assessment; a Non-CNS Scale; or a member of the APA group of scales.
4. The method according to claim 1 , wherein at least one unselected question is not presented to the subject.
5. The method according to claim 1 , further comprising:
based on the second input, selecting a third question from the plurality of questions; and
presenting the selected third question to the subject and receiving a third input from the subject in response thereto.
6. The method according to claim 5 , wherein selection of the third question is based on a plurality of inputs by the subject.
7. The method according to claim 1 , wherein the questions are presented by visual, audio, or multimedia display.
8. The method according to claim 1 , wherein the inputs are received directly from the subject by means of a keyboard, a pointing device, a touch-screen device, or voice response.
9. The method according to claim 1 , wherein the computerized assessment is performed within 24 hours after a clinical interview at which the clinical evaluation is made.
10. The method according to claim 1 , wherein the clinical interview and computerized assessment are performed during the same clinic visit.
11. The method according to claim 1 , wherein the step of receiving comprises receiving a plurality of rater inputs reflecting a given rater's clinical evaluation of severity of the previously diagnosed condition in a plurality of subjects at a plurality of clinical interviews; and
wherein the step of determining comprises determining whether the plurality of rater inputs are in accordance with computerized severity scores for the previously diagnosed condition in the subjects.
12. The method according to claim 1 , wherein the step of receiving comprises receiving a plurality of rater inputs reflecting a given rater's clinical evaluation of severity of a previously diagnosed condition in a single subject at a plurality of clinical interviews of that subject; and
wherein the step of determining comprises determining whether the plurality of rater inputs are in accordance with a plurality of computerized severity scores for the previously diagnosed condition in that subject determined in conjunction with the plurality of clinical interviews.
13. A method for improving performance of a clinical rater participating or being trained to participate in a clinical trial, comprising:
receiving one or more rater inputs reflecting the clinical rater's clinical evaluation of a severity of a previously diagnosed condition in a subject;
performing a computerized assessment of the subject to quantify severity of the previously diagnosed condition in the subject through a computerized interview that comprises:
presenting a first question to the subject and receiving a first input from the subject in response thereto;
based on the first input, selecting a second question from a plurality of questions;
presenting the selected second question to the subject and receiving a second input from the subject in response thereto; and
based on one or more inputs received from the subject, determining in accordance with a clinical rating scale a computerized severity score for the previously diagnosed condition in the subject;
determining, via a processor, whether the one or more rater inputs are in accordance with the computerized severity score; and
if the one or more rater inputs substantially differs from the computerized severity score, providing an indication that the clinical rater is in need of remedial training or providing remedial training to the clinical rater.
14. The method of claim 13 , wherein the remedial training includes one or more of the following: providing explanatory material concerning how to evaluate the severity of the previously diagnosed condition, presenting a remedial script, and presenting a remedial semi-scripted interview to the clinical rater.
15. A computing device comprising:
a processing unit; and
a memory having computer executable instructions stored thereon that when executed by the processor cause the processor to perform the following steps:
receive one or more rater inputs reflecting the clinical rater's clinical evaluation of a severity of a previously diagnosed condition in a subject;
perform a computerized assessment of the subject to quantify severity of the previously diagnosed condition in the subject through a computerized interview that comprises:
presenting a first question to the subject and receiving a first input from the subject in response thereto;
based on the first input, selecting a second question from a plurality of questions;
presenting the selected second question to the subject and receiving a second input from the subject in response thereto; and
based on one or more inputs received from the subject, determining in accordance with a clinical rating scale a computerized severity score for the previously diagnosed condition in the subject;
determine, via a processor, whether the one or more rater inputs are in accordance with the computerized severity score; and
if the one or more rater inputs substantially differs from the computerized severity score, provide an indication that the clinical rater is in need of remedial training or providing remedial training to the clinical rater.
16. A computer readable medium wherein the computer readable medium is not a transient signal having stored thereon instructions that, when executed by a processor, cause the processor to perform the following steps:
receive one or more rater inputs reflecting the clinical rater's clinical evaluation of a severity of a previously diagnosed condition in a subject;
perform a computerized assessment of the subject to quantify severity of the previously diagnosed condition in the subject through a computerized interview that comprises:
presenting a first question to the subject and receiving a first input from the subject in response thereto;
based on the first input, selecting a second question from a plurality of questions;
presenting the selected second question to the subject and receiving a second input from the subject in response thereto; and
based on one or more inputs received from the subject, determining in accordance with a clinical rating scale a computerized severity score for the previously diagnosed condition in the subject;
determine, via a processor, whether the one or more rater inputs are in accordance with the computerized severity score; and
if the one or more rater inputs substantially differs from the computerized severity score, provide an indication that the clinical rater is in need of remedial training or providing remedial training to the clinical rater.
17. A system comprising:
an output device configured to present selected questions to a subject;
an input device configured to accept inputs from the subject; and
one or more computing devices configured to:
present, via the output device, a first question to the subject;
receive, via the input device, a first input from the subject in response to the first question;
based on the first input, select a second question from a plurality of questions;
present, via the output device, the selected second question to the subject;
receive, via the input device, a second input from the subject in response to the second question; and
determine a computerized severity score, based on the inputs received from the subject in accordance with a rating scale, wherein the computerized severity score quantifies a severity of a previously diagnosed condition in the subject.
18. A computerized method for assisting or training a rater for clinical trials to evaluate severity of symptoms in a subject participating in a clinical trial and having a previously diagnosed medical condition, comprising:
presenting a first output to the rater, wherein the first output to the rater conveys a first suggestion concerning a question to be posed to the subject or a clinical observation to be made concerning the subject;
receiving a first input from the rater in response to the first output;
based on the first input that was received in response to the first output, selecting, processor, a second output from a plurality of alternative second outputs, wherein the second output concerns a question to be posed to the subject by the rater in an interview for the clinical trial or a clinical observation to be made by the rater concerning the subject in the interview for the clinical trial; and
presenting the second output to the rater.
19. The method according to claim 18 , further comprising:
based on the first input and the second input received from the rater, determining in accordance with a clinical rating scale a computerized severity score for the medical condition.
20. The method according to claim 18 , wherein the rater is a health care provider evaluating the severity of symptoms in a patient or evaluating a response of a patient to treatment or placebo.
21. A computing device configured to assist or train a rater for clinical trials to evaluate severity of symptoms in a subject participating in a clinical trial and having a previously diagnosed medical condition, the computing device comprising:
a processing unit; and
a memory having computer executable instructions stored thereon that when executed by the processor cause the processor to perform the following steps:
present a first output to the rater, wherein the first output to the rater conveys a first suggestion concerning a question to be posed to the subject or a clinical observation to be made concerning the subject;
receive a first input from the rater in response to the first output;
based on the first input that was received in response to the first output, select, via a processor, a second output from a plurality of alternative second outputs, wherein the second output concerns a question to be posed to the subject by the rater in an interview for the clinical trial or a clinical observation to be made by the rater concerning the subject in the interview for the clinical trial; and
present the second output to the rater.
22. A computer readable medium wherein the computer readable medium is not a transient signal configured to assist or train a rater for clinical trials to evaluate severity of symptoms in a subject participating in a clinical trial and having a previously diagnosed medical condition, the computer readable medium having stored thereon instructions that, when executed by a processor, cause the processor to perform the following steps
present a first output to the rater conveying a first suggestion concerning a question to be posed to the subject or a clinical observation to be made concerning the subject;
receive a first input from the rater in response to the first output; and
based on the first input that was received in response to the first output, select a second output from a plurality of alternative second outputs concerning a question to be posed to the subject by the rater in an interview for the clinical trial or a clinical observation to be made by the rater concerning the subject in the interview for the clinical trial; and
present the second output to the rater.
23. A system configured to assist or train a rater for clinical trials to evaluate a severity of symptoms in a subject participating in a clinical trial and having a previously diagnosed medical condition, the system comprising:
an output device configured to present outputs to the rater;
an input device configured to accept inputs from the rater; and
one or more computers configured to perform the following steps:
present, via the output device, a first output to the rater conveying a first suggestion concerning a question to be posed to the subject or a clinical observation to be made concerning the subject;
receive, via the input device, a first input from the rater in response to the first output; and
based on the first input that was received in response to the first output, select a second output from a plurality of alternative second outputs concerning a question to be posed to the subject by the rater in an interview for the clinical trial or a clinical observation to be made by the rater concerning the subject in the interview for the clinical trial; and
present the second output to the rater.
24. A computerized method for implementing a clinical rating scale to evaluate a previously diagnosed condition in a subject presented with a series of questions to determine whether to include the subject in a clinical trial, the method comprising:
establishing a bank of questions directed to measurement of an item in the clinical rating scale;
presenting to the subject a first question selected from the bank of questions and receiving a first input from the subject in response to the first question;
presenting to the subject a second question selected from the bank of questions based on the first input and receiving a second input from the subject in response to the second question; and
based on the received first input and the received second input:
generating, via a processor, a score for the item in the clinical rating scale, wherein a given response to at least one question results in different scores for the item dependent on the place of the at least one question within the series of questions; and
determining whether to include or exclude the subject in the clinical trial.
25. The method of claim 24 , wherein each input to a question results either in generating a score for the item in the clinical rating scale or in selection of another question from the bank of questions.
26. The method of claim 24 , wherein the step of establishing comprises establishing a plurality of banks of questions, each bank directed to measurement of a distinct item in the clinical rating scale; and
the step of generating comprises generating scores for a plurality of items in the clinical rating scale.
27. The method according to claim 24 , further comprising the step of:
determining whether at least one input to a question is an acceptable input and presenting the question to the subject again if the at least one input is not acceptable.
28. The method according to claim 24 , wherein the series of questions includes a multiple choice question such that the subject selects from a list of inputs for responding to the multiple choice question.
29. The method according to claim 28 , wherein two or more inputs in the list of inputs result in a common question being presented to the subject.
30. The method according to claim 24 , further comprising:
automatically determining by previously established criteria whether or not the subject qualifies for inclusion in the clinical trial.
31. A computer readable medium wherein the computer readable medium is not a transient signal having stored thereon instructions that, when executed by a processor, cause the processor to perform the following steps:
establish a bank of questions directed to measurement of an item in the clinical rating scale;
present to the subject a first question selected from the bank of questions and receiving a first input from the subject in response to the first question;
present to the subject a second question selected from the bank of questions based on the first input and receiving a second input from the subject in response to the second question; and
based on the received first input and the received second input:
generate a score for the item in the clinical rating scale, wherein a given response to at least one question results in different scores for the item dependent on the place of the at least one question within the series of questions; and
determine whether to include or exclude the subject from a clinical trial.
32. A computing device comprising:
a processing unit; and
a memory having computer executable instructions stored thereon that when executed by the processor cause the processor to perform the following steps:
establish a bank of questions directed to measurement of an item in the clinical rating scale;
present to the subject a first question selected from the bank of questions and receiving a first input from the subject in response to the first question;
present to the subject a second question selected from the bank of questions based on the first input and receiving a second input from the subject in response to the second question; and
based on the received first input and the received second input:
generate a score for the item in the clinical rating scale, wherein a given response to at least one question results in different scores for the item dependent on the place of the at least one question within the series of questions; and
determine whether to include or exclude the subject from a clinical trial.