IP Library Granted Patent US 10,769,962
Granted Patent B1
US 10,769,962 · App. 15/145,459 · Granted Sep 8, 2020

Systems and methods for generating a personalization score for a constructed response

Inventors: Beata Beigman Klebanov (Hopewell, NJ); Jill Burstein (Princeton, NJ)
Assignee: Educational Testing Service
G09B19/00
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,769,962
App. No.
15/145,459
Granted
Sep 8, 2020
Kind
B1
Abstract

Computer-implemented systems and methods for generating a personalization score for a constructed response are provided. A constructed response generated by a user is received. The constructed response is processed with a processing system to generate one or more feature values representative of one or more aspects of the constructed response. A personalization score for the constructed response is generated using the processing system by applying a computer scoring model to the one or more feature values. The personalization score provides a measure of a personalization of the constructed response.

Claims (71)

1. A computer-implemented method of generating a personalization score for a constructed response, the method comprising:

receiving, by a processing system having one or more data processors and one or more computer-readable medium encoded with instructions for commanding the processing system, a constructed response generated by a user;

processing the constructed response with the processing system to generate one or more feature values representative of one or more aspects of the constructed response, wherein processing the constructed response comprises:

identifying, using the processing system, words of the constructed response;

extracting, using the processing system, features from the words of the constructed response, wherein the extracted features are indicative of a level of utility value articulated in the constructed response and comprise a log frequency per 1000 words of first person pronouns and a total number of indefinite pronouns in the constructed response; and

generating, using the processing system, the one or more feature values based on the extracted features;

generating a personalization score for the constructed response using the processing system by applying a computer scoring model to the one or more feature values, the personalization score providing a measure indicative of the level of utility value articulated in the constructed response; and

displaying the personalization score.

2. The computer-implemented method of claim 1 , wherein the one or more feature values are indicative of pronoun usage in the constructed response.

3. The computer-implemented method of claim 2 , wherein the extracted features comprise:

a total number of first person singular pronouns in the constructed response;

a total number of first person plural pronouns in the constructed response;

a total number of second person pronouns in the constructed response; and

a total number of possessive determiners in the constructed response.

4. The computer-implemented method of claim 1 ,

wherein the constructed response is on a topic, and

wherein the one or more feature values are indicative of a degree to which the constructed response describes a relevance of the topic to the user or to one or more other persons.

5. The computer-implemented method of claim 4 , wherein the topic relates to science, technology, engineering, or mathematics.

6. The computer-implemented method of claim 4 , wherein the one or more feature values are indicative of vocabulary usage in the constructed response that is characteristic of the topic.

7. The computer-implemented method of claim 6 , wherein processing the constructed response comprises:

identifying words of the constructed response that are characteristic of the topic based on a set of words previously determined to be characteristic of the topic,

wherein the one or more feature values are based on a proportion of words in the constructed response that are characteristic of the topic.

8. The computer-implemented method of claim 1 , wherein the one or more feature values are indicative of vocabulary usage in the constructed response.

9. The computer-implemented method of claim 8 , wherein the one or more feature values include:

an average word length of the constructed response;

a median word frequency of the constructed response;

a value indicative of a degree to which the constructed response utilizes academic words;

a value indicative of a degree to which the constructed response utilizes concrete words;

a count of a number of different words used in the constructed response.

10. The computer-implemented method of claim 1 , wherein the computer scoring model comprises multiple weighted variables determined by training the computer scoring model relative to a plurality of training data.

11. The computer-implemented method of claim 1 , wherein the one or more feature values are (i) indicative of vocabulary usage in the constructed response that is characteristic of argumentative writing, and (ii) indicative of vocabulary usage in the constructed response that is characteristic of hedging or boosting.

12. The computer-implemented method of claim 1 , wherein the one or more feature values are indicative of vocabulary usage in the constructed response that is characteristic of narrativity.

13. The computer-implemented method of claim 12 , further comprising:

using a part-of-speech tagger to associate words of the constructed response with part-of speech tags;

processing the part-of-speech tags with the processing system to generate a value indicative of a degree to which the constructed response utilizes past-tense verbs, wherein the one or more feature values include the value, and

wherein the one or more feature values further include a value indicative of a degree to which the constructed response utilizes common verbs.

14. The computer-implemented method of claim 1 , wherein the one or more feature values are (i) indicative of vocabulary usage in the constructed response that expresses positive or negative affect, (ii) indicative of vocabulary usage in the constructed response that expresses social relations or social interactions, (iii) indicative of vocabulary usage in the constructed response that expresses cognitive engagement, (iv) indicative of vocabulary usage in the constructed response that refers to health or disease, (v) indicative of vocabulary usage in the constructed response that refers to risk, and (vi) indicative of vocabulary usage in the constructed response that refers to ingestion.

15. A computer-implemented system for generating a personalization score for a constructed response, the system comprising:

a processing system; and

computer-readable memory in communication with the processing system encoded with instructions for commanding the processing system to execute steps comprising:

receiving, by the processing system having one or more data processors and one or more computer-readable medium encoded with instructions for commanding the processing system, a constructed response generated by a user;

processing the constructed response with the processing system to generate one or more feature values representative of one or more aspects of the constructed response, wherein processing the constructed response comprises:

identifying, using the processing system, words of the constructed response;

extracting, using the processing system, features from the words of the constructed response, wherein the extracted features are indicative of a level of utility value articulated in the constructed response and comprise a total number of first person pronouns and a log frequency per 1000 words of indefinite pronouns in the constructed response; and

generating, using the processing system, the one or more feature values based on the extracted features;

generating a personalization score for the constructed response by applying a computer scoring model to the one or more feature values, the personalization score providing a measure indicative of the level of utility value articulated in the constructed response; and

displaying the personalization score.

16. The computer-implemented system of claim 15 , wherein the one or more feature values are indicative of pronoun usage in the constructed response.

17. The computer-implemented system of claim 15 ,

wherein the constructed response is on a topic, and

wherein the one or more feature values are indicative of a degree to which the constructed response describes a relevance of the topic to the user or to one or more other persons.

18. The computer-implemented system of claim 15 , wherein the one or more feature values are indicative of vocabulary usage in the constructed response that is characteristic of a topic.

19. The computer-implemented system of claim 15 , wherein the one or more feature values include

an average word length of the constructed response,

a median word frequency of the constructed response,

a value indicative of a degree to which the constructed response utilizes academic words,

a value indicative of a degree to which the constructed response utilizes concrete words, and

a count of a number of different words used in the constructed response.

20. The computer-implemented system of claim 15 , wherein the one or more feature values are (i) indicative of vocabulary usage in the constructed response that is characteristic of argumentative writing, and (ii) indicative of vocabulary usage in the constructed response that is characteristic of hedging or boosting.

21. The computer-implemented system of claim 15 , wherein the one or more feature values are indicative of vocabulary usage in the constructed response that is characteristic of narrativity.

22. The computer-implemented system of claim 15 , wherein the one or more feature values are (i) indicative of vocabulary usage in the constructed response that expresses positive or negative affect, (ii) indicative of vocabulary usage in the constructed response that expresses social relations or social interactions, (iii) indicative of vocabulary usage in the constructed response that expresses cognitive engagement, (iv) indicative of vocabulary usage in the constructed response that refers to health or disease, (v) indicative of vocabulary usage in the constructed response that refers to risk, and (vi) indicative of vocabulary usage in the constructed response that refers to ingestion.

23. A non-transitory computer-readable storage medium for generating a personalization score for a constructed response, the computer-readable storage medium comprising computer executable instructions which, when executed, cause a processing system to execute steps including:

receiving, by a processing system having one or more data processors for commanding the processing system, a constructed response at a tokenizer generated by a user;

processing the constructed response with the processing system to generate one or more feature values representative of one or more aspects of the constructed response, wherein processing the constructed response comprises:

identifying, using the processing system, words of the constructed response;

tokenizing, using the tokenizer, the words of the constructed response into individual words;

extracting, using the processing system, one or more features from the words of the constructed response, wherein the extracted features are indicative of a level of utility value articulated in the constructed response and comprise a log frequency per 1000 words of first person pronouns and a log frequency per 1000 words of indefinite pronouns in the constructed response; and

generating the one or more feature values based on the one or more extracted features;

generating a personalization score for the constructed response by applying a computer scoring model to the one or more feature values, the personalization score providing a measure indicative of the level of utility value articulated in the constructed response; and

displaying the personalization score.

24. The computer-implemented method of claim 1 , wherein utility value is indicative of the user's belief that a subject of the constructed response is useful and relevant to other aspects of the user's life.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 27, 2016
From: BEIGMAN KLEBANOV, BEATA; BURSTEIN, JILL
To: EDUCATIONAL TESTING SERVICE
Reel/Frame 039266/0080 →
Continuity (3)
Provisional Application 62156374 · May 4, 2015
Provisional Application 62262524 · Dec 3, 2015
Provisional Application 62274529 · Jan 4, 2016
Cited By (1)
US 12,400,658