Systems and methods for scoring story narrations

1. A computer-implemented method for generating a narrative computer scoring model for assessing story narratives, comprising:

retrieving, by a processing system, story narratives, wherein each story narrative is associated with a prompt that elicited the story narrative;

retrieving, by the processing system, reference corpora associated with the story narratives, wherein each story narrative's associated reference corpus includes content words relating to the associated prompt;

for each of the retrieved story narratives:

determining, by the processing system, overlaps between words in the story narrative and the content words in the associated reference corpus; and

computing, by the processing system, at least one relevance feature associated with the story narrative using the determined overlaps;

retrieving, by the processing system, predetermined scores assigned to the story narratives by human raters;

generating, by the processing system, a narrative computer scoring model using the relevance features associated with the story narratives and the predetermined scores assigned to the story narratives; and

storing, by the processing system, the narrative computer scoring model;

wherein the narrative computer scoring model is configured to output computer assessments of story narratives.

2. The method of claim 1 ,

wherein the story narratives are transcriptions of audio story narratives; and

wherein the prompts are pictures.

3. The method of claim 1 , further comprising:

removing stop words from the story narratives and the reference corpora; and

lemmatizing the story narratives and the reference corpora.

4. The method of claim 1 , further comprising:

identifying synonyms for the content words in the reference corpora; and

expanding the reference corpora by including the identified synonyms.

5. The method of claim 1 , further comprising:

for each of the retrieved story narratives:

identifying, in each sentence of the story narrative, groups of a predetermined number of adjacent words;

determining a likely occurrence value for each of the groups of adjacent words, wherein the likely occurrence value represents a likelihood of the associated adjacent words occurring in a corpus; and

computing at least one collocation feature associated with the story narrative using the likely occurrence values;

wherein the training of the narrative computer scoring model further comprises using the collocation features associated with the story narratives.

6. The method of claim 5 , further comprising:

for each of the retrieved story narratives:

determining a statistical distribution of the groups based on the associated likely occurrence values;

wherein the computing of the at least one collocation feature uses the statistical distribution.

7. The method of claim 6 , wherein determining the statistical distribution includes conceptually assigning each of the groups to one of a plurality of bins based on the group's likely occurrence value.

8. The method of claim 5 , wherein the likely occurrence value is determined using pointwise mutual information (PMI).

9. The method of claim 1 , further comprising:

for each of the retrieved story narratives:

identifying transition cues within the story narrative using one or more lexicons; and

computing at least one discourse feature associated with the story narrative using the identified transition cues;

wherein the training of the narrative computer scoring model further comprises using the discourse features associated with the story narratives.

10. The method of claim 9 ,

wherein identifying the transition cues includes identifying temporal transition cues and identifying causal transition cues; and

wherein the computing of the at least one discourse feature includes using the identified temporal transition cues and the causal transition cues.

11. The method of claim 1 , further comprising:

for each of the retrieved story narratives:

identifying syntactic categories of words in the story narrative using a part-of-speech tagger;

determining statistical measures of the words falling in one or more particular syntactic categories; and

computing at least one detailing feature associated with the story narrative using the statistical measures;

wherein the training of the narrative computer scoring model further comprises using the detailing features associated with the story narratives.

12. The method of claim 11 , wherein the one or more particular syntactic categories are selected from the group consisting of proper nouns, adjectives, and adverbs.

13. The method of claim 1 , further comprising:

for each of the retrieved story narratives:

identifying sentiment words and subjective words within the story narrative using one or more lexicons; and

computing at least one sentiment feature associated with the story narrative using the identified sentiment words and subjective words;

wherein the training of the narrative computer scoring model further comprises using the sentiment features associated with the story narratives.

14. The method of claim 13 , wherein the sentiment words and subjective words used for computing the at least one sentiment feature have either positive or negative polarity.

15. The method of claim 13 , wherein the sentiment words and subjective words used for computing the at least one sentiment feature have neutral polarity.

16. A computer-implemented system for generating a narrative computer scoring model for assessing story narratives, comprising:

one or more data processors; and

a memory comprising instructions which when executed cause the one or more data processors to execute steps comprising:

retrieving story narratives, wherein each story narrative is associated with a prompt that elicited the story narrative;

retrieving reference corpora associated with the story narratives, wherein each story narrative's associated reference corpus includes content words relating to the associated prompt;

for each of the retrieved story narratives:

determining overlaps between words in the story narrative and the content words in the associated reference corpus; and

computing at least one relevance feature associated with the story narrative using the determined overlaps;

retrieving predetermined scores assigned to the story narratives by human raters;

generating a narrative computer scoring model using the relevance features associated with the story narratives and the predetermined scores assigned to the story narratives; and

storing the narrative computer scoring model;

wherein the narrative computer scoring model is configured to output computer assessments of story narratives.

17. The system of claim 16 , wherein the memory comprises instructions for causing the one or more data processors to execute further steps comprising:

for each of the retrieved story narratives:

identifying, in each sentence of the story narrative, groups of a predetermined number of adjacent words;

determining a likely occurrence value for each of the groups of adjacent words, wherein the likely occurrence value represents a likelihood of the associated adjacent words occurring in a corpus; and

computing at least one collocation feature associated with the story narrative using the likely occurrence values;

wherein the training of the narrative computer scoring model further comprises using the collocation features associated with the story narratives.

18. The system of claim 16 , wherein the memory comprises instructions for causing the one or more data processors to execute further steps comprising:

for each of the retrieved story narratives:

identifying transition cues within the story narrative using one or more lexicons; and

computing at least one discourse feature associated with the story narrative using the identified transition cues;

wherein the training of the narrative computer scoring model further comprises using the discourse features associated with the story narratives.

19. The system of claim 16 , wherein the memory comprises instructions for causing the one or more data processors to execute further steps comprising:

for each of the retrieved story narratives:

identifying syntactic categories of words in the story narrative using a part-of-speech tagger;

determining statistical measures of the words falling in one or more particular syntactic categories; and

computing at least one detailing feature associated with the story narrative using the statistical measures;

wherein the training of the narrative computer scoring model further comprises using the detailing features associated with the story narratives.

20. The system of claim 16 , wherein the memory comprises instructions for causing the one or more data processors to execute further steps comprising:

for each of the retrieved story narratives:

identifying sentiment words and subjective words within the story narrative using one or more lexicons; and

computing at least one sentiment feature associated with the story narrative using the identified sentiment words and subjective words;

wherein the training of the narrative computer scoring model further comprises using the sentiment features associated with the story narratives.

21. A non-transitory computer-readable medium encoded with instructions for generating a narrative computer scoring model for assessing story narratives, the instructions being configured to cause a computer processing system to execute steps comprising:

retrieving story narratives, wherein each story narrative is associated with a prompt that elicited the story narrative;

retrieving reference corpora associated with the story narratives, wherein each story narrative's associated reference corpus includes content words relating to the associated prompt;

for each of the retrieved story narratives:

determining overlaps between words in the story narrative and the content words in the associated reference corpus; and

computing at least one relevance feature associated with the story narrative using the determined overlaps;

retrieving predetermined scores assigned to the story narratives by human raters;

generating a narrative computer scoring model using the relevance features associated with the story narratives and the predetermined scores assigned to the story narratives; and

storing the narrative computer scoring model;

wherein the narrative computer scoring model is configured to output computer assessments of story narratives.

22. The non-transitory computer-readable medium of claim 21 , wherein the instructions being configured to cause the computer processing system to execute further steps comprising:

for each of the retrieved story narratives:

identifying, in each sentence of the story narrative, groups of a predetermined number of adjacent words;

determining a likely occurrence value for each of the groups of adjacent words, wherein the likely occurrence value represents a likelihood of the associated adjacent words occurring in a corpus; and

computing at least one collocation feature associated with the story narrative using the likely occurrence values;

wherein the training of the narrative computer scoring model further comprises using the collocation features associated with the story narratives.

23. The non-transitory computer-readable medium of claim 21 , wherein the instructions being configured to cause the computer processing system to execute further steps comprising:

for each of the retrieved story narratives:

identifying transition cues within the story narrative using one or more lexicons; and

computing at least one discourse feature associated with the story narrative using the identified transition cues;

wherein the training of the narrative computer scoring model further comprises using the discourse features associated with the story narratives.

24. The non-transitory computer-readable medium of claim 21 , wherein the instructions being configured to cause the computer processing system to execute further steps comprising:

for each of the retrieved story narratives:

identifying syntactic categories of words in the story narrative using a part-of-speech tagger;

determining statistical measures of the words falling in one or more particular syntactic categories; and

computing at least one detailing feature associated with the story narrative using the statistical measures;

wherein the training of the narrative computer scoring model further comprises using the detailing features associated with the story narratives.

25. The non-transitory computer-readable medium of claim 21 , wherein the instructions being configured to cause the computer processing system to execute further steps comprising:

for each of the retrieved story narratives:

identifying sentiment words and subjective words within the story narrative using one or more lexicons; and

computing at least one sentiment feature associated with the story narrative using the identified sentiment words and subjective words;

wherein the training of the narrative computer scoring model further comprises using the sentiment features associated with the story narratives.