Methods and apparatus for identification and analysis of temporally differing corpora
View Patent ↗Differences are identified, at the lexical unit and/or phrase level, between time-varying corpora. A corpus for a time period of interest is compared with a reference corpus. N-grams are generated for both the corpus of interest and reference corpus. Numbers of occurrences are counted. An average number of occurrences, for each n-gram of the reference corpus, is determined. A difference value, between number of occurrences in corpus of interest and average number of occurrences, is determined. Each difference value is normalized. N-grams can be selected for display, or for further processing, on the basis of the normalized difference value. Further processing can include selecting a sample period. A plurality of reference corpora are produced, where a begin time, for each sub-corpus of the plurality of reference corpora, differs, from a begin time for the corpus of interest, by an integer multiple of the sample period. Word Cloud visualization is shown.
1. A method for identifying n-grams about an object, comprising:
identifying, as a result of computing hardware and programmable memory, an object-specific corpus, that is a subset of a first corpus, where approximately all statements of the object-specific corpus are about a same first object;
identifying, as a result of computing hardware and programmable memory, statements of the object-specific corpus, for inclusion in a corpus of interest, upon a basis of a statement relating to a time period of interest;
identifying, as a result of computing hardware and programmable memory, statements of the object-specific corpus, for inclusion in a reference corpus, upon a basis of a statement relating to a reference time period that is different from the time period of interest;
identifying, as a result of computing hardware and programmable memory, n-grams of the corpus of interest, for inclusion in a corpus-of-interest list of n-grams;
identifying, as a result of computing hardware and programmable memory, n-grams of the reference corpus, for inclusion in a reference-corpus list of n-grams;
identifying, as a result of computing hardware and programmable memory, for each n-gram of the corpus-of-interest list of n-grams, for subsequent access in conjunction with an n-gram, a number of occurrences of the n-gram in the corpus of interest;
identifying, as a result of computing hardware and programmable memory, for each n-gram of the reference-corpus list of n-grams, for subsequent access in conjunction with an n-gram, a number of occurrences of the n-gram in the reference corpus;
determining, as a result of computing hardware and programmable memory, a selected list of n-grams from the corpus-of-interest list of n-grams or the reference-corpus list of n-grams;
determining, as a result of computing hardware and programmable memory, for each n-gram of the selected list of n-grams, for subsequent access in conjunction with an n-gram, an average number of occurrences of the n-gram, in the reference-corpus;
determining, as a result of computing hardware and programmable memory, for each n-gram of the selected list of n-grams, for subsequent access in conjunction with an n-gram, a difference value, between a number of occurrences of the n-gram in the corpus of interest and an average number of occurrences of the n-gram in the reference-corpus;
normalizing, as a result of computing hardware and programmable memory, for each n-gram of the selected list of n-grams, for subsequent access in conjunction with an n-gram, the difference value to produce a normalized difference value; and
determining, as a result of computing hardware and programmable memory, from the selected list of n-grams, a second selected list of n-grams, on a basis of the normalized difference value.
2. The method of claim 1 , wherein a duration of the reference time period is equal to an integer times a duration of the time period of interest.
3. The method of claim 1 , wherein the step of determining a selected list of n-grams further comprises:
determining an n-gram, for inclusion in the selected list of n-grams, on a basis of a number of occurrences of the n-gram in the corpus of interest or a number of occurrences of the n-gram in the reference corpus.
4. The method of claim 1 , wherein the step of determining an average number of occurrences of an n-gram further comprises:
determining a first multiple that, when a time duration of the corpus of interest is multiplied by, produces a time duration represented by the reference corpus; and
using the first multiple to determine an average number of occurrences of an n-gram in the reference-corpus.
5. The method of claim 4 , wherein the step of using the first multiple further comprises:
dividing, for each n-gram of the selected list of n-grams, a number of occurrences of the n-gram in the reference corpus by the multiple.
6. The method of claim 1 , wherein the step of normalizing, to produce a normalized difference value, further comprises:
dividing, for each n-gram of the selected list of n-grams, a difference value by an average number of occurrences of the n-gram in the reference-corpus.
7. The method of claim 1 , wherein the step of normalizing, to produce a normalized difference value, further comprises:
dividing, for each n-gram of the selected list of n-grams, a difference value by a number of occurrences of the n-gram in the corpus of interest.
8. The method of claim 1 , wherein the step of normalizing, to produce a normalized difference value, further comprises:
dividing, for each n-gram of the selected list of n-grams, a difference value by a standard deviation of occurrences of the n-gram in the reference-corpus.
9. The method of claim 1 , wherein the step of determining a second selected list of n-grams further comprises:
ordering the selected list of n-grams on a basis of decreasing normalized difference value; and
selecting a first predetermined number of n-grams from the ordered selected list of n-grams.
10. The method of claim 1 , further comprising:
varying a first graphical dimension or characteristic, for a display of each n-gram of the second selected list of n-grams, on a basis of the normalized difference value of the n-gram.
11. The method of claim 10 , wherein the first graphical dimension is a font size, for a display of each n-gram of the second selected list of n-grams.
12. The method of claim 11 , wherein the second selected list of n-grams is displayed as a word cloud.
13. The method of claim 10 , further comprising:
varying a second graphical dimension or characteristic, for a display of each n-gram of the second selected list of n-grams, on a basis of a number of occurrences of the n-gram.
14. The method of claim 13 , wherein the first graphical characteristic is a font transparency and the second graphical dimension is a font size, for a display of each n-gram of the second selected list of n-grams.
15. The method of claim 14 , wherein the second selected list of n-grams is displayed as a word cloud.
16. The method of claim 1 , further comprising:
selecting a sample period;
producing a plurality of reference corpora, wherein a begin time, for each sub-corpus of the plurality of reference corpora, differs, from a begin time for the corpus of interest, by an integer multiple of the sample period;
identifying, for each n-gram of the second selected list of n-grams and for each sub-corpus of the plurality of reference corpora, for subsequent access in conjunction with an n-gram, a number of occurrences of the n-gram in the sub-corpus;
determining, for each n-gram of the second selected list of n-grams, for subsequent access in conjunction with an n-gram, a second average number of occurrences of the n-gram, from a number of occurrences of the n-gram in each sub-corpus;
determining, for each n-gram of the second selected list of n-grams, for subsequent access in conjunction with an n-gram, a second difference value, between a number of occurrences of the n-gram in the corpus of interest and a second average number of occurrences of the n-gram;
normalizing, for each n-gram of the second selected list of n-grams, for subsequent access in conjunction with an n-gram, the second difference value to produce a second normalized difference value; and
determining, from the second selected list of n-grams, a third selected list of n-grams, on a basis of the second normalized difference value.
17. The method of claim 16 , wherein the step of normalizing, to produce a second normalized difference value, further comprises:
determining, for each n-gram of the second selected list of n-grams, a standard deviation from a number of occurrences of the n-gram in each sub-corpus;
dividing, for each n-gram of the second selected list of n-grams, a second difference value by a standard deviation.
18. The method of claim 16 , further comprising:
varying a first graphical dimension or characteristic, for a display of each n-gram of the third selected list of n-grams, on a basis of the second normalized difference value of the n-gram.
19. The method of claim 18 , wherein the first graphical dimension is a font size, for a display of each n-gram of the second selected list of n-grams.
20. The method of claim 18 , further comprising:
varying a second graphical dimension or characteristic, for a display of each n-gram of the second selected list of n-grams, on a basis of a number of occurrences of the n-gram.
21. The method of claim 20 , wherein the first graphical characteristic is a font transparency and the second graphical dimension is a font size, for a display of each n-gram of the second selected list of n-grams.
22. The method of claim 1 , wherein the first object is a brand.
23. A system for identifying n-grams about an object, comprising:
one or more processors and programmable memory, wherein the system is configured:
to accomplish identifying an object-specific corpus, that is a subset of a first corpus, where approximately all statements of the object-specific corpus are about a same first object;
to accomplish identifying statements of the object-specific corpus, for inclusion in a corpus of interest, upon a basis of a statement relating to a time period of interest;
to accomplish identifying statements of the object-specific corpus, for inclusion in a reference corpus, upon a basis of a statement relating to a reference time period that is different from the time period of interest;
to accomplish identifying n-grams of the corpus of interest, for inclusion in a corpus-of-interest list of n-grams;
to accomplish identifying n-grams of the reference corpus, for inclusion in a reference-corpus list of n-grams;
to accomplish identifying, for each n-gram of the corpus-of-interest list of n-grams, for subsequent access in conjunction with an n-gram, a number of occurrences of the n-gram in the corpus of interest;
to accomplish identifying, for each n-gram of the reference-corpus list of n-grams, for subsequent access in conjunction with an n-gram, a number of occurrences of the n-gram in the reference corpus;
to accomplish determining a selected list of n-grams from the corpus-of-interest list of n-grams or the reference-corpus list of n-grams;
to accomplish determining, for each n-gram of the selected list of n-grams, for subsequent access in conjunction with an n-gram, an average number of occurrences of the n-gram, in the reference-corpus;
to accomplish determining, for each n-gram of the selected list of n-grams, for subsequent access in conjunction with an n-gram, a difference value, between a number of occurrences of the n-gram in the corpus of interest and an average number of occurrences of the n-gram in the reference-corpus;
to accomplish normalizing, for each n-gram of the selected list of n-grams, for subsequent access in conjunction with an n-gram, the difference value to produce a normalized difference value; and
to accomplish determining, from the selected list of n-grams, a second selected list of n-grams, on a basis of the normalized difference value.
24. The system of claim 23 , wherein the configuration to accomplish determining, an average number of occurrences of an n-gram, is further configured:
to accomplish determining a first multiple that, when a time duration of the corpus of interest is multiplied by, produces a time duration represented by the reference corpus; and
to accomplish using the first multiple to determine an average number of occurrences of an n-gram in the reference-corpus.
25. The system of claim 23 , wherein the configuration to accomplish normalizing, to produce a normalized difference value, is further configured:
to accomplish dividing, for each n-gram of the selected list of n-grams, a difference value by a standard deviation of occurrences of the n-gram in the reference-corpus.
26. The system of claim 23 , further configured:
to accomplish varying a first graphical dimension or characteristic, for a display of each n-gram of the second selected list of n-grams, on a basis of the normalized difference value of the n-gram.
27. The system of claim 26 , further configured:
to accomplish varying a second graphical dimension or characteristic, for a display of each n-gram of the second selected list of n-grams, on a basis of a number of occurrences of the n-gram.
28. The system of claim 23 , further configured:
to accomplish selecting a sample period;
to accomplish producing a plurality of reference corpora, wherein a begin time, for each sub-corpus of the plurality of reference corpora, differs, from a begin time for the corpus of interest, by an integer multiple of the sample period;
to accomplish identifying, for each n-gram of the second selected list of n-grams and for each sub-corpus of the plurality of reference corpora, for subsequent access in conjunction with an n-gram, a number of occurrences of the n-gram in the sub-corpus;
to accomplish determining, for each n-gram of the second selected list of n-grams, for subsequent access in conjunction with an n-gram, a second average number of occurrences of the n-gram, from a number of occurrences of the n-gram in each sub-corpus;
to accomplish determining, for each n-gram of the second selected list of n-grams, for subsequent access in conjunction with an n-gram, a second difference value, between a number of occurrences of the n-gram in the corpus of interest and a second average number of occurrences of the n-gram;
to accomplish normalizing, for each n-gram of the second selected list of n-grams, for subsequent access in conjunction with an n-gram, the second difference value to produce a second normalized difference value; and
to accomplish determining, from the second selected list of n-grams, a third selected list of n-grams, on a basis of the second normalized difference value.