Conversational lexicon analyzer
View Patent ↗A system and a method for analyzing conversational data comprising colloquial or informal terms and having an informal structure. A corpus of training language maps, each associated with an entity, is generated from conversational data retrieved from sources previously associated with entities. Subsequently received conversational data is processed to generate a conversational language map which is compared to a plurality of the stored training language maps. A confidence value is generated describing the similarity of the conversational language map to each of the plurality of the stored training language maps. The entity associated with the training language map having the highest confidence value is then associated with the conversational language map.
1. A computer implemented method for analyzing conversational data, the method comprising:
receiving first conversational data that is produced by an entity, the conversational data received from a source that is associated with the entity, the entity comprising a category of persons;
identifying a first set of lexical features from the first conversational data, the first set of lexical features comprising groups of one or more words from the first conversational data;
reducing, by a computer, the first set of lexical features based on a frequency of lexical features in the first conversational data and also based on overlaps between lexical features in the first set and lexical features in a corpus of language maps, the corpus comprising a plurality of language maps that are associated with different entities;
generating a first language map based on the reduced set of lexical features; and
storing the first language map into the corpus in association with the entity.
2. The method of claim 1 , further comprising
receiving second conversational data that is produced by an unknown entity;
identifying a second set of lexical features from the second conversational data;
generating a second language map based on the second set of lexical features;
computing confidence scores for a plurality of training language maps in the corpus by comparing the second language map to the corpus; and
identifying the entity associated with the language map having the highest confidence score as the entity of the second conversational data.
3. A computer implemented method for analyzing conversational data, the method comprising:
receiving first conversational data that is produced by an entity;
identifying a first set of lexical features from the first conversational data;
reducing, by a computer, the first set of lexical features to generate a first language map; and
storing the first language map into a corpus of language maps in association with the entity, the corpus comprising a plurality of language maps that are associated with different entities.
4. The method of claim 3 , wherein receiving first conversational data that is produced by an entity comprises receiving first conversational data from a source associated with the entity.
5. The method of claim 3 , wherein an entity comprises a category of persons.
6. The method of claim 3 , wherein first conversational data comprises colloquial terms used by the entity.
7. The method of claim 3 , wherein lexical features comprise conversation units that are groups of one or more words, and identifying the first set of lexical features comprises identifying conversation units based on word stems in the first conversational data.
8. The method of claim 3 , wherein reducing the first set of lexical features to generate a first language map comprises reducing the first set of lexical features based a frequency of lexical features in the first conversational data.
9. The method of claim 3 , wherein reducing the first set of lexical features to generate a first language map comprises reducing the first set of lexical features based on overlaps between lexical features in the first set and lexical features in the corpus.
10. The method of claim 3 , wherein reducing the first set of lexical features to generate a first language map comprises:
identifying attachment scores for the first set of lexical features, each attachment score representing a strength of a relationship between a lexical feature and the entity; and
adding the attachment scores to the first language map.
11. The method of claim 3 , wherein storing the first language map into the corpus comprises updating an existing language map that was previously stored in the corpus in association with the entity.
12. The method of claim 3 , further comprising:
receiving second conversational data produced by an unknown entity;
identifying a second set of lexical features from the second conversational data;
generating a second language map based on the second set of lexical features;
comparing the second conversation language map to the corpus of language maps to identify a language map that best matches the second language map; and
identifying the entity associated with the language map that best matches the second language map as the entity of the second conversational data.
13. The method of claim 12 , wherein generating a second language map comprises reducing the second set of lexical features.
14. The method of claim 12 , wherein comparing the conversational language map to the corpus comprises:
computing confidence scores for a plurality of training language maps in the corpus; and
identifying the training language map with the highest confidence score.
15. A computer program product for analyzing conversational data, the computer program product comprising a non-transitory computer-readable medium containing instructions, the instructions executable by one or more processors for:
receiving first conversational data that is produced by an entity, the conversational data received from a source that is associated with the entity, the entity comprising a category of persons;
identifying a first set of lexical features from the first conversational data, the first set of lexical features comprising groups of one or more words from the first conversational data;
reducing the first set of lexical features based on a frequency of lexical features in the first conversational data and also based on overlaps between lexical features in the first set and lexical features in the corpus;
generating a first language map based on the reduced set of lexical features; and
storing the first language map into the corpus in association with the entity.
16. The computer program product of claim 15 , the instructions further executable by the one or more processors for:
receiving second conversational data that is produced by an unknown entity;
identifying a second set of lexical features from the second conversational data;
generating a second language map based on the second set of lexical features;
computing confidence scores for a plurality of training language maps in the corpus by comparing the second language map to the corpus; and
identifying the entity associated with the language map having the highest confidence score as the entity of the second conversational data.
17. A computer program product for analyzing conversational data, the computer program product comprising a non-transitory computer-readable medium containing instructions, the instructions executable by one or more processors for:
receiving first conversational data that is produced by an entity;
identifying a first set of lexical features from the first conversational data;
reducing the first set of lexical features to generate a first language map; and
storing the first language map into a corpus of language maps in association with the entity, the corpus comprising a plurality of language maps that are associated with different entities.
18. The computer program product of claim 17 , wherein reducing the first set of lexical features to generate a first language map comprises reducing the first set of lexical features based a frequency of lexical features in the first conversational data.
19. The computer program product of claim 17 , wherein reducing the first set of lexical features to generate a first language map comprises reducing the first set of lexical features based on overlaps between lexical features in the first set and lexical features in the corpus.
20. The computer program product of claim 17 , the instructions further executable by the one or more processors for:
receiving second conversational data produced by an unknown entity;
identifying a second set of lexical features from the second conversational data;
generating a second language map based on the second set of lexical features;
comparing the second conversation language map to the corpus of language maps to identify a language map that best matches the second language map; and
identifying the entity associated with the language map that best matches the second language map as the entity of the second conversational data.