Ontology expansion using entity-association rules and abstract relations
A method for expanding an initial ontology via processing of communication data, wherein the initial ontology is a structural representation of language elements comprising a set of entities, a set of terms, a set of term-entity associations, a set of entity-association rules, a set of abstract relations, and a set of relation instances. A method for extracting a set of significant phrases and a set of significant phrase co-occurrences from an input set of documents further includes utilizing the terms to identify relations within the training set of communication data, wherein a relation is a pair of terms that appear in proximity to one another.
1. A method comprising:
providing a set of documents from a source by a computing device;
iterating over the documents of the set of documents to subdivide the documents of the set of documents into a plurality of meaning units by the computing device;
for each meaning unit, determining an associated count for each n-gram that occurs in the meaning unit by the computing device;
for each meaning unit:
for each n-gram that occurs in the meaning unit:
computing a prominence score for the n-gram based on the determined count;
if the prominence score for the n-gram is below a prominence threshold, discarding the n-gram; and
if the prominence score for the n-gram is not below the prominence threshold:
if the n-gram is not in a set of candidate n-grams, inserting the n-gram into the set of candidate n-grams and setting the associated count to 1;
if the n-gram is in the set of candidate n-grams, incrementing the associated count; and
if the n-gram is in the set of candidate n-grams and the associated count is greater than 1:
computing a stickiness score for the n-gram; and
if the stickiness score for the n-gram is below a stickiness threshold, discarding the n-gram from the set of candidate n-grams;
based on the associated count for each n-gram in the set of candidate n-grams, estimating a language-model probability for each n-gram by the computing device;
obtaining a source-specific language model based on the estimated language-model probabilities for each n-gram by the computing device;
loading the source-specific language model into a memory by the computing device;
receiving a plurality of transcribed interactions by the computing device;
tagging the plurality of transcribed interactions using the loaded source-specific language model by the computing device; and
generating analytics based on the tagged plurality of transcribed interactions by the computing device.
2. The method of claim 1 , wherein the meaning units are sentences.
3. The method of claim 1 , wherein the documents are subdivided into meaning using a zoning algorithm.
4. The method of claim 1 , wherein determining the associated count for each n-gram that occurs in the meaning unit comprises:
determining a count for each unigram that occurs in the meaning unit;
determining a count for each bigram that occurs in the meaning unit; and
determining a count for each trigram that occurs in the meaning unit.
5. The method of claim 1 , further comprising:
if the stickiness score for the n-gram is not below the stickiness threshold:
if there is a partially overlapping n-gram in the set of candidate n-grams that has a greater associated count than the n-gram, discarding the n-gram, and
if there is no partially overlapping n-gram in the set of candidate n-grams that has the greater associated count than the n-gram, incrementing the associated count of the n-gram in the set of candidate n-grams.
6. The method of claim 5 , further comprising:
for each n-gram in the set of candidate n-grams, calculating a phrase score for the n-gram; and
determining n-grams with a phrase score above a threshold phrase score as significant phrases.
7. A system comprising:
at least one computing device; and
a computer-readable medium storing computer-executable instructions that when executed by the at least one computing device cause the system to:
provide a set of documents from a source;
iterate over the documents of the set of documents to subdivide the documents of the set of documents into a plurality of meaning units;
for each meaning unit, determine an associated count for each n-gram that occurs in the meaning unit;
for each meaning unit:
for each n-gram that occurs in the meaning unit:
compute a prominence score for the n-gram based on the determined count;
if the prominence score for the n-gram is below a prominence threshold, discard the n-gram; and
if the prominence score for the n-gram is not below the prominence threshold:
if the n-gram is not in a set of candidate n-grams, insert the n-gram into the set of candidate n-grams and set the associated count to 1;
if the n-gram is in the set of candidate n-grams, increment the associated count; and
if the n-gram is in the set of candidate n-grams and the associated count is greater than 1:
computing a stickiness score for the n-gram; and
if the stickiness score for the n-gram is below a stickiness threshold, discard the n-gram from the set of candidate n-grams;
based on the associated count for each n-gram in the set of candidate n-grams, estimate a language-model probability for each n-gram; and
obtain a source-specific language model based on the estimated language-model probabilities for each n-gram;
load the source-specific language model into a memory;
receive a plurality of transcribed interactions;
tag the plurality of transcribed interactions using the loaded source-specific language model; and
generate analytics based on the tagged plurality of transcribed interactions.
8. The system of claim 7 , wherein the meaning units are sentences.
9. The system of claim 7 , wherein the documents are subdivided into meaning using a zoning algorithm.
10. The system of claim 7 , wherein determining the associated count for each n-gram that occurs in the meaning unit comprises:
determining a count for each unigram that occurs in the meaning unit;
determining a count for each bigram that occurs in the meaning unit; and
determining a count for each trigram that occurs in the meaning unit.
11. The system of claim 7 , further comprising computer-executable instructions that when executed by the at least one computing device cause the system to:
if the stickiness score for the n-gram is not below the stickiness threshold:
if there is a partially overlapping n-gram in the set of candidate n-grams that has a greater associated count than the n-gram, discard the n-gram, and
if there is no partially overlapping n-gram in the set of candidate n-grams that has the greater associated count than the n-gram, incrementing the associated count of the n-gram in the set of candidate n-grams.
12. The system of claim 11 , further comprising computer-executable instructions that when executed by the at least one computing device cause the system to:
for each n-gram in the set of candidate n-grams, calculate a phrase score for the n-gram;
determine n-grams with a phrase score above a threshold phrase score as significant phrases.
13. A non-transitory computer-readable medium storing computer-executable instructions that when executed by at least one computing device cause a system to:
provide a generic language model;
provide a set of documents from a source;
iterate over the documents of the set of documents to subdivide the documents of the set of documents into a plurality of meaning units;
for each meaning unit, determine an associated count for each n-gram that occurs in the meaning unit;
for each meaning unit:
for each n-gram that occurs in the meaning unit:
compute a prominence score for the n-gram based on the determined count;
if the prominence score for the n-gram is below a prominence threshold, discard the n-gram; and
if the prominence score for the n-gram is not below the prominence threshold:
if the n-gram is not in a set of candidate n-grams, insert the n-gram into the set of candidate n-grams and set the associated count to 1;
if the n-gram is in the set of candidate n-grams, increment the associated count; and
if the n-gram is in the set of candidate n-grams and the associated count is greater than 1:
computing a stickiness score for the n-gram; and
if the stickiness score for the n-gram is below a stickiness threshold, discard the n-gram from the set of candidate n-grams;
based on the associated count for each n-gram in the set of candidate n-grams, estimate a language-model probability for each n-gram; and
obtain a source-specific language model based on the estimated language-model probabilities for each n-gram;
load the source-specific language model into a memory;
receive a plurality of transcribed interactions;
tag the plurality of transcribed interactions using the loaded source-specific language model; and
generate analytics based on the tagged plurality of transcribed interactions.
14. The computer-readable medium of claim 13 , wherein the meaning units are sentences.
15. The computer-readable medium of claim 13 , wherein the documents are subdivided into meaning using a zoning algorithm.
16. The computer-readable medium of claim 13 , wherein determining the associated count for each n-gram that occurs in the meaning unit comprises:
determining a count for each unigram that occurs in the meaning unit;
determining a count for each bigram that occurs in the meaning unit; and
determining a count for each trigram that occurs in the meaning unit.
17. The computer-readable medium of claim 13 , further comprising computer-executable instructions that when executed by the at least one computing device cause the system to:
if the stickiness score for the n-gram is not below the stickiness threshold:
if there is a partially overlapping n-gram in the set of candidate n-grams that has a greater count than the n-gram, discard the n-gram, and
if there is no partially overlapping n-gram in the set of candidate n-grams that has the greater count than the n-gram, increment the count associated with the n-gram in the set of candidate n-grams.
18. The computer-readable medium of claim 17 , further comprising:
for each n-gram in the set of candidate n-grams, calculate a phrase score for the n-gram;
determine n-grams with a phrase score above a threshold phrase score as significant phrases.