Method for calculating entity similarities
View Patent ↗One embodiment of the present invention provides a system for estimating a similarity level between semantic entities. During operation, the system selects two or more semantic entities associated with a number documents. The system subsequently parses the documents into sub-parts, and calculates the similarity level between the semantic entities based on occurrences of the semantic entities within the sub-parts of the documents.
1. A computer-executable method for estimating a similarity level between semantic entities, the method comprising:
extracting, by a computing device, two or more semantic entities from a set of documents, wherein a semantic entity includes a meaningful character sequence;
parsing a respective document to partition the respective document into a plurality of sub-parts; and
calculating the similarity level between a first extracted semantic entity and a second extracted entity based on a co-occurrence of the first and second entities in the plurality of document sub-parts, wherein calculating the similarity level involves calculating:
sim
subpart
(
first_entity
,
second_entitiy
)
=
idf
first_entity
idf
second_entity
·
co
subpart
(
first_entity
,
second_entity
)
occ
subpart
(
first_entity
)
,
wherein idf first — entity and idf second — entity are inverse-document-frequency (IDF) values of the first and second entities, respectively,
wherein co subpart (first_entity, second_entity) indicates a number of document subparts within which the first and second entities co-occur, and
wherein occ subpart (first_entity) indicates a number of document subparts within which the first entity occurs.
2. The method of claim 1 , wherein a respective document subpart includes an individual sentence and/or an individual paragraph.
3. The method of claim 2 , wherein calculating the similarity level involves determining sentence-based and/or paragraph-based co-occurrence frequencies of the semantic entities.
4. The method of claim 3 , wherein the sentence-based co-occurrence frequency of the semantic entities is calculated based on a total number of sentences within which the semantic entities co-occur, and wherein the paragraph-based co-occurrence frequency of the semantic entities is calculated based on a total number of paragraphs within which the semantic entities co-occur.
5. The method of claim 3 , further comprising calculating weighted IDF values of the first and second entities.
6. The method of claim 1 , wherein the documents include an email message and/or an email conversation.
7. The method of claim 6 , wherein the semantic entities include names of people, and wherein calculating the similarity level between the names of people involves determining a co-occurrence frequency of the names of people within sender and/or recipient lists of the email message.
8. The method of claim 1 , wherein the semantic entities include at least one of:
people's names;
companies' names;
industry-specific terms;
dates and times;
street addresses;
email addresses;
uniform resource locators (URLs); and
telephone numbers.
9. The method of claim 1 , wherein calculating the similarity level involves determining a weight function for at least one document.
10. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for estimating a similarity level between semantic entities, the method comprising:
extracting two or more semantic entities from a set of documents;
parsing a respective document to partition the respective document into a plurality of sub-parts; and
calculating the similarity level between a first extracted semantic entity and a second extracted entity based on a co-occurrence of the first and second entities in the plurality of document sub-parts, wherein calculating the similarity level involves calculating:
sim
subpart
(
first_entity
,
second_entitiy
)
=
idf
first_entity
idf
second_entity
·
co
subpart
(
first_entity
,
second_entity
)
occ
subpart
(
first_entity
)
,
wherein idf first — entity and idf second — entity are inverse-document-frequency (IDF) values of the first and second entities, respectively,
wherein co subpart (first_entity, second_entity) indicates a number of document subparts within which the first and second entities co-occur, and
wherein occ subpart (first_entity) indicates a number of document subparts within which the first entity occurs.
11. The computer-readable storage medium of claim 10 , wherein a respective document subpart includes an individual sentence and/or an individual paragraph.
12. The computer-readable storage medium of claim 11 , wherein calculating the similarity level involves determining sentence-based and/or paragraph-based co-occurrence frequencies of the semantic entities.
13. The computer-readable storage medium of claim 12 , wherein calculating the similarity level of a first semantic entity in relation to a second semantic entity involves calculating a ratio of a co-occurrence frequency of the first and second semantic entities to an occurrence frequency of the second semantic entity.
14. The computer-readable storage medium of claim 12 , wherein the method further comprises calculating weighted IDF values of the first and second entities.
15. The computer-readable storage medium of claim 10 , wherein the documents include an email message and/or an email conversation.
16. The computer-readable storage medium of claim 15 , wherein the semantic entities include names of people, and wherein calculating the similarity level between the names of people involves determining a co-occurrence of the names of people within sender and/or recipient lists of the email message.
17. The computer-readable storage medium of claim 10 , wherein the semantic entities include at least one of:
people's names;
companies' names;
industry-specific terms;
dates and times;
street addresses;
email addresses;
uniform resource locators (URLs); and
telephone numbers.
18. The computer-readable storage medium of claim 10 , wherein calculating the similarity level involves determining a weight function for at least one document.
19. A system for estimating a similarity level between semantic entities, comprising:
a processor;
a memory;
an extracting mechanism configured to extract two or more semantic entities associated with a set of documents;
a parsing mechanism configured to parse a respective document to partition the respective document into a plurality of sub-parts; and
an entity-similarity-calculation mechanism configured to calculate the similarity level between a first extracted semantic entity and a second extracted entity based on a co-occurrence of the first and second entities in the plurality of document sub-parts, wherein while calculating the similarity level, the entity-similarity-calculation mechanism is configured to calculate:
sim
subpart
(
first_entity
,
second_entitiy
)
=
idf
first_entity
idf
second_entity
·
co
subpart
(
first_entity
,
second_entity
)
occ
subpart
(
first_entity
)
,
wherein idf first — entity and idf second — entity are inverse-document-frequency (IDF) values of the first and second entities, respectively,
wherein co subpart (first_entity, second_entity) indicates a number of document subparts within which the first and second entities co-occur, and
wherein occ subpart (first_entity) indicates a number of document subparts within which the first entity occurs.
20. The system of claim 19 , wherein a respective document subpart includes an individual sentence and/or an individual paragraph.
21. The system of claim 20 , wherein calculating the similarity level involves determining sentence-based and/or paragraph-based co-occurrence frequencies of the semantic entities.
22. The system of claim 21 , wherein the sentence-based co-occurrence frequency of the semantic entities is calculated based on a total number of sentences within which the semantic entities co-occur, and wherein the paragraph-based co-occurrence frequency of the semantic entities is calculated based on a total number of paragraphs within which the semantic entities co-occur.