Systems and methods for augmenting rare disease dictionaries
Comprehensive and high-quality disease dictionaries are invaluable resources for tasks such as building ontologies, automated relation extraction, text summarization, question answering etc. Such curated resources are useful to clinicians, researchers, and various Biomedical Natural Language Processing tasks. However, these are manually curated and are labor and time intensive, and additionally suffer from lower recall and coverage is also less. Present disclosure provides systems and methods for augmenting rare disease dictionaries, wherein the system retrieves (new) rare diseases terms from medical literature that are related to the given dictionary terms (seed terms) and recommends new terms (or NPs) in a ranked order. This method is useful for rare diseases dictionary augmentation as a significant fraction of the top recommendations are new synonym candidates for dictionary augmentation. The method uses syntactic and semantic similarity measures in combination with efficient nearest neighbor search for efficient retrieval.
1 . A processor implemented method, further comprising:
obtaining, via one or more hardware processors, one or more medical literatures as an input;
extracting, by using at least one of a pattern-based noun phrase (NP) chunking and a n-gram-based NP chunking via the one or more hardware processors, a plurality of NPs from one or more medical literatures;
identifying, via the one or more hardware processors, a plurality of relevant NPs from the plurality of NPs based on a comparison of the plurality of NPs with one or more seed terms comprised in a seed dictionary, wherein the seed dictionary comprises a plurality of records pertaining to one or more rare diseases, and wherein each record from the plurality of records comprises an identifier (ID), a rare disease name, and one or more associated synonyms;
performing, via the one or more hardware processors, a semantic analysis on the plurality of relevant NPs to identify one or more candidate terms for each seed term comprised in the seed dictionary; and
applying, via the one or more hardware processors, one or more filtering techniques on the one or more candidate terms to obtain one or more augmentation terms.
2 . The processor implemented method of claim 1 , wherein the pattern-based noun phrase chunking comprises a set of pre-defined patterns, and the n-gram-based noun phrase chunking comprises at least one of a pre-trained unigram, a pre-trained bigram and a pre-trained trigram machine learning (ML) model on a pre-defined corpus.
3 . The processor implemented method of claim 1 , wherein the comparison comprises performing a syntactic overlapping of the plurality of NPs with the seed dictionary.
4 . The processor implemented method of claim 1 , wherein the one or more relevant NPs have at least a pre-defined syntactic overlap with the seed dictionary.
5 . The processor implemented method of claim 1 , wherein the step of performing the semantic analysis on the plurality of relevant NPs to identify the one or more candidate terms for each seed term comprised in the seed dictionary comprises:
obtaining word2vec vectors for each relevant NP from the plurality of relevant NPs and each seed term comprised in the seed dictionary, by using a word2vec embedding technique;
computing, for each relevant NP, one or more k-nearest neighbors from each seed term comprised in the seed dictionary; and
applying a cosine similarity function and a Euclidean distance to the one or more k-nearest neighbors from each relevant NP amongst the plurality of relevant NPs to identify the one or more candidate terms.
6 . The processor implemented method of claim 1 , wherein the one or more filtering techniques comprise at least one of an inflection technique, a plural form identification technique, a difference technique, and a special character identification technique.
7 . The processor implemented method of claim 1 , wherein the step of obtaining, via the one or more hardware processors, one or more medical literatures as the input is preceded by:
selecting, via the one or more hardware processors, a pre-defined rare disease database from a plurality of rare disease databases; and
generating, via the one or more hardware processors, the seed dictionary for the one or more rare diseases, using the pre-defined rare disease database.
8 . The processor implemented method of claim 1 , further comprising augmenting the seed dictionary with the one or more augmentation terms associated with one or more rare diseases terms.
9 . A system, comprising:
a memory storing instructions;
one or more communication interfaces; and
one or more hardware processors coupled to the memory via the one or more communication interfaces, wherein the one or more hardware processors are configured by the instructions to:
obtain one or more medical literatures as an input;
extract, by using at least one of a pattern-based NP chunking and a n-gram-based NP chunking, a plurality of NPs from one or more medical literatures;
identify a plurality of relevant NPs from the plurality of NPs based on a comparison of the plurality of NPs with one or more seed terms comprised in a seed dictionary, wherein the seed dictionary comprises a plurality of records pertaining to one or more rare diseases, and wherein each record from the plurality of records comprises an identifier (ID), a rare disease name, and one or more associated synonyms;
perform a semantic analysis on the plurality of relevant NPs to identify one or more candidate terms for each seed term comprised in the seed dictionary; and
apply one or more filtering techniques on the one or more candidate terms to obtain one or more augmentation terms.
10 . The system of claim 9 , wherein the pattern-based noun phrase chunking comprises a set of pre-defined patterns, and the n-gram-based noun phrase chunking comprises at least one of a pre-trained unigram, a pre-trained bigram and a pre-trained trigram machine learning (ML) model on a pre-defined corpus.
11 . The system of claim 9 , wherein the comparison comprises performing a syntactic overlapping of the plurality of NPs with the seed dictionary.
12 . The system of claim 9 , wherein the one or more relevant NPs have at least a pre-defined syntactic overlap with the seed dictionary.
13 . The system of claim 9 , wherein the semantic analysis comprises:
obtaining word2vec vectors for each relevant NP from the plurality of relevant NPs and each seed term comprised in the seed dictionary, by using a word2vec embedding technique;
computing, for each relevant NP, one or more k-nearest neighbors from each seed term comprised in the seed dictionary; and
applying a cosine similarity function and a Euclidean distance to the one or more k-nearest neighbors from each relevant NP amongst the plurality of relevant NPs to identify the one or more candidate terms.
14 . The system of claim 9 , wherein the one or more filtering techniques comprise at least one of an inflection technique, a plural form identification technique, a difference technique, and a special character identification technique.
15 . The system of claim 9 , wherein prior to obtaining the one or more medical literatures the one or more hardware processors is configured to:
select a pre-defined rare disease database from a plurality of rare disease databases; and
generate the seed dictionary for the one or more rare diseases, using the pre-defined rare disease database.
16 . The system of claim 9 , wherein the one or more hardware processors are further configured by the instructions to augment the seed dictionary with the one or more augmentation terms associated with one or more rare diseases terms.
17 . One or more non-transitory machine-readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors cause:
obtaining one or more medical literatures as an input;
extracting, by using at least one of a pattern-based noun phrase (NP) chunking and a n-gram-based NP chunking, a plurality of NPs from one or more medical literatures;
identifying a plurality of relevant NPs from the plurality of NPs based on a comparison of the plurality of NPs with one or more seed terms comprised in a seed dictionary, wherein the seed dictionary comprises a plurality of records pertaining to one or more rare diseases, and wherein each record from the plurality of records comprises an identifier (ID), a rare disease name, and one or more associated synonyms;
performing a semantic analysis on the plurality of relevant NPs to identify one or more candidate terms for each seed term comprised in the seed dictionary; and
applying one or more filtering techniques on the one or more candidate terms to obtain one or more augmentation terms.
18 . The one or more non-transitory machine-readable information storage mediums of claim 17 , wherein the pattern-based noun phrase chunking comprises a set of pre-defined patterns, and the n-gram-based noun phrase chunking comprises at least one of a pre-trained unigram, a pre-trained bigram and a pre-trained trigram machine learning (ML) model on a pre-defined corpus,
wherein the comparison comprises performing a syntactic overlapping of the plurality of NPs with the seed dictionary,
wherein the one or more relevant NPs have at least a pre-defined syntactic overlap with the seed dictionary,
wherein the step of performing the semantic analysis on the plurality of relevant NPs to identify the one or more candidate terms for each seed term comprised in the seed dictionary comprises:
obtaining word2vec vectors for each relevant NP from the plurality of relevant NPs and each seed term comprised in the seed dictionary, by using a word2vec embedding technique;
computing, for each relevant NP, one or more k-nearest neighbors from each seed term comprised in the seed dictionary; and
applying a cosine similarity function and a Euclidean distance to the one or more k-nearest neighbors from each relevant NP amongst the plurality of relevant NPs to identify the one or more candidate terms, and
wherein the one or more filtering techniques comprise at least one of an inflection technique, a plural form identification technique, a difference technique, and a special character identification technique.
19 . The one or more non-transitory machine-readable information storage mediums of claim 17 , wherein the step of obtaining one or more medical literatures as the input is preceded by:
selecting a pre-defined rare disease database from a plurality of rare disease databases; and
generating the seed dictionary for the one or more rare diseases using the pre-defined rare disease database.
20 . The one or more non-transitory machine-readable information storage mediums of claim 17 , wherein the one or more instructions which when executed by the one or more hardware processors further cause augmenting the seed dictionary with the one or more augmentation terms associated with one or more rare diseases terms.