System and method for detecting drug adverse effects in social media and mobile applications data
Some implementations provide a computer-implemented method for identifying, from on-line postings, reports of potential adverse effects resulting from consuming a healthcare product, the method including: receiving a log of on-line postings regarding consuming the healthcare product; receiving a database comprising a healthcare taxonomy and a set of linguistic rules; analyzing, based on the healthcare taxonomy, the log of on-line postings to identify a report of at least one adverse effect resulting from consuming the healthcare product; generating a score for the identified report according to the healthcare taxonomy and the set of linguistic rules; comparing the generated score with a threshold; and in response to determining that the generated score is above the threshold, flagging the identified report as a report of a potential adverse effect.
1. A computer-implemented method, comprising:
assigning, by a computer processor, weights to words of a healthcare taxonomy;
consolidating, by a reader interface, postings of multiple online forums into a log of postings;
filtering, by a criterion setting engine, the log of postings based on a search parameter to identify a subset of the postings that relate to an adverse effect resulting from consuming a healthcare product;
identifying, by the computer processor, a portion of the words of the healthcare taxonomy in the log of postings;
generating, by a text mining engine, a relevance score for each of the postings of the subset of postings based on linguistic rules and the weights assigned to the words of the healthcare taxonomy identified in the respective postings;
selecting, by the computer processor, postings for which the relevance scores meet a threshold;
receiving, by the computer processor, feedback regarding precision of the relevance scores of the selected postings;
adjusting, by the computer processor, the weights assigned to the words of the healthcare taxonomy identified in the selected postings in accordance with the feedback; and
repeating the consolidating, the filtering and the generating based on the adjusted weights;
wherein the consolidating comprises,
sending requests to multiple search engines and/or databases by a metasearch engine of the reader interface, and
aggregating results of the requests into the log of postings by a feed aggregator of the reader interface.
2. The method of claim 1 , wherein the generating includes text mining the log of postings based on natural language processing using the linguistic rules.
3. The method of claim 2 , wherein the filtering comprises searching the log of postings for an identifiable consumer of the healthcare product, an identifiable reporter of a posting, a reference to the adverse effect resulting from consuming the healthcare product, and/or a reference to the healthcare product.
4. The method of claim 2 , further comprising identifying, with a typographic correction engine, typographic errors in the log of postings based on a discussion context.
5. The method of claim 4 , further comprising correcting the typographic errors with the typographic correction engine according to the discussion context, a special meaning in a context of the adverse effect resulting from consuming the healthcare product, and/or a meaning in a context of the healthcare product.
6. The method of claim 1 , wherein the assigning comprises assigning the weights based on contributions from a proximity of keywords, a discussion context, an amount of detail with regard to the adverse effect resulting from consuming the healthcare product, and a relevancy degree with regard to the healthcare product, as defined in the linguistic rules.
7. The method of claim 1 , wherein the assigning comprises weighing previous manual inspections for adverse effects comparable to the adverse effect resulting from consuming the healthcare product or comparable healthcare product.
8. The method of claim 1 , wherein the postings comprise on-line postings from two or more of a social networking web-site, an on-line discussion forum, an on-line app-store, and a blogging web-site.
9. The method of claim 1 , further comprising alerting, by the computer processor, a user of the selected postings.
10. The method of claim 9 , wherein the alerting comprises sending an email alert, a short message service message, and/or a voicemail.
11. The method of claim 9 , further comprising, by the computer processor:
detecting and consolidating duplicate postings in the log of postings.
12. A system, comprising:
a computer processor configured to assign weights to words of a healthcare taxonomy;
a reader interface configured to consolidate postings of multiple online forums into a log of postings;
a criterion setting engine configured to filter the log of postings based on a search parameter to identify a subset of the postings that relate to an adverse effect resulting from consuming a healthcare product;
the computer processor further configured to identify a portion of the words of the healthcare taxonomy in the log of postings;
a text mining engine configured to generate a relevance score for each of the postings of the subset of postings based on linguistic rules and the weights assigned to words of the healthcare taxonomy identified in the respective postings;
wherein the computer processor is further configured to,
select postings for which the generated relevance scores meet a predefined threshold,
receive feedback regarding precision of the relevance scores of the selected postings, and
adjust the weights assigned to the words of the healthcare taxonomy identified in the selected postings in accordance with the feedback; and
wherein the system is further configured to identify subsequent online postings that relate to the adverse effect resulting from consuming the healthcare product based on the adjusted weights assigned to the words of the healthcare taxonomy; and
wherein the reader interface comprises,
a metasearch engine configured to send requests to multiple search engines and/or databases, and
a feed aggregator to aggregate results of the requests into the log of postings.
13. The system of claim 12 , wherein the text mining engine is further configured to text mine the log of postings based on natural language processing using the linguistic rules.
14. The system of claim 13 , wherein the criterion setting engine is further configured to search the log of postings for an identifiable consumer of the healthcare product, an identifiable reporter of a posting, a reference to the adverse effect resulting from consuming the healthcare product, and/or a reference to the healthcare product.
15. The system of claim 13 , further comprising a typographic correction engine configured to identify typographic errors in the log of postings based on a discussion context.
16. The system of claim 15 , wherein the typographic correction engine is further configured to correct the typographic errors according to the discussion context, a special meaning in a context of the adverse effect resulting from consuming the healthcare product, and/or a meaning in a context of the healthcare product.
17. The system of claim 12 , wherein the computer processor is further configured to assign the weights based on contributions from a proximity of keywords, a discussion context, an amount of detail with regard to the adverse effect resulting from consuming the healthcare product, and a relevancy degree with regard to the healthcare product, as defined in the linguistic rules.
18. The system of claim 12 , wherein the computer processor is further configured to weigh previous manual inspections for adverse effects comparable to the adverse effect resulting from consuming the healthcare product or comparable healthcare product when assigning the weights to the words of the healthcare taxonomy.
19. The system of claim 12 , wherein the postings comprise on-line postings from two or more of a social networking web-site, an on-line discussion forum, an on-line app-store, and a blogging web-site.
20. The system of claim 12 , wherein the computer processor is further configured to alert a user of the selected postings.
21. The system of claim 20 , wherein the computer processor is further configured to send an email alert, a short message service (SMS) message, or a voicemail to alert the user of the selected postings.
22. The system of claim 20 , wherein the computer processor is further configured to:
detect and consolidate duplicate postings in the log of postings.
23. The method of claim 1 , wherein the adjusting comprises:
performing an adaptive learning process by the computer processor.
24. The method of claim 23 , wherein the adjusting further comprises:
using a genetic algorithm to refine the weights to reduce differences between the relevance scores generated by the text mining engine and scores received in the feedback.
25. The method of claim 1 , wherein the consolidating comprises:
retrieving postings from multiple web feeds with respective feed readers of the reader interface; and
aggregating the multiple web feeds into the log of postings by the feed aggregator of the reader interface.
26. The method of claim 1 , wherein the generating comprises:
generating the relevance scores by the text mining engine over a distributed network of computer systems according to load balancing tactics that factor in computing power of each computer system of the distributed network of computer systems, proximity of the computer systems to postings of the log, and/or a likelihood of hits when searching respective portions of the log of postings, efficiency, latency, and/or throughput.
27. The method of claim 1 , wherein the generating comprises:
generating the relevance scores by the text mining engine over a distributed network of computer systems according to load balancing tactics that factor in efficiency, latency, and/or throughput.
28. The system of claim 12 , wherein the computer processor is further configured to:
adjust the weights with an adaptive learning process.
29. The system of claim 28 , wherein the computer processor is further configured to:
adjust the weights with a genetic algorithm to reduce differences between the relevance scores generated by the text mining engine and scores received in the feedback.
30. The system of claim 12 , wherein the reader interface comprises:
multiple feed readers, each to retrieve postings from a respective one of multiple web feeds; and
the feed aggregator to aggregate the multiple web feeds into the log of postings.
31. The system of claim 12 , wherein the text mining engine is further configured to generate the relevance scores over a distributed network of computer systems according to load balancing tactics that factor in computing power of each computer system of the distributed network of computer systems, proximity of the computer systems to postings of the log, and/or a likelihood of hits when searching respective portions of the log of postings, efficiency, latency, and/or throughput.
32. The system of claim 12 , wherein the text mining engine is further configured to generate the relevance scores over a distributed network of computer systems according to load balancing tactics that factor in efficiency, latency, and/or throughput.