Configurable, streaming hybrid-analytics platform
An analytics platform for the extraction of structured observations from largely narrative sources using a hybrid approach of user configuration and machine learning is provided. The analytics platform collects and normalizes data from public and private sources and applies extractions to the data to create a world view of objects, traits, and relationships of interest and maintains that world view as data and/or extractions are updated. The platform is further configured to apply queries to the extracted world view for a variety of purposes including scoring objects for prioritized attention, generating notifications when specific conditions are met, providing data sets for exploratory analysis, and triggering the automatic collection of enhancing data from external sources.
1. A configurable, streaming hybrid-analytics platform comprising:
an extraction, translate, and load (ETL) module configured to collect one or more documents from one or more document sources;
an extraction engine configured to receive a document for extraction from the ETL module and to perform an extraction process, the extraction process including:
identifying a rule book to apply during extraction, wherein the rule book includes one or more text extraction rules, wherein each of the text extraction rules includes at least one match expression, and wherein the at least one match expression includes at least one pattern;
searching a pattern tree on a first set of text in the document to determine whether a pattern hit exists, wherein the pattern tree represents the at least one pattern included in the at least one match expression in a text extraction rule in the identified rule book, and wherein a hit indicates a match between the document text and a pattern in the pattern tree;
mapping each identified pattern hit to a rule in the one or more text extraction rules in the rule book to generate a set of mapped rules;
for each mapped rule, evaluating one or more predicates included in the mapped rule to determine whether a rule hit exists;
generating a citation for each identified rule hit;
determining whether additional rule books remain to apply during extraction;
if additional rule books remain to apply, identifying a next rule book and repeating the searching, mapping, evaluating, generating, and determining steps; and
if no additional rule books remain to apply, storing the generated citations in a citation database; and
a query engine configured to search for user-defined patterns or events.
2. The platform of claim 1 , wherein the extraction engine is further configured to translate text associated with a rule hit into objects, object traits, relationships, and relationship traits and generate an associated object, trait, and relationship (OTR) record.
3. The platform of claim 2 , further comprising: an OTR database configured to store OTR records.
4. The platform of claim 3 , wherein the extraction engine is further configured to associate a citation with an OTR record and a text field.
5. The platform of claim 4 , wherein the extraction engine is further configured to compare the OTR record against the OTR records in the OTR database to determine if the OTR record is a known OTR record.
6. The platform of claim 5 , wherein the extraction engine is further configured to add the OTR record to the OTR database if the OTR record is not already known.
7. The platform of claim 1 , wherein the ETL module is further configured to perform one or more publish actions.
8. The platform of claim 7 wherein the publish actions include reformatting the received document.
9. The platform of claim 1 further comprising: a user interface configured to accept updates to the configuration of the extraction engine.
10. The platform of claim 9 , wherein the configuration update includes an addition of an object type, an object trait type, or a relationship trait type.
11. The platform of claim 1 , wherein the ETL module, the extraction engine, and the query engine are included on a same computer.
12. The platform of claim 11 , wherein the computer comprises at least one processor and at least one memory coupled to the at least one processor and storing instructions for performing the ETL module and the extraction engine operations.
13. The platform of claim 1 , wherein as an option, the rule books and other rule books share a common pattern tree, and wherein the hits against patterns in the common pattern tree drive which rule book or rule books need to be applied.
14. A method comprising:
receiving, via one or more processors, a document for extraction; and
performing, via one or more processors, an extraction process by:
identifying a rule book to apply during extraction, wherein the rule book includes one or more text extraction rules, wherein each of the text extraction rules includes at least one match expression, and wherein the at least one match expression includes at least one pattern,
searching a pattern tree on document text in the document to determine whether a pattern hit exists, wherein the pattern tree represents the at least one pattern included in the at least one match expression in a text extraction rule in the identified rule book, and wherein a hit indicates a match between the document text and a pattern in the pattern tree;
mapping each identified pattern hit to a rule in the one or more text extraction rules in the rule book to generate a set of mapped rules;
for each mapped rule, evaluating one or more predicates included in the mapped rule to determine whether a rule hit exists;
generating a citation for each identified rule hit;
determining whether additional rule books remain to apply during extraction;
if additional rule books remain to apply, identifying a next rule book and repeating the searching, mapping, evaluating, generating, and determining steps; and
if no additional rule books remain to apply, storing the generated citations in a citation database.
15. The method of claim 14 , further comprising:
translating, via one or more processors, text associated with a rule hit into objects, object traits, relationships, and relationship traits and generate an associated object, trait, and relationship (OTR) record; and
associating, via one or more processors, a citation with an OTR record and a text field.
16. The method of claim 15 , further comprising:
comparing, via one or more processors, the OTR record against OTR records in a OTR database to determine if the OTR record is a known OTR record.
17. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause operations comprising:
receiving a document for extraction; and
performing an extraction process by:
identifying a rule book to apply during extraction, wherein the rule book includes one or more text extraction rules, wherein each of the text extraction rules includes at least one match expression, and wherein the at least one match expression includes at least one pattern;
searching a pattern tree on document text in the document to determine whether a pattern hit exists, wherein the pattern tree represents the at least one pattern included in the at least one match expression in a text extraction rule in the identified rule book, and wherein a hit indicates a match between the document text and a pattern in the pattern tree;
mapping each identified pattern hit to a rule in the one or more text extraction rules in the rule book to generate a set of mapped rules;
for each mapped rule, evaluating one or more predicates included in the mapped rule to determine whether a rule hit exists;
generating a citation for each identified rule hit;
determining whether additional rule books remain to apply during extraction;
if additional rule books remain to apply, identifying a next rule book and repeating the searching, mapping, evaluating, generating, and determining steps; and
if no additional rule books remain to apply, storing the generated citations in a citation database.
18. The computer-readable media of claim 17 , the operations further comprising:
translating text associated with a rule hit into objects, object traits, relationships, and relationship traits and generate an associated object, trait, and relationship (OTR) record; and
associating a citation with an OTR record and a text field.
19. The computer-readable media of claim 18 , the operations further comprising:
comparing the OTR record against OTR records in a OTR database to determine if the OTR record is a known OTR record.
20. The computer-readable media of claim 19 , the operations further comprising:
adding the OTR record to the OTR database if the OTR record is not already known.