Automated scalable contextual data collection and extraction system
A system for contextual data collection and extraction is provided, comprising an extraction engine configured to receive context from a user for desired information to extract, connect to a data source providing a richly formatted dataset, retrieve the richly formatted dataset, process the richly formatted dataset and extract information from a plurality of linguistic modalities within the richly formatted, and transform the extracted data into a extracted dataset; and a knowledge base construction service configured to retrieve the extracted dataset, create a knowledge base for storing the extracted dataset, and store the knowledge base in a data store.
1. A system for automated scalable contextual data collection and extraction, comprising:
a computing device comprising a memory, a processor, and a non-volatile data storage device;
an extraction engine comprising a first plurality of programming instructions stored in the memory and operable on the processor, wherein the first plurality of programming instructions, when operating on the processor, cause the computing device to:
receive a formatted dataset for analysis;
receive a context in which to analyze the formatted dataset;
extract structural information from the formatted dataset
associate the structural information with the context; and
store the formatted dataset, the context, the structural information, and the associations in the non-volatile data storage device; and
a knowledge base construction service comprising a second plurality of programming instructions stored in the memory and operable on the processor, wherein the second plurality of programming instructions, when operating on the processor, cause the computing device to:
retrieve the retrieve the formatted dataset, the context, the structural information, and the associations from the non-volatile data storage device;
create a knowledge graph comprising relationships among the formatted dataset, the context, the structural information, and the associations; and
store the knowledge graph in the non-volatile data storage device.
2. The system of claim 1 , wherein a previously created knowledge graph is retrieved to store newly extracted information.
3. The system of claim 1 , wherein data extracted from a first modality is be used to augment data in a second modality.
4. The system of claim 1 , further comprising a proxy connection service comprising a third plurality of programming instructions stored in the memory and operable on the processor, wherein the third plurality of programming instructions, when operating on the processor, cause the computing device to:
determine that an intermediate proxy connection is required based at least in part on connection status in connecting to a data source.
5. The system of claim 4 , wherein the proxy connection service determines an optimal proxy network to use as the intermediate proxy connection based at least on successfully connecting to a target data source.
6. The system of claim 1 , wherein a data marker in labeled data is used by the system to identify and label previously unlabeled similar data.
7. The system of claim 1 , further comprising a phase transition analyzer comprising a fourth plurality of programming instructions stored in the memory and operable on the processor, wherein the fourth plurality of programming instructions, when operating on the processor, cause the computing device to:
retrieve a knowledge graph from the data store; and
perform a plurality of graph analysis and transformations on the knowledge graph to identify data transitions over time.
8. The system of claim 1 , wherein a previously created knowledge graph is monitored for unwanted data exfiltration.
9. A method for automated scalable contextual data collection and extraction, comprising the steps of:
receiving a formatted dataset for analysis;
receiving a context in which to analyze the formatted dataset;
extracting structural information from the formatted dataset; and
associating the structural information with the context; and
storing the formatted dataset, the context, the structural information, and the associations in a non-volatile data storage device;
creating a knowledge graph comprising relationships among the formatted dataset, the context, the structural information, and the associations; and
storing the knowledge graph in the non-volatile data storage device.
10. The method of claim 9 , wherein a previously created knowledge graph is retrieved to store newly extracted information.
11. The method of claim 9 , wherein data extracted from a first modality is be used to augment data in a second modality.
12. The method of claim 9 , further comprising the step of determining that an intermediate proxy connection is required based at least in part on connection status in connecting to a data source.
13. The method of claim 12 , further comprising the step of determining an optimal proxy network to use as the intermediate proxy connection based at least on successfully connecting to a target data source.
14. The method of claim 9 , further comprising the step of labeling data used by the system to identify and label previously unlabeled similar data.
15. The method of claim 9 , further comprising the steps of:
retrieving a knowledge graph from the data store; and
performing a plurality of graph analysis and transformations on the knowledge graph to identify data transitions over time.
16. The method of claim 9 , further comprising the step of monitoring a previously created knowledge graph for unwanted data exfiltration.