Automated scalable contextual data collection and extraction system
A system for contextual data collection and extraction is provided, comprising an extraction engine configured to receive context from a user for desired information to extract, connect to a data source providing a richly formatted dataset, retrieve the richly formatted dataset, process the richly formatted dataset and extract information from a plurality of linguistic modalities within the richly formatted, and transform the extracted data into a extracted dataset; and a knowledge base construction service configured to retrieve the extracted dataset, create a knowledge base for storing the extracted dataset, and store the knowledge base in a data store.
1. A system for automated scalable contextual data collection and extraction, comprising:
an extraction engine comprising a memory, a processor, and a plurality of programming instructions stored in the memory thereof and operable on the processor thereof, wherein the programmable instructions, when operating on the processor, cause the processor to:
receive a context target from a user for desired information to extract;
connect to a data source providing a richly formatted dataset;
retrieve the richly formatted dataset;
process the richly formatted dataset;
extract information from a plurality of linguistic modalities within the richly formatted dataset relating at least in part to a context provided by the user; and
transform the extracted information into a graph and time series-based dataset; and
a knowledge base construction service comprising a memory, a processor, and a plurality of programming instructions stored in the memory thereof and operable on the processor thereof, wherein the programmable instructions, when operating on the processor, cause the processor to:
retrieve the graph and time series-based dataset;
create a knowledge base for storing the graph and time series-based dataset; and
store the knowledge base in a data store for later reference.
2. The system of claim 1 , wherein a previously created knowledge base is retrieved to store newly extracted information.
3. The system of claim 1 , wherein data extracted from a first modality is be used to augment data in a second modality.
4. The system of claim 1 , further comprising a proxy connection service comprising a memory, a processor, and a plurality of programming instructions stored in the memory thereof and operable on the processor thereof, wherein the programmable instructions, when operating on the processor, cause the processor to:
determine that an intermediate proxy connection is required based at least in part on connection status in connecting to a data source.
5. The system of claim 4 , wherein the proxy connection service determines an optimal proxy network to use as the intermediate proxy connection based at least on successfully connecting to a target data source.
6. The system of claim 1 , wherein a data marker in labeled data is used by the system to identify and label previously unlabeled similar data.
7. The system of claim 1 , further comprising a phase transition analyzer comprising a proxy connection service comprising a memory, a processor, and a plurality of programming instructions stored in the memory thereof and operable on the processor thereof, wherein the programmable instructions, when operating on the processor, cause the processor to:
retrieve a knowledge base from the data store; and
perform a plurality of graph analysis and transformations on the knowledge base to identify data transitions over time.
8. The system of claim 1 , wherein a previously created knowledge base is monitored for unwanted data exfiltration.
9. A method for automated scalable contextual data collection and extraction, comprising the steps of:
(a) receiving a context target from a user for desired information to extract, using an extraction engine;
(b) connecting to a data source providing a richly formatted dataset, using the extraction engine;
(c) retrieving the richly formatted dataset, using the extraction engine;
(d) processing the richly formatted dataset and extracting information from a plurality of linguistic modalities within the richly formatted dataset relating at least in part to the context target provided by the user, using the extraction engine;
(e) transforming the extracted information into a graph and time series-based dataset, using the extraction engine;
(f) retrieving the graph and time series-based dataset, using a knowledge base construction service;
(g) creating a knowledge base for storing the graph and time series-based dataset, using the knowledge base construction service; and
(h) storing the knowledge base in a data store for later reference, using the knowledge base construction service.
10. The method of claim 9 , wherein a previously created knowledge base is retrieved to store newly extracted information.
11. The method of claim 9 , wherein data extracted from a first modality is be used to augment data in a second modality.
12. The method of claim 9 , further comprising a proxy connection service comprising a memory, a processor, and a plurality of programming instructions stored in the memory thereof and operable on the processor thereof, wherein the programmable instructions, when operating on the processor, cause the processor to:
determine that an intermediate proxy connection is required based at least in part on connection status in connecting to a data source.
13. The method of claim 12 , wherein the proxy connection service determines an optimal proxy network to use as the intermediate proxy connection based at least on successfully connecting to a target data source.
14. The method of claim 9 , wherein a data marker in labeled data is used by the system to identify and label previously unlabeled similar data.
15. The method of claim 9 , further comprising a phase transition analyzer comprising a proxy connection service comprising a memory, a processor, and a plurality of programming instructions stored in the memory thereof and operable on the processor thereof, wherein the programmable instructions, when operating on the processor, cause the processor to:
retrieve a knowledge base from the data store; and
perform a plurality of graph analysis and transformations on the knowledge base to identify data transitions over time.
16. The method of claim 9 , wherein a previously created knowledge base is monitored for unwanted data exfiltration.