Metadata extraction from big data sources
An example system for programmatically extracting data from a big data source includes: a processor; and system memory encoding instructions which, when executed by the processor, cause the system to: extract metadata from the big data source using the utility; transform the metadata into a standard format; and load the metadata in a repository.
1. A system for programmatically extracting data from a big data source, the system comprising:
a processor; and
system memory encoding instructions which, when executed by the processor, cause the system to:
invoke a connector through an application programming interface to access the big data source;
extract metadata from the big data source using the connector by systematically querying the big data source to perform multiple passes, including to:
identify indexes in a first pass;
identify source types in a second pass; and
identify fields in a third pass;
transform the metadata into a standard format; and
load the metadata in a repository.
2. The system of claim 1 , wherein the big data source comprises a Splunk big data source.
3. The system of claim 2 , wherein the system memory encodes further instructions which, when executed by the processor, cause the system to use an application programming interface of the Splunk big data source to access the metadata.
4. The system of claim 1 , wherein the system memory encodes further instructions which, when executed by the processor, cause the system to perform multiple passes to extract the metadata.
5. The system of claim 1 , wherein the system memory encodes further instructions which, when executed by the processor, cause the system to use a transport mechanism to transfer data from the big data source.
6. The system of claim 1 , wherein the system memory encodes further instructions which, when executed by the processor, cause the system to use curl to transfer data from the big data source.
7. The system of claim 1 , wherein the system memory encodes further instructions which, when executed by the processor, cause the system to create a change set in the repository.
8. The system of claim 7 , wherein the system memory encodes further instructions which, when executed by the processor, cause the system to publish the metadata in the repository once the change set is approved.
9. A method for programmatically extracting data from a big data source, the method comprising:
invoking a connector through an application programming interface to access the big data source;
extracting metadata from the big data source using the connector by systematically querying the big data source to perform multiple passes, including:
identifying indexes in a first pass;
identifying source types in a second pass; and
identifying fields in a third pass;
transforming the metadata into a standard format; and
loading the metadata in a repository.
10. The method of claim 9 , wherein the big data source comprises a Splunk big data source.