Metadata extraction from big data sources
An example system for programmatically extracting data from a big data source includes: a processor; and system memory encoding instructions which, when executed by the processor, cause the system to: extract metadata from the big data source using the utility; transform the metadata into a standard format; and load the metadata in a repository.
1. A system for programmatically extracting data from a big data source, the system comprising:
a processor; and
memory encoding instructions which, when executed by the processor, cause the system to:
invoke a utility to access the big data source;
extract metadata from the big data source using the utility;
transform the metadata into a standard format by making changes to the metadata;
load the metadata in a repository;
create a change set in the repository, the change set being a temporary location that records the metadata and the changes; and
publish the metadata in the repository once the change set, including the changes, is approved.
2. The system of claim 1 , comprising further instructions which, when executed by the processor, cause the system to run a script to execute the utility.
3. The system of claim 2 , wherein the script is programmed to determine one or more of a data store name, a data store size, and a definition of a sampling clause.
4. The system of claim 1 , wherein the extract is through an application programming interface for the big data source.
5. The system of claim 1 , wherein the extract includes multiple passes.
6. The system of claim 1 , comprising further instructions which, when executed by the processor, cause the system to use a transport mechanism to transfer the data from the big data source.
7. The system of claim 1 , wherein the transform includes to move the metadata to the standard format.
8. The system of claim 1 , wherein the big data source is unstructured or semi-structured.
9. The system of claim 1 , wherein the big data source comprises a MongoDB big data source.
10. A method for programmatically extracting data from a big data source, the method comprising:
invoking, by a computing device, a utility to access the big data source;
extracting, by the computing device, metadata from the big data source using the utility;
transforming, by the computing device, the metadata into a standard format by making changes to the metadata;
loading, by the computing device, the metadata in a repository;
creating, by the computing device, a change set in the repository, the change set being a temporary location that records the metadata and the changes; and
publishing the metadata in the repository once the change set, including the changes, is approved.
11. The method of claim 10 , further comprising running a script to execute the utility.
12. The method of claim 11 , wherein the script is programmed to determine one or more of a data store name, a data store size, and a definition of a sampling clause.
13. The method of claim 10 , wherein the extracting is through an application programming interface for the big data source.
14. The method of claim 10 , wherein the extracting includes multiple passes.
15. The method of claim 10 , further comprising using a transport mechanism to transfer data from the big data source.
16. The method of claim 10 , wherein the transforming includes moving the metadata to the standard format.
17. The method of claim 10 , wherein the big data source is unstructured or semi-structured.
18. The method of claim 10 , wherein the big data source comprises a MongoDB big data source.