Metadata-driven data ingestion
An electronic system for increasing the speed of preparing data with a specified data quality for storage by automatically identifying for a user, with minimal user input, common contexts among (i) fields in disparate datasets, and (ii) names the user has specified as potentially describing the fields, and by using those common contexts to govern the disparate datasets prior to storage to ensure the specified data quality.
1 . A method implemented by a data processing system of increasing the speed of preparing data with a specified data quality for storage by automatically identifying for a user, with minimal user input, common contexts among (I) fields in disparate datasets, and (ii) names the user has specified as potentially describing the fields, and by using those common contexts to govern the disparate datasets prior to storage to ensure the specified data quality, including:
retrieving, by a hardware storage device, datasets from external data sources;
identifying, by a data processing system from the retrieved datasets, items of metadata describing technical attributes that characterize fields in the retrieved datasets;
receiving, by a data processing system and from a client device, user-defined names that are candidates to describe the fields in the retrieved datasets;
selecting, by a data processing system, one or more of the user-defined names corresponding to one or more technical attributes that characterize a given field in the retrieved datasets, including:
selecting, from among the user-defined names, a user-defined name representing a semantic meaning of the one or more technical attributes that characterize the given field;
generating, by a data processing system, an association among
one or more items of metadata describing the one or more technical attributes that characterize the given field in the retrieved datasets, and
at least one of the one or more of the user-defined names selected, from among the candidates, that correspond to the one or more technical attributes that characterize the given field in the retrieved datasets;
accessing, by a data processing system, one or more rules that specify one or more operations, with the one or more operations referencing the at least one of the one or more of the user-defined names that correspond to the one or more technical attributes that characterize the given field in the retrieved datasets;
based on the identified association, detecting, by a data processing system and in a retrieved dataset, an item of metadata describing the one or more technical attributes that correspond to the at least one of the one or more of the user-defined names referenced in at least one of the one or more rules;
responsive to detecting, applying, by a data processing system, the at least one of the one or more rules to one or more data items in the retrieved dataset and described by the detected item of metadata; and
following application of the at least one of the one or more rules, storing, in one or more data stores, the one or more data items to which the at least one of the one or more rules are applied.
2 . The method of claim 1 , wherein at least one of the items of metadata includes a field name for a field of a retrieved dataset.
3 . The method of claim 2 , further including:
determining a label for at least one item of metadata, the label representing a semantic meaning for the at least one item of metadata;
identifying a match between the label for the at least one item of metadata and the user-defined name; and
responsive to the match, generating a link between the at least one item of metadata and the user-defined name.
4 . The method of claim 3 , wherein determining the label includes performing one or more semantic discovery processes on at least one of: the at least one item of metadata, or data items described by the at least one item of metadata.
5 . The method of claim 1 , wherein generating the association includes:
generating a link between at least one item of metadata and the user-defined name.
6 . The method of claim 3 , wherein generating the link between the at least one item of metadata and the user-defined name includes generating at least one data structure including data representing the at least one item of metadata and a pointer to data representing the user-defined name.
7 . The method of claim 3 , wherein generating the link between the at least one item of metadata and the user-defined name includes generating at least one data structure including data representing the user-defined name and a pointer to the at least one item of metadata.
8 . The method of claim 1 , including:
causing display of a graphical user interface at the client device, the graphical user interface displaying one or more visual representations of the user-defined names, and an input portion for defining the one or more rules with regard to at least one of the user-defined names; and
receiving data representing an input into the input portion of the graphical user interface, the data defining the one or more rules with regard to the at least one of the user-defined names.
9 . The method of claim 8 , wherein the graphical user interface is configured to display one or more visual representations of one or more of the items of metadata that are assigned to the user-defined names.
10 . The method of claim 1 , wherein the one or more rules include at least one of a personally identifiable information (PII) rule, or a data quality rule.
11 . The method of claim 1 , further including detecting, in a retrieved dataset, an item of metadata that is associated with a user-defined name referenced in at least one of the one or more rules by:
identifying at least one field name included in the retrieved dataset;
comparing the at least one field name with the items of metadata to identify a match between the at least one field name and the item of metadata;
comparing the item of metadata with stored linkage information to identify the user-defined name that is assigned to the item of metadata; and
identifying the at least one of the one or more rules that includes the user-defined name.
12 . The method of claim 1 , wherein a technical attribute includes format and/or structure of the data in the datasets.
13 . A non-transitory computer readable medium for increasing the speed of preparing data with a specified data quality for storage by automatically identifying for a user, with minimal user input, common contexts among (i) fields in disparate datasets, and (ii) names the user has specified as potentially describing the fields, and by using those common contexts to govern the disparate datasets prior to storage to ensure the specified data quality, the non-transitory computer readable medium storing instructions that are executable by one or more processing devices to perform operations including:
retrieving datasets from external data sources;
identifying, from the retrieved datasets, items of metadata describing technical attributes that characterize fields in the retrieved datasets;
receiving, from a client device, user-defined names that are candidates to describe the fields in the retrieved datasets;
selecting one or more of the user-defined names corresponding to one or more technical attributes that characterize a given field in the retrieved datasets, including:
selecting, from among the user-defined names, a user-defined name representing a semantic meaning of the one or more technical attributes that characterize the given field;
generating an association among
one or more items of metadata describing the one or more technical attributes that characterize the given field in the retrieved datasets, and
at least one of the one or more of the user-defined names selected, from among the candidates, that correspond to the one or more technical attributes that characterize
the given field in the retrieved datasets;
accessing one or more rules that specify one or more operations, with the one or more operations referencing the at least one of the one or more of the user-defined names that correspond to the one or more technical attributes that characterize the given field in the retrieved datasets;
based on the identified association, detecting, in a retrieved dataset, an item of metadata describing the one or more technical attributes that correspond to the at least one of the one or more of the user-defined names referenced in at least one of the one or more rules;
responsive to detecting, applying the at least one of the one or more rules to one or more data items in the retrieved dataset and described by the detected item of metadata; and
following application of the at least one of the one or more rules, storing, in one or more data stores, the one or more data items to which the at least one of the one or more rules are applied.
14 . The non-transitory computer readable medium of claim 13 , wherein at least one of the items of metadata includes a field name for a field of a retrieved dataset.
15 . The non-transitory computer readable medium of claim 14 , wherein the operations further include:
determining a label for at least one item of metadata, the label representing a semantic meaning for the at least one item of metadata;
identifying a match between the label for the at least one item of metadata and the user-defined name; and
responsive to the match, generating a link between the at least one item of metadata and the user-defined name.
16 . The non-transitory computer readable medium of claim 15 , wherein determining the label includes performing one or more semantic discovery processes on at least one of: the at least one item of metadata, or data items described by the at least one item of metadata.
17 . The non-transitory computer readable medium of claim 13 , wherein generating the association includes:
generating a link between at least one item of metadata and the user-defined name.
18 . The non-transitory computer readable medium of claim 15 , wherein generating the link between the at least one item of metadata and the user-defined name includes generating at least one data structure including data representing the at least one item of metadata and a pointer to data representing the user-defined name.
19 . The non-transitory computer readable medium of claim 15 , wherein generating the link between the at least one item of metadata and the user-defined name includes generating at least one data structure including data representing the user-defined name and a pointer to the at least one item of metadata.
20 . An electronic system for increasing the speed of preparing data with a specified data quality for storage by automatically identifying for a user, with minimal user input, common contexts among (i) fields in disparate datasets, and (ii) names the user has specified as potentially describing the fields, and by using those common contexts to govern the disparate datasets prior to storage to ensure the specified data quality, including:
one or more processing devices; and
a hardware storage device storing instructions that are executable by the one or more processing devices to perform operations including:
retrieving datasets from external data sources;
identifying, from the retrieved datasets, items of metadata describing technical attributes that characterize fields in the retrieved datasets;
receiving, from a client device, user-defined names that are candidates to describe the fields in the retrieved datasets;
selecting one or more of the user-defined names corresponding to one or more technical attributes that characterize a given field in the retrieved datasets, including:
selecting, from among the user-defined names, a user-defined name representing a semantic meaning of the one or more technical attributes that characterize the given field;
generating an association among
one or more items of metadata describing the one or more technical attributes that characterize the given field in the retrieved datasets, and
at least one of the one or more of the user-defined names selected, from among the candidates, that correspond to the one or more technical attributes that characterize the given field in the retrieved datasets;
accessing one or more rules that specify one or more operations, with the one or more operations referencing the at least one of the one or more of the user-defined names that correspond to the one or more technical attributes that characterize the given field in the retrieved datasets;
based on the identified association, detecting, in a retrieved dataset, an item of metadata describing the one or more technical attributes that correspond to the at least one of the one or more of the user-defined names referenced in at least one of the one or more rules;
responsive to detecting, applying the at least one of the one or more rules to one or more data items in the retrieved dataset and described by the detected item of metadata; and
following application of the at least one of the one or more rules, storing, in one or more data stores, the one or more data items to which the at least one of the one or more rules are applied.
21 . The electronic system of claim 20 , wherein at least one of the items of metadata includes a field name for a field of a retrieved dataset.
22 . The electronic system of claim 21 , wherein the operations further include:
determining a label for at least one item of metadata, the label representing a semantic meaning for the at least one item of metadata;
identifying a match between the label for the at least one item of metadata and the user- defined name; and
responsive to the match, generating a link between the at least one item of metadata and the user-defined name.
23 . The electronic system of claim 22 , wherein determining the label includes performing one or more semantic discovery processes on at least one of: the at least one item of metadata, or data items described by the at least one item of metadata.
24 . The electronic system of claim 20 , wherein generating the association includes:
generating a link between at least one item of metadata and the user-defined name.
25 . The electronic system of claim 24 , wherein generating the link between the at least one item of metadata and the user-defined name includes generating at least one data structure including data representing the at least one item of metadata and a pointer to data representing the user-defined name.