On-demand retrieval of structured data in aggregating data across distinct sources
A method for enabling a user to generate a complex aggregation on their own by providing the user with a graphical user interface that displays data items in a data catalog and that provides controls for the user to select data items to be used in generating the complex aggregation, and to select a type of aggregation, and based on the user's selections, automatically generating computer instructions to generate a value of the complex aggregation is described.
1 . A method implemented by a data processing system for providing a user with a graphical user interface that displays data items in a data catalog and that provides controls for the user to select data items to be used in generating one or more aggregations, wherein the controls also enable the user to select a type of aggregation, and based on user's selections, automatically generating computer instructions to generate a value of the one or more aggregations that the user had selected, the method including:
accessing, from a data catalog, names of fields of datasets of a plurality of data sources, with a dataset including multiple data elements, each data element structured with fields;
displaying a graphical user interface that provides first visual representations of at least a plurality of the names of the fields of the datasets of the plurality of data sources accessed from the data catalog, with the at least the plurality of the names representing candidate inputs for defining an aggregation;
displaying, in the graphical user interface, one or more first controls for specifying which of the candidate inputs represented by the at least the plurality of the names accessed from the data catalog are selected for defining the aggregation;
wherein a first control specifies which of the candidate inputs are selected by enabling selection of a first visual representation, wherein a selected, first visual representation specifies a name of a field of a dataset of the plurality of data sources selected for defining the aggregation;
identifying first visual representations selected by at least one of the one or more first controls;
identifying, by a data processing system executing logic, names of fields represented by the selected first visual representations;
based on the identified names of fields, determining one or more types of aggregations that are permissible for aggregating data items associated with the identified names of the fields, wherein a type of aggregation is associated with one or more names of one or more fields;
displaying, in the graphical user interface, one or more second visual representations of the one or more types of aggregations that are permissible for aggregating the data items;
displaying, in the graphical user interface, one or more second controls for specifying which type of aggregation is selected;
wherein a second control specifies which type of aggregation is selected by enabling selection of a second visual representation;
whereby the graphical user interface provides for selection of one or more resources;
in response to selection of at least one of the resources, displaying, in the graphical user interface, one or more third visual representations of fields associated with the at least one of the resources selected;
based on first visual representations selected by at least one of the one or more first controls, one or more second visual representations selected by at least one of the one or more second controls and at least a selected one of the one or more third visual representations, generating, by the data processing system, computer instructions that are executable to:
detect data items identified by one or more of the names represented by the selected first visual representations; and
based on the detected data items and the at least selected one of the one or more third visual representations, generate one or more values of one or more aggregations of one or more types represented by one or more selected second visual representations; and
storing, in memory, the computer instructions.
2 . The method of claim 1 , further including:
based on the first visual representations selected by the at least one of the one or more first controls and the one or more second visual representations selected by the at least one of the one or more second controls, generating a definition of the aggregation;
wherein the definition specifies names represented by the selected first visual representations, and
wherein the definition specifies a type represented by the one or more selected second visual representations.
3 . The method of claim 1 ,
wherein the detecting of data items identified by one or more names represented by the selected first visual representations includes detecting a first item from a first data source and a second data item from a second data source, wherein the first and second data items are identified by one or more names represented by the selected first visual representations, and wherein the first and second data sources are distinct and/or different data sources; and
wherein the generating of the one or more values of the one or more aggregations of the one or more types represented by the one or more selected second visual representations includes generating the values of the aggregations of the types represented by the selected second visual representations based on the detected first and second data items.
4 . The method of claim 1 , the generating of the computer instructions further including:
generating a first transform based on the selected first visual representations, wherein the first transform is configured to be inserted into one or more placeholders in one or more pre-configured templates of one or more computation graphs.
5 . The method of claim 4 , the generating of the computer instructions further including:
generating a second transform based on the selected second visual representation, wherein the second transform is configured to be inserted into a placeholder in the one or more pre-configured templates of the one or more computation graphs.
6 . The method of claim 5 , the generating of the computer instructions further including:
inserting the first transform and the second transform into the respective placeholders in the one or more pre-configured templates of the one or more computation graphs for generating the aggregation.
7 . The method of claim 4 , wherein the one or more templates of the one or more computation graphs include a template batch graph, wherein the template batch graph includes a placeholder for insertion of the first transform such that the template batch graph with the first transform inserted into the placeholder of the template batch graph is configured to perform, in predetermined time intervals, batch retrieval from disk of data items used for the one or more aggregates.
8 . The method of claim 7 , wherein the batch retrieval from disk of the data items is performed by querying the data items in a single query from the disk.
9 . The method of claim 7 , wherein the one or more templates of the one or more computation graphs include a template real-time graph, and wherein the template real-time graph includes a placeholder for insertion of the first transform, such that the template real-time graph with the first transform inserted into the placeholder of the template real-time graph is configured to perform real-time retrieval from memory of data items used for the one or more aggregates.
10 . The method of claim 9 , wherein the memory is volatile memory.
11 . The method of claim 9 , wherein the real-time retrieval from memory of the data items is performed by querying the data items in a single query from the memory.
12 . The method of claim 9 , wherein the one or more templates of the one or more computation graphs further include a template aggregate graph, which includes a placeholder for insertion of the second transform.
13 . The method of claim 12 , wherein the template aggregate graph is connected with the output of the template batch graph and the output of the template real-time graph such that, with the second transform inserted into the placeholder of the template aggregate graph, results of the batch retrieval are supplemented with results from the real-time retrieval to generate one or more values of one or more aggregations of one or more types represented by one or more selected second visual representations.
14 . The method of claim 1 , further including:
in response to receiving a request for the aggregate, executing the stored computer instructions to detect data items identified by one or more names represented by the selected first visual representations and to generate the one or more values of the one or more aggregations of the one or more types represented by the one or more selected second visual representations based on the detected data items.
15 . The method of claim 14 , further including:
wherein the one or more aggregates include multiple aggregates, and the computer instructions are configured such that the data items used for the multiple aggregates are to be queried in a single query from one or more data storages.
16 . The method of claim 1 , wherein the displaying, in the graphical user interface, of the one or more second visual representations of the one or more types of aggregations that are permissible for aggregating the data items is based on the selected first visual representation, preferably such that only second visual representations of types of aggregations that are permissible for aggregating the data items identified by the names identifier that are specified by the selected first visual representation are displayed in the graphical user interface.
17 . The method of claim 1 , further including:
displaying, in the graphical user interface, one or more third controls for specifying a duration over which the aggregation is generated.
18 . The method of claim 17 , further including:
displaying, in the graphical user interface, one or more fourth controls for specifying an event type over which the aggregation is generated.
19 . The method of claim 1 , wherein the data catalog is distinct from the plurality of data sources.
20 . A data processing system for providing a user with a graphical user interface that displays data items in a data catalog and that provides controls for the user to select data items to be used in generating one or more aggregations, wherein the controls also enable the user to select a type of aggregation, and based on user's selections, automatically generating computer instructions to generate a value of the one or more aggregations that the user had selected, the data processing system including:
one or more processors; and
one or more machine-readable hardware storage devices storing instructions that are executable by the one or more processors to perform actions of:
accessing, from a data catalog, names of fields of datasets of a plurality of data sources, with a dataset including multiple data elements, each data element structured with fields;
displaying a graphical user interface that provides first visual representations of at least a plurality of the names of the fields of the datasets of the plurality of data sources accessed from the data catalog, with the at least the plurality of the names representing candidate inputs for defining an aggregation;
displaying, in the graphical user interface, one or more first controls for specifying which of the candidate inputs represented by the at least the plurality of the names accessed from the data catalog are selected for defining the aggregation;
wherein a first control specifies which of the candidate inputs are selected by enabling selection of a first visual representation, wherein a selected, first visual representation specifies a name of a field of a dataset of the plurality of data sources selected for defining the aggregation;
identifying first visual representations selected by at least one of the one or more first controls;
identifying, by a data processing system executing logic, names of fields represented by the selected first visual representations;
based on the identified names of fields, determining one or more types of aggregations that are permissible for aggregating data items associated with the identified names of the fields, wherein a type of aggregation is associated with one or more names of one or more fields;
displaying, in the graphical user interface, one or more second visual representations of the one or more types of aggregations that are permissible for aggregating the data items;
displaying, in the graphical user interface, one or more second controls for specifying which type of aggregation is selected;
wherein a second control specifies which type of aggregation is selected by enabling selection of a second visual representation;
whereby the graphical user interface provides for selection of one or more resources;
in response to selection of at least one of the resources, displaying, in the graphical user interface, one or more third visual representations of fields associated with the at least one of the resources selected;
based on first visual representations selected by at least one of the one or more first controls, one or more second visual representations selected by at least one of the one or more second controls and at least a selected one of the one or more third visual representations, generating, by the data processing system, computer instructions that are executable to:
detect data items identified by one or more of the names represented by the selected first visual representations; and
based on the detected data items and the at least selected one of the one or more third visual representations, generate one or more values of one or more aggregations of one or more types represented by one or more selected second visual representations; and
storing, in memory, the computer instructions.
21 . The data processing system of claim 20 wherein the data processing system is configured to further performs the actions of:
based on the first visual representations selected by the at least one of the one or more first controls and the one or more second visual representations selected by the at least one of the one or more second controls, generating a definition of the aggregation;
wherein the definition specifies names represented by the selected first visual representations, and
wherein the definition specifies a type represented by the one or more selected second visual representations.
22 . The data processing system of claim 20 wherein the data processing system is configured to further perform the actions of:
generating a first transform based on the selected first visual representations, wherein the first transform is configured to be inserted into one or more placeholders in one or more pre-configured templates of one or more computation graphs.
23 . One or more non-transitory machine-readable hardware storage devices for providing a user with a graphical user interface that displays data items in a data catalog and that provides controls for the user to select data items to be used in generating one or more aggregations, wherein the controls also enable the user to select a type of aggregation, and based on user's selections, automatically generating computer instructions to generate a value of the one or more aggregations that the user had selected, the one or more non-transitory machine-readable hardware storage devices storing instructions that are executable by a data processing system to perform the actions of:
accessing, from a data catalog, names of fields of datasets of a plurality of data sources, with a dataset including multiple data elements, each data element structured with field;
displaying a graphical user interface that provides first visual representations of at least a plurality of the names of the fields of the datasets of the plurality of data sources accessed from the data catalog, with the at least the plurality of the names representing candidate inputs for defining an aggregation;
displaying, in the graphical user interface, one or more first controls for specifying which of the candidate inputs represented by the at least the plurality of the names accessed from the data catalog are selected for defining the aggregation;
wherein a first control specifies which of the candidate inputs are selected by enabling selection of a first visual representation, wherein a selected, first visual representation specifies a name of a field of a dataset of the plurality of data sources selected for defining the aggregation;
identifying first visual representations selected by at least one of the one or more first controls;
identifying, by a data processing system executing logic, names of fields represented by the selected first visual representations;
based on the identified names of fields, determining one or more types of aggregations that are permissible for aggregating data items associated with the identified names of the fields, wherein a type of aggregation is associated with one or more names of one or more fields;
displaying, in the graphical user interface, one or more second visual representations of one or more types of aggregations that are permissible for aggregating the data items;
displaying, in the graphical user interface, one or more second controls for specifying which type of aggregation is selected;
wherein a second control specifies which type of aggregation is selected by enabling selection of a second visual representation;
whereby the graphical user interface provides for selection of one or more resources;
in response to selection of at least one of the resources, displaying, in the graphical user interface, one or more third visual representations of fields associated with the at least one of the resources selected;
based on first visual representations selected by at least one of the one or more first controls, one or more second visual representations selected by at least one of the one or more second controls and at least a selected one of the one or more third visual representations, generating, by the data processing system, computer instructions that are executable to:
detect data items identified by one or more of the names represented by the selected first visual representations; and
based on the detected data items and the at least selected one of the one or more third visual representations, generate one or more values of one or more aggregations of one or more types represented by one or more selected second visual representations; and
storing, in memory, the computer instructions.
24 . The one or more non-transitory machine-readable hardware storage devices of claim 22 , wherein the actions further include
based on the first visual representations selected by the at least one of the one or more first controls and the one or more second visual representations selected by the at least one of the one or more second controls, generating a definition of the aggregation;
wherein the definition specifies names represented by the selected first visual representations, and
wherein the definition specifies a type represented by the one or more selected second visual representations.
25 . The one or more non-transitory machine-readable hardware storage devices of claim 22 , wherein the actions further include generating a first transform based on the selected first visual representations, wherein the first transform is configured to be inserted into one or more placeholders in one or more pre-configured templates of one or more computation graphs.