Distributed query execution and aggregation across multiple datasets
Computer-implemented methods and systems are disclosed for receiving and indexing a plurality of files for later querying, for dynamically generating scripts to be executed during a query of a data store, and for horizontally distributing a query and aggregating results of the distributed query.
1 . A system for dynamically generating scripts to be executed during a query of a data store, comprising:
a server comprising one or more processor; and
non-transitory memory comprising instructions that, when executed by the one or more processors of the server, cause the one or more processors to:
receive a query comprising a time window of files to searched, key values to search on in two or more distinct datasets, one or more enrichment fields, and one or both of a filter selecting subset of the files to be searched and an aggregation of data from all of the files or from all of the files that are filtered;
dynamically generate a script based on contents of the query, optimizing the script from a template to include only features necessary to satisfy the query; and
distribute the generated script horizontally to a plurality of computing devices;
wherein each computing device of the plurality of computing devices performs a pseudo-join that automatically integrates data from a table of translation, specified by the one or more enrichment fields, into a raw data structure with fields loaded from at least one of the two or more distinct datasets to introduce a shared enriched value;
the generated script begins to parse the raw data structure after the occurrence of the pseudo-join;
call the generated script on each of the files to be searched, wherein each of the files includes the raw data structure; and
join the two or more distinct datasets into single records based upon the shared enriched value that is introduced by the pseudo-join which joins the two or more distinct datasets where no common key previously existed among the two or more distinct datasets, and output the single records in response to the query.
2 . The system of claim 1 , wherein the two or more distinct datasets comprise two or more key fields, respectively, having a same value stored as a different datatype, and wherein automatic conversion of datatype is performed to a shared datatype to enable a join on a same key field.
3 . The system of claim 1 , wherein one of the two or more distinct dataset air unstructured data and thereby lacks a key field present in another of the two or more distinct datasets, and wherein an expand tag is present in the query, causing a key field to be generated with unique values for the one of the datasets lacking the key field, and wherein the generated key field enables integration of the datasets before execution and query processing without modifying the two or more distinct datasets or requiring a predefined primary key.
4 . The system of claim 1 , wherein one of the two or more distinct datasets lacks at least one field present in another of the two or more distinct datasets, and wherein a placeholder value is inserted for the at least one field in every record of the one dataset.
5 . The system of claim 4 , wherein the at least one field an object rather than a primitive datatype, and wherein the placeholder is a default instance of the object.
6 . The system of claim 4 , wherein the at least one field represents a set, array, or dictionary, and wherein the placeholder is an empty instance of the set, array, or dictionary.
7 . The system of claim 1 , wherein the query is expressed in Javascript Object Notation (JSON) format.
8 . The system of claim 1 , wherein the generated script is in the Python scripting language.
9 . A computer-implemented method for dynamically generating scripts to se executed during a query of a data store, comprising:
receiving a query comprising a time window of files to be searched, key values to search on in two of ore distinct datasets, one or more enrichment fields, and one or both of a filter selecting a subset of the files to be searched and an aggregation of data from all of the files or from all of the files that are filtered;
dynamically generating a script based on contents of the query, optimizing the script from a template to include on features necessary to satisfy the query;
distributing the generated script horizontally to a plurality of computing devices;
wherein each computing device of the plurality of computing devices executes the query by first performing a pseudo-join that automatically integrates data from a table of translation, specified by the one or more enrichment fields, into a raw data structure with fields loaded from at least one of the two or more distinct datasets to introduce a shared enriched value;
the generated script begins to parse the raw data structure after the occurrence of the pseudo-join;
calling the generated script on each of the files to be searched, wherein each of the files includes the raw data structure; and
joining the two or more distinct datasets into single records based upon the shared enriched value that is introduced by the pseudo-join which joins the two or more distinct datasets where no common key previously existed among the two or more distinct datasets, and output the single records in response to the query.
10 . The method of claim 9 , wherein the two or more distinct datasets comprise two or more key fields, respectively, having a same value stored as a different datatype, and wherein automatic conversion of datatype is performed to a shared datatype to enable a join on a same key field.
11 . The method of claim 9 , wherein one of the two or more distinct datasets lacks a key field present in another of the two or more distinct datasets, and wherein an expand tag is present in the query, wherein the method further includes generating a key field with unique values for the two or more distinct datasets lacking the key field, and linking the two or more distinct datasets via the generated key field before execution, allowing accurate record association and query processing can occur without modifying the two or more distinct datasets or requiring a predefined primary key.
12 . The method of claim 9 , wherein one of the two or more distinct datasets lacks at least one field present in another of the two or more distinct datasets, and wherein a placeholder value is inserted for the at least one field in every record of the one dataset.
13 . The method of claim 12 , wherein the at least one field represents an object rather than a primitive datatype, and wherein the placeholder is a default instance of the object.
14 . The method of claim 12 , wherein the at least one field represents a set, array, or dictionary, and wherein the placeholder is an empty instance of the set, array, or dictionary.
15 . The method of claim 9 , wherein the query is expressed in JavaScript Object Notation (JSON) format.
16 . The method of claim 9 , wherein the generated script is in the Python scripting language.