Dynamic script generation for distributed query execution and aggregation
Computer-implemented methods and systems are disclosed for receiving and indexing a plurality of files for later querying, for dynamically generating scripts to be executed during a query of a data store, and for horizontally distributing a query and aggregating results of the distributed query.
1. A system for dynamically generating scripts to be executed during a query of a data store, comprising:
a server comprising one or more processors; and
non-transitory memory comprising instructions that, when executed by the one or more processors of the server, cause the one or more processors to:
receive a query comprising key values to search on and one or both of: a filter selecting a subset of files to be searched and an aggregation of data from all of the files or from all of the files that are filtered, wherein the files are stored in a storage that exclusively receives the files during a predetermined window of time;
generate an index for a fixed window of time corresponding to a subset of the files;
dynamically generate a script based on contents of the query, optimizing the script from a template to include only features necessary to satisfy the query and to omit at least one feature unnecessary to satisfy the query; and
distribute the generated script and the index to at least one computing device that will execute the query by calling the generated script on each of the files to be searched; and
organize the subset of the files using the index.
2. The system of claim 1 , wherein the query comprises one or more enrichment fields augmenting data based on a value from a key column in the query, a value from a filter in the query, or an output column of the query with values from an external data source.
3. The system of claim 1 , wherein an index for a field in the query exists, and wherein a set of files to be searched are downloaded from storage and acted upon by the generated script only if the index indicates that each file from the set of files to be searched contains a value specified by the query for that field.
4. The system of claim 1 , wherein an index for a field in the query does not exist, and wherein a set of files to be searched are downloaded from storage and acted upon by the generated script only if a Parquet filter checking for a value specified by the query for that field indicates that each file from the set of files to be searched contains the value.
5. The system of claim 1 , wherein the generated script comprises code to short circuit a Boolean AND by examining a field of the Boolean AND first and returning a file if the field does not satisfy the query.
6. The system of claim 1 , wherein the generated script comprises code to short circuit a Boolean OR by examining a field of the Boolean OR first and returning a file if the field satisfies the query.
7. The system of claim 1 , wherein the generated script comprises code to avoid consulting any indexes if there exists a Boolean OR of at least one non-indexed column.
8. The system of claim 1 , wherein the query is expressed in JavaScript Object Notation (JSON) format.
9. The system of claim 1 , wherein the generated script is in the Python scripting language.
10. The system of claim 1 , wherein the system is locked to work on one query at a time.
11. A computer-implemented method for dynamically generating scripts to be executed during a query of a data store, comprising:
receiving a query comprising key values to search on and one or both of: a filter selecting a subset of files to be searched and an aggregation of data from all of the files or from all of the files that are filtered, wherein the files are stored in a storage that exclusively receives the files during a predetermined window of time,
generating an index for a fixed window of time corresponding to a subset of the files;
dynamically generating a script based on contents of the query, optimizing the script from a template to include only features necessary to satisfy the query and to omit at least one feature unnecessary to satisfy the query; and
distributing the generated script and the index to at least one computing device that will execute the query by calling the generated script on each of the files to be searched; and
organizing the subset of the files using the index.
12. The computer-implemented method of claim 11 , wherein the query comprises one or more enrichment fields augmenting data based on a value from a key column in the query, a value from a filter in the query, or an output column of the query with values from an external data source.
13. The computer-implemented method of claim 11 , wherein an index for a field in the query exists, and wherein a set of files to be searched are downloaded from storage and acted upon by the generated script only if the index indicates that each file from the set of files to be searched contains a value specified by the query for that field.
14. The computer-implemented method of claim 11 , wherein an index for a field in the query does not exist, and wherein a set of files to be searched are downloaded from storage and acted upon by the generated script only if a Parquet filter checking for a value specified by the query for that field indicates that each file from the set of files to be searched contains the value.
15. The computer-implemented method of claim 11 , wherein the generated script comprises code to short circuit a Boolean AND by examining a field of the Boolean AND first and returning a file if the field does not satisfy the query.
16. The computer-implemented method of claim 11 , wherein the generated script comprises code to short circuit a Boolean OR by examining a field of the Boolean OR first and returning a file if the field satisfies the query.
17. The computer-implemented method of claim 11 , wherein the generated script comprises code to avoid consulting any indexes if there exists a Boolean OR of at least one non-indexed column.
18. The computer-implemented method of claim 11 , wherein the query is expressed in JavaScript Object Notation (JSON) format.
19. The computer-implemented method of claim 11 , wherein the generated script is in the Python scripting language.
20. The computer-implemented method of claim 11 , further comprising locking a system to work on one query at a time.