System and method for generating training sets for neural networks
A system and method for generating training sets for training neural networks. The method includes determining a segmentation based on a column from a columnar database table; generating a group-by query based on the segmentation; generating a plurality of reduced queries based on the group-by query; executing the group-by query on a table of a database to obtain a result table, wherein the result table includes a plurality of results, wherein each result corresponds to a respective reduced query of the plurality of reduced queries; and generating a plurality of training query pairs by pairing each reduced query with its corresponding reduced result.
1. A method for generating training sets for training neural networks, comprising:
determining a segmentation based on a column from a columnar database table;
generating a group-by query based on the segmentation;
generating a plurality of reduced queries based on the group-by query;
executing the group-by query on a table of a database to obtain a result table, wherein the result table includes a plurality of results, wherein each result corresponds to a respective reduced query of the plurality of reduced queries; and
generating a plurality of training query pairs by pairing each reduced query with its corresponding reduced result.
2. The method of claim 1 , wherein the segmentation is based on dimensions.
3. The method of claim 1 , wherein the segmentation is based on measures.
4. The method of claim 3 , further comprising:
determining a distribution of values pertaining to the measures, wherein the segmentation is determined based on the distribution of values.
5. The method of claim 1 , wherein generating the group-by query further comprises:
generating at least one nested query within the group-by query.
6. The method of claim 1 , further comprising:
generating a plurality of training queries based on a set of queries generated by at least one user.
7. The method of claim 6 , wherein generating each training query further comprises:
determining a variable element of a query of the set of queries; and
determining a variance of the variable element, wherein the training query is generated based on the determined variable element and the determined variance.
8. The method of claim 1 , further comprising:
training a neural network at least partially using the plurality of training query pairs.
9. The method of claim 8 , wherein the neural network is further trained when a predicted result generated by the neural network differs from a real result generated based on a dataset above a threshold, further comprising:
generating an updated predicted result based on the predicted result and the real result, wherein the updated predicted result is utilized as a training input to the neural network.
10. A non-transitory computer readable medium having stored thereon instructions for causing a processing circuitry to execute a process, the process comprising:
determining a segmentation based on a column from a columnar database table;
generating a group-by query based on the segmentation;
generating a plurality of reduced queries based on the group-by query;
executing the group-by query on a table of a database to obtain a result table, wherein the result table includes a plurality of results, wherein each result corresponds to a respective reduced query of the plurality of reduced queries; and
generating a plurality of training query pairs by pairing each reduced query with its corresponding reduced result.
11. A system for generating training sets for training neural networks, comprising:
a processing circuitry; and
a memory, the memory containing instructions that, when executed by the processing circuitry, configure the system to:
determine a segmentation based on a column from a columnar database table;
generate a group-by query based on the segmentation;
generate a plurality of reduced queries based on the group-by query;
execute the group-by query on a table of a database to obtain a result table, wherein the result table includes a plurality of results, wherein each result corresponds to a respective reduced query of the plurality of reduced queries; and
generate a plurality of training query pairs by pairing each reduced query with its corresponding reduced result.
12. The system of claim 11 , wherein the segmentation is based on dimensions.
13. The system of claim 11 , wherein the segmentation is based on measures.
14. The system of claim 11 , wherein the system is further configured to:
determine a distribution of values pertaining to the measures, wherein the segmentation is determined based on the distribution of values.
15. The system of claim 11 , wherein the system is further configured to:
generate at least one nested query within the group-by query.
16. The system of claim 11 , wherein the system is further configured to:
generate a plurality of training queries based on a set of queries generated by at least one user.
17. The system of claim 16 , wherein the system is further configured to:
determine a variable element of a query of the set of queries; and
determine a variance of the variable element, wherein the training query is generated based on the determined variable element and the determined variance.
18. The system of claim 11 , wherein the system is further configured to:
train a neural network at least partially using the plurality of training query pairs.
19. The system of claim 18 , wherein the neural network is further trained when a predicted result generated by the neural network differs from a real result generated based on a dataset above a threshold, wherein the system is further configured to:
generate an updated predicted result based on the predicted result and the real result, wherein the updated predicted result is utilized as a training input to the neural network.