Intelligent compute request scoring and routing
A system and methods for determining computer resource allocation, the system having a network communication interface configured to receive a query from a device, the query indicating a request to perform a processing operation on a portion of one or more data set. The system may also include data storage for data including historical information related to processing of data sets by back-end computer resources, and hardware processors configured to determine one or more scores associated with a query and based at least in part on the historical information. The system may determine a particular back-end computer resource with a highest score, provide a compute request to the particular back-end computer resource to perform a processing operation on a portion of one or more data sets, and store processing information related to the processing of the compute request by the particular back-end computer resource as historical information.
1. A system comprising:
one or more computer processors configured to execute computer-executable instructions to cause the system to at least:
receive, via a user input, a query indicating a request to perform a processing operation on a portion of one or more data sets;
determine that the query includes user defined code for running by a back-end computer resource, the user defined code including a custom script to perform specific operations;
determine one or more scores associated with the query, including at least:
a first score based at least in part on the query including the user defined code, wherein the first score indicates a level of risk of executing the user defined code; and
a second score based at least in part on a scope of the processing operation including one or more of a type of processing being performed or a size of the one or more data sets being processed;
in response to determining that the query includes user defined code, filter a set of back-end resources to exclude back-end resources not configured to execute the user defined code;
determine, based on the received query, the determined one or more scores, and historical information of prior processing outcomes, a back-end resource of the filtered set of back-end resources to execute the query;
determine a stop parameter based on the user defined code, the stop parameter indicating one or more of a run-time limit, a memory usage limit, or a post-processing action configured to stop the determined back-end computer resource;
generate a compute request including the query and the stop parameter; and
provide the compute request to the determined back-end computer resource.
2. The system of claim 1 , wherein the back-end computer resource that receives the query is indicated by a highest score of the one or more scores associated with the query.
3. The system of claim 1 , wherein at least one of the scores is based on an originator of the query.
4. The system of claim 3 , wherein the originator is associated with a predefined group of a number of groups that the query may be associated with.
5. The system of claim 1 , wherein the historical information includes processing time.
6. The system of claim 1 , wherein the historical information includes size of the data set.
7. The system of claim 1 , wherein the historical information includes characteristics of the back-end computer resource.
8. The system of claim 1 , wherein the historical information includes query originator data.
9. The system of claim 1 , wherein the historical information includes data set information corresponding to a particular data set processed by the back-end computer resource.
10. The system of claim 1 , wherein the one or more computer processors are further configured to execute the computer-executable instructions to at least receive processing information from two or more back-end computer resources and save the processing information as historical information.
11. The system of claim 1 , wherein the one or more computer processors are further configured to execute the computer-executable instructions to control a lifecycle parameter of the back-end computer resource.
12. The system of claim 11 , wherein controlling a lifecycle parameter comprises setting a start parameter of the back-end computer resource.
13. The system of claim 11 , wherein controlling a lifecycle parameter comprises controlling a start of a back-end computer resource with similar compute parameters.
14. The system of claim 1 , wherein the one or more computer processors are further configured to execute computer-executable instructions to start the back-end computer resource to process the compute request for a query with user defined code and stop the determined back-end computer resource when the compute request is completed.
15. The system of claim 1 , wherein the user defined code is not selectable from a user interface.
16. The system of claim 1 , wherein the level of risk indicates a risk of causing a divide by zero operation or an incomplete processing flow, and the instructions further cause the system to:
trim the query by replacing a portion of the query with previously computed data associated with previous execution of the portion of the query.
17. The system of claim 1 , wherein the instructions further cause the system to:
dynamically adjust one or more scores in real-time based on current health status of the back-end resources.
18. A method of resource allocation, comprising:
receiving, via a user input, a query indicating a request to perform a processing operation on a portion of one or more data sets;
determining that the query includes user defined code for running by a back-end computer resource, the user defined code including a custom script to perform specific operations;
determining one or more scores associated with the query, including at least:
a first score based at least in part on the query including the user defined code, wherein the first score indicates a level of risk of executing the user defined code; and
a second score based at least in part on a scope of the processing operation including one or more of a type of processing being performed or a size of the one or more data sets being processed;
in response to determining that the query includes user defined code, filtering a set of back-end resources to exclude back-end resources not configured to execute the user defined code;
determining, based on the received query, the determined one or more scores, and historical information of prior processing outcomes from previous queries, a back-end resource of the filtered set of back-end resources to execute the query;
determining a stop parameter based on the user defined code, the stop parameter indicating one or more of a run-time limit, a memory usage limit, or a post-processing action configured to stop the determined back-end computer resource;
generating a compute request including the query and the stop parameter; and
providing the compute request to the determined back-end resource;
wherein the method is performed by one or more computer hardware processors configured to execute computer-executable instructions stored on a non-transitory computer storage medium.
19. The method of claim 18 , further comprising starting the back-end resource for a query with user defined code, and stopping the back-end resource when the compute request is completed.
20. The method of claim 18 , further comprising:
starting the back-end computer resource;
providing a compute request to the back-end resource; and
stopping the determined backend resource based on the stop parameter.
21. The method of claim 18 , wherein the user defined code is not selectable from a user interface.