Systems and methods for generating runtime predictions in distributed computer architectures

1 . A method for generating runtime predictions in distributed computer architectures, comprising:

receiving, by a runtime prediction computer program executed by an electronic device, training data regarding completion of a plurality of risk compute jobs;

extracting, by the runtime prediction computer program, bucketing statistics and instrument level features from the training data;

aggregating, by the runtime prediction computer program, the instrument level features a plurality of instruments from the training data at a task level to generate a task level aggregated feature count;

training, by the runtime prediction computer program, a machine learning model with the training data, the extracted bucketing statistics, and the instrument level features, wherein the extracted bucketing statistics comprise one or more of a task level calculation time, instrument names, and a unique hash key indicating how each task is scheduled and distributed on a cloud compute infrastructure, wherein the machine learning model is trained using an ElasticNet model for linear regression with cross validation, wherein the ElasticNet model for linear regression determines a feature weight for each instrument of the plurality of instruments from the training data multiplied by the task level aggregated feature count;

storing, by the runtime prediction computer program on a database, the feature weight for each instrument of the plurality of instruments from the training data;

receiving, by the runtime prediction computer program, a plurality of jobs for a period of time;

calculating, by the runtime prediction computer program, an instrument cost for each of the plurality of jobs using the machine learning model upon determining the instrument level feature of the plurality of jobs is known to the machine learning model;

retraining, by the runtime prediction computer program, the machine learning model periodically using rolling data; and

distributing, by a scheduling process executed on the cloud compute infrastructure, one or more instruments of the plurality of instruments to at least one job of the plurality of jobs based on the instrument cost.

2 . The method of claim 1 , wherein the instrument level features comprise one or more of market dependencies, instrument types, and instrument maturities.

3 . The method of claim 1 , further comprising:

routing, by the runtime prediction computer program, one or more of the plurality of jobs to low-cost cloud processing in response to the instrument cost being below a threshold.

4 . The method of claim 1 , further comprising:

routing, by the runtime prediction computer program, one or more of the plurality of jobs to high-cost cloud processing in response to the instrument cost being above a threshold.

5 . The method of claim 1 , further comprising:

grouping, by the runtime prediction computer program, the instruments based on their feature weights.

6 . A system, comprising:

a data source comprising data regarding completion of a plurality of risk compute jobs; and

an electronic device executing a runtime prediction computer program and a machine learning model, wherein:

the runtime prediction computer program receives training data regarding completion of the plurality of risk compute jobs;

the runtime prediction computer program extracts bucketing statistics and instrument level features from the training data, wherein the bucketing statistics comprise one or more of a task level calculation time, instrument names, and a unique hash key indicating how each task is scheduled and distributed on a cloud compute infrastructure;

the runtime prediction computer program aggregates, by the runtime prediction computer program, the instrument level features a plurality of instruments from the training data at a task level to generate a task level aggregated feature count;

the runtime prediction computer program trains the machine learning model with the training data, the extracted bucketing statistics, and the instrument level features, wherein the machine learning model is trained using an ElasticNet model for linear regression with cross validation, wherein the ElasticNet model for linear regression determines a feature weight for each instrument of the plurality of instruments from the training data multiplied by the task level aggregated feature count;

the runtime prediction computer program stores on a database the feature weight for each instrument of the plurality of instruments from the training data;

the runtime prediction computer program receives a plurality of jobs for a period of time;

the runtime prediction computer program calculates an instrument cost for each of the plurality of jobs using the machine learning model upon determining the instrument level feature of the plurality of jobs is known to the machine learning model;

the runtime prediction computer program, retrains the machine learning model periodically using rolling data; and

a scheduling process executed on the cloud compute infrastructures distributes one or more instruments of the plurality of instruments to at least one job of the plurality of jobs based on the instrument cost.

7 . The system of claim 6 , wherein the instrument level features comprise one or more of market dependencies, instrument types, and instrument maturities.

8 . The system of claim 6 , wherein the runtime prediction computer program routes one or more of the plurality of jobs to low-cost cloud processing in response to the instrument cost being below a threshold.

9 . The system of claim 6 wherein the runtime prediction computer program routes one or more of the plurality of jobs to high-cost cloud processing in response to the instrument cost being above a threshold.

10 . The system of claim 6 , wherein the runtime prediction computer program groups the instruments based on their feature weights.

11 . A non-transitory computer readable storage medium, including instructions stored thereon, which when read and executed by one or more computer processors, cause the one or more computer processors to perform steps comprising:

receiving training data regarding completion of a plurality of risk compute jobs;

extracting bucketing statistics and instrument level features from the training data, wherein the bucketing statistics comprise one or more of a task level calculation time, instrument names, and a unique hash key indicating how each task is scheduled and distributed on a cloud compute infrastructure and the instrument level features comprise one or more of market dependencies, instrument types, and instrument maturities;

training a machine learning model with the training data, the extracted bucketing statistics, and the instrument level features, wherein the machine learning model is trained using an ElasticNet model for linear regression with cross validation, wherein the ElasticNet model for linear regression determines a feature weight for each instrument of the plurality of instruments from the training data multiplied by the task level aggregated feature coun;

storing on a database the feature weight for each instrument of the plurality of instruments from the training data;

receiving a plurality of jobs for a period of time;

calculating an instrument cost for each of the plurality of jobs using the machine learning model upon determining the instrument level feature of the plurality of jobs is known to the machine learning model; and

retraining, by the runtime prediction computer program, the machine learning model periodically using rolling data; and

distributing, by a scheduling process executed on the cloud compute infrastructures, one or more instruments of the plurality of instruments to at least one job of the plurality of jobs based on the instrument cost.

12 . The non-transitory computer readable storage medium of claim 11 , further comprising instructions stored thereon, which when read and executed by one or more computer processors, cause the one or more computer processors to perform steps comprising:

routing one or more of the plurality of jobs to low-cost cloud processing in response to the instrument cost being below a threshold.

13 . The non-transitory computer readable storage medium of claim 11 , further comprising instructions stored thereon, which when read and executed by one or more computer processors, cause the one or more computer processors to perform steps comprising:

routing, by the runtime prediction computer program, one or more of the plurality of jobs to high-cost cloud processing in response to the instrument cost being above a threshold.

14 . The non-transitory computer readable storage medium of claim 11 , further comprising instructions stored thereon, which when read and executed by one or more computer processors, cause the one or more computer processors to perform steps comprising:

grouping, by the runtime prediction computer program, the instruments based on their feature weights.