Resource allocation optimization for multi-dimensional machine learning environments
Some embodiments of the present application include obtaining first data from a data feed to be provided to a plurality of machine learning models and detecting a changepoint in the first data. In response to the changepoint being detected, a first machine learning model may be executed on the first data to obtain first output datasets. A first performance score for the first machine learning model may be computed based on the first output datasets. A second machine learning model may be caused to execute on the first data based on the first performance score satisfying a first condition.
1. A system for optimizing resource allocation in a multi-thread, multi-dimensional machine learning environment, the system comprising:
memory storing computer program instructions; and
one or more processors that, when executing the computer program instructions, effectuate operations comprising:
obtaining, via a data feed, production data to be provided to a plurality of machine learning models, wherein the plurality of machine learning models comprise a first machine learning model and a second machine learning model each having a first execution frequency;
detecting a changepoint in the production data based on a value of a first feature of the production data being determined to differ from an expected value for the first feature by more than a threshold amount;
responsive to the changepoint being detected in the production data, causing both the first machine learning model and the second machine learning model to be executed on the production data to obtain first output datasets and second output datasets from the first machine learning model and the second machine learning model, respectively;
computing (i) a first performance score for the first machine learning model based on the first output datasets and (ii) a second performance score for the second machine learning model based on the second output datasets; and
in response to determining that at least one of the first performance score or the second performance score satisfies a first condition, building a third machine learning model having a second execution frequency less than the first execution frequency, wherein the third machine learning model is executed on the production data, and wherein the first condition being satisfied comprises the first performance score or the second performance score being less than a threshold performance score.
2. The system of claim 1 , wherein the operations further comprise:
responsive to the changepoint not being detected in the production data, causing the first machine learning model to be executed on the production data in lieu of the second machine learning model being executed on the production data.
3. The system of claim 1 , wherein the plurality of machine learning models further comprises a fourth machine learning model having the second execution frequency, the operations further comprise:
causing the third machine learning model and the fourth machine learning model to be executed on the production data to obtain third output datasets and fourth output datasets, wherein the third machine learning model is assigned as a primary model, and the fourth machine learning model is assigned as a secondary model;
computing (i) a third performance score for the third machine learning model based on the third output datasets and (ii) a fourth performance score for the fourth machine learning model based on the fourth output datasets; and
causing, based on the third performance score and the fourth performance score, the third machine learning model, the fourth machine learning model, or a different machine learning model to remain or to be assigned as a primary model.
4. The system of claim 1 , wherein the plurality of machine learning models further comprises a fourth machine learning model having the second execution frequency, the operations further comprise:
preventing the fourth machine learning model from being executed on the production data in response to determining that at least one of the first performance score or the second performance score satisfies the first condition.
5. A non-transitory computer-readable medium storing computer program instructions that, when executed by one or more processors, effectuate operations comprising:
obtaining first data from a data feed to be provided to a plurality of machine learning models;
detecting a changepoint in the first data;
responsive to the changepoint being detected, causing a first machine learning model to be executed on the first data to obtain first output datasets;
computing a first performance score for the first machine learning model based on the first output datasets; and
causing a second machine learning model to execute on the first data based on the first performance score satisfying a first condition.
6. The non-transitory computer-readable medium of claim 5 , wherein the operations further comprise:
in response to determining that the first performance score satisfies the first condition, building the second machine learning model.
7. The non-transitory computer-readable medium of claim 5 , wherein:
the first machine learning model has a first execution frequency;
the second machine learning model has a second execution frequency; and
the second execution frequency is less than the first execution frequency.
8. The non-transitory computer-readable medium of claim 5 , wherein obtaining the first data comprises:
obtaining the first data via a data feed configured to receive updated application data from one or more real-time applications.
9. The non-transitory computer-readable medium of claim 5 , wherein detecting the changepoint comprises:
determining that a value of a first feature of the first data differs from an expected value for the first feature by more than a threshold amount.
10. The non-transitory computer-readable medium of claim 5 , wherein the operations further comprise:
responsive to the changepoint being detected, causing a third machine learning model to be executed on the first data to obtain second output datasets; and
computing a second performance score for the second machine learning model based on the second output datasets, wherein the second machine learning model is caused to execute on the first data based on the first performance score and the second performance score satisfying the first condition, wherein the first condition being satisfied comprises the first performance score and the second performance score being less than a threshold performance score.
11. The non-transitory computer-readable medium of claim 5 , wherein the operations further comprise:
determining that the first data is to be provided to the first machine learning model and a third machine learning model; and
responsive to the changepoint not being detected, preventing the third machine learning model from being executed on the first data.
12. The non-transitory computer-readable medium of claim 5 , wherein the operations further comprise:
determining that the first data is to be provided to a third machine learning model having an execution frequency less than that of the first machine learning model; and
prior to the third machine learning model executing on the first data, preventing the third machine learning model from executing on the first data based on the first performance score satisfying the first condition.
13. The non-transitory computer-readable medium of claim 5 , wherein second output data is obtained based on the second machine learning model executing on the first data, the operations further comprise:
determining that the first data is to be provided to a third machine learning model having an execution frequency less than that of the first machine learning model; and
causing the third machine learning model to be executed on the first data to obtain third output data, wherein the second machine learning model is assigned as a primary model and the third machine learning model is assigned as a secondary model.
14. The non-transitory computer-readable medium of claim 13 , wherein the operations further comprise:
computing a set of performance metrics for the second machine learning model and the third machine learning model; and
selecting the third machine learning model to be assigned as the primary model and the second machine learning model to be assigned as the secondary model based on the set of performance metrics computed.
15. A method implemented by one or more processors configured to execute computer program instructions, the method comprising:
obtaining first data from a data feed to be provided to a plurality of machine learning models;
detecting a changepoint in the first data;
responsive to the changepoint being detected, causing a first machine learning model to be executed on the first data to obtain first output datasets;
computing a first performance score for the first machine learning model based on the first output datasets; and
causing a second machine learning model to execute on the first data based on the first performance score satisfying a first condition.
16. The method of claim 15 , further comprising:
in response to determining that the first performance score satisfies the first condition, building the second machine learning model, wherein the first condition being satisfied comprises the first performance score being less than a threshold performance score.
17. The method of claim 15 , wherein detecting the changepoint comprises:
determining that a value of a first feature of the first data differs from an expected value for the first feature by more than a threshold amount.
18. The method of claim 15 , further comprising:
determining that the first data is to be provided to the first machine learning model and a third machine learning model; and
responsive to the changepoint not being detected, preventing the third machine learning model from being executed on the first data.
19. The method of claim 15 , further comprising:
determining that the first data is to be provided to a third machine learning model having an execution frequency less than that of the first machine learning model; and
prior to the third machine learning model executing on the first data, preventing the third machine learning model from executing on the first data based on the first performance score satisfying the first condition.
20. The method of claim 15 , wherein second output data is obtained based on the second machine learning model executing on the first data, the method further comprises:
determining that the first data is to be provided to a third machine learning model having an execution frequency less than that of the first machine learning model;
causing the third machine learning model to be executed on the first data to obtain third output data, wherein the second machine learning model is assigned as a primary model and the third machine learning model is assigned as a secondary model;
computing a set of performance metrics for the second machine learning model and the third machine learning model; and
selecting the third machine learning model to be assigned as the primary model and the third machine learning model to be assigned as the secondary model based on the set of performance metrics computed.