Model training method, data processing method, electronic device, and program product
Embodiments of the present disclosure relate to a model training method, a data processing method, an electronic device, and a computer program product. The method includes: acquiring storage information associated with a simulated network environment; and training a reinforcement learning model using simulated data and based on a simulated-data read request for a node among multiple nodes included in the simulated network environment and each having a cache. With the technical solutions of the present disclosure, the cache allocation and cache replacement problems can be simultaneously solved by using a reinforcement learning model to determine in a dynamic environment a data caching scheme that meets predetermined criteria, so that it is possible to not only improve the accuracy and efficiency of determining the data caching scheme with less cost overhead, but also improve the user experience of users using the caching system.
1. A method, comprising:
acquiring storage information associated with a simulated network environment, wherein the simulated network environment includes multiple nodes each having a cache, and the storage information indicates the usage of the cache of each node; and
training a reinforcement learning model using simulated data and based on a simulated-data read request for a node among the multiple nodes, wherein the reinforcement learning model utilizes one of a first reinforcement learning algorithm that incorporates an environment model and a second reinforcement learning algorithm that does not incorporate an environment model, and wherein the reinforcement learning model includes an environment corresponding to the simulated network environment, the reinforcement learning model includes an agent corresponding to the multiple nodes, a state corresponding to the storage information, an action corresponding to a candidate data caching scheme, and a reward corresponding to a performance evaluation of the candidate data caching scheme with regard to the simulated-data read request, and the reinforcement learning model is trained such that the performance evaluation in the reinforcement learning model satisfies a predetermined performance evaluation requirement, and such that the reinforcement learning model can be used for determining a data caching scheme for input data;
wherein acquiring the storage information includes acquiring the storage information and topological information of the multiple nodes;
wherein the reinforcement learning model includes the state corresponding to the storage information and the topological information;
wherein the performance evaluation is indicated by a performance evaluation score which includes a weighted sum of sub-performance evaluation scores of multiple sub-performance evaluations; and
wherein training the reinforcement learning model includes determining a data caching scheme using the trained reinforcement learning model and based on the candidate data caching scheme corresponding to a higher performance evaluation score relative to other performance evaluation scores in the trained reinforcement learning model.
2. The method according to claim 1 , wherein the simulated-data read request is received from the environment in the reinforcement learning model.
3. The method according to claim 1 , wherein the data caching scheme indicates at least one of the following:
a cache in which the simulated data is to be cached; and
cached data to be cleared from the cache, wherein the remaining available space of the cache in which the simulated data is to be cached is insufficient.
4. The method according to claim 1 , wherein training the reinforcement learning model includes:
acquiring an initial data caching scheme; and
training the reinforcement learning model based on the initial data caching scheme.
5. The method according to claim 1 , wherein training the reinforcement learning model includes:
acquiring restrictions on data caching schemes; and
training the reinforcement learning model based on the restrictions.
6. The method according to claim 1 , wherein training the reinforcement learning model includes:
training the reinforcement learning model such that the performance evaluation in the trained reinforcement learning model is better than a performance evaluation of a predetermined candidate data caching scheme with regard to the simulated-data read request.
7. The method according to claim 1 , wherein the performance evaluation is indicated by a performance evaluation score which includes a weighted sum of sub-performance evaluation scores of multiple sub-performance evaluations, and training the reinforcement learning model includes:
training the reinforcement learning model such that the performance evaluation score in the trained reinforcement learning model is higher than a predetermined performance evaluation score.
8. The method according to claim 1 , further comprising:
receiving the input data; and
determining the data caching scheme for the input data using the reinforcement learning model.
9. An electronic device, including:
at least one hardware processing unit; and
at least one memory coupled to the at least one hardware processing unit and storing instructions for execution by the at least one hardware processing unit, wherein the instructions, when executed by the at least one hardware processing unit, cause the electronic device to perform actions including:
acquiring storage information associated with a simulated network environment, wherein the simulated network environment includes multiple nodes each having a cache, and the storage information indicates the usage of the cache of each node; and
training a reinforcement learning model using simulated data and based on a simulated-data read request for a node among the multiple nodes, wherein the reinforcement learning model utilizes one of a first reinforcement learning algorithm that incorporates an environment model and a second reinforcement learning algorithm that does not incorporate an environment model, and wherein the reinforcement learning model includes an environment corresponding to the simulated network environment, the reinforcement learning model includes an agent corresponding to the multiple nodes, a state corresponding to the storage information, an action corresponding to a candidate data caching scheme, and a reward corresponding to a performance evaluation of the candidate data caching scheme with regard to the simulated-data read request, and the reinforcement learning model is trained such that the performance evaluation in the reinforcement learning model satisfies a predetermined performance evaluation requirement, and such that the reinforcement learning model can be used for determining a data caching scheme for input data;
wherein acquiring the storage information includes acquiring the storage information and topological information of the multiple nodes;
wherein the reinforcement learning model includes the state corresponding to the storage information and the topological information;
wherein the performance evaluation is indicated by a performance evaluation score which includes a weighted sum of sub-performance evaluation scores of multiple sub-performance evaluations; and
wherein training the reinforcement learning model includes determining a data caching scheme using the trained reinforcement learning model and based on the candidate data caching scheme corresponding to a higher performance evaluation score relative to other performance evaluation scores in the trained reinforcement learning model.
10. The electronic device according to claim 9 , wherein the simulated-data read request is received from the environment in the reinforcement learning model.
11. The electronic device according to claim 9 , wherein the data caching scheme indicates at least one of the following:
a cache in which the simulated data is to be cached; and
cached data to be cleared from the cache, wherein the remaining available space of the cache in which the simulated data is to be cached is insufficient.
12. The electronic device according to claim 9 , wherein training the reinforcement learning model includes:
acquiring an initial data caching scheme; and
training the reinforcement learning model based on the initial data caching scheme.
13. The electronic device according to claim 9 , wherein training the reinforcement learning model includes:
acquiring restrictions on data caching schemes; and
training the reinforcement learning model based on the restrictions.
14. The electronic device according to claim 9 , wherein training the reinforcement learning model includes:
training the reinforcement learning model such that the performance evaluation in the trained reinforcement learning model is better than a performance evaluation of a predetermined candidate data caching scheme with regard to the simulated-data read request.
15. The electronic device according to claim 9 , wherein the performance evaluation is indicated by a performance evaluation score which includes a weighted sum of sub-performance evaluation scores of multiple sub-performance evaluations, and training the reinforcement learning model includes:
training the reinforcement learning model such that the performance evaluation score in the trained reinforcement learning model is higher than a predetermined performance evaluation score.
16. A computer program product tangibly stored on a non-transitory computer-readable medium and including machine-executable instructions, wherein the machine-executable instructions, when executed, cause a machine to perform steps of a method, the method comprising:
acquiring storage information associated with a simulated network environment, wherein the simulated network environment includes multiple nodes each having a cache, and the storage information indicates the usage of the cache of each node; and
training a reinforcement learning model using simulated data and based on a simulated-data read request for a node among the multiple nodes, wherein the reinforcement learning model utilizes one of a first reinforcement learning algorithm that incorporates an environment model and a second reinforcement learning algorithm that does not incorporate an environment model, and wherein the reinforcement learning model includes an environment corresponding to the simulated network environment, the reinforcement learning model includes an agent corresponding to the multiple nodes, a state corresponding to the storage information, an action corresponding to a candidate data caching scheme, and a reward corresponding to a performance evaluation of the candidate data caching scheme with regard to the simulated-data read request, and the reinforcement learning model is trained such that the performance evaluation in the reinforcement learning model satisfies a predetermined performance evaluation requirement, and such that the reinforcement learning model can be used for determining a data caching scheme for input data;
wherein acquiring the storage information includes acquiring the storage information and topological information of the multiple nodes;
wherein the reinforcement learning model includes the state corresponding to the storage information and the topological information;
wherein the performance evaluation is indicated by a performance evaluation score which includes a weighted sum of sub-performance evaluation scores of multiple sub-performance evaluations; and
wherein training the reinforcement learning model includes determining a data caching scheme using the trained reinforcement learning model and based on the candidate data caching scheme corresponding to a higher performance evaluation score relative to other performance evaluation scores in the trained reinforcement learning model.
17. The computer program product according to claim 16 , wherein the data caching scheme indicates at least one of the following:
a cache in which the simulated data is to be cached; and
cached data to be cleared from the cache, wherein the remaining available space of the cache in which the simulated data is to be cached is insufficient.
18. The computer program product according to claim 16 , wherein training the reinforcement learning model includes:
acquiring an initial data caching scheme; and
training the reinforcement learning model based on the initial data caching scheme.
19. The computer program product according to claim 16 , wherein training the reinforcement learning model includes:
acquiring restrictions on data caching schemes; and
training the reinforcement learning model based on the restrictions.
20. The computer program product according to claim 16 , wherein training the reinforcement learning model includes:
training the reinforcement learning model such that the performance evaluation in the trained reinforcement learning model is better than a performance evaluation of a predetermined candidate data caching scheme with regard to the simulated-data read request.