Distributed storage workload management
Workloads, e.g., synthetic workloads, on one or more storage systems in an dynamic, automated manner, for example, to load test the one or more storage systems. A distributed system may be employed in which a workload information server (WIS) serves one or more clients referred to herein as workload control components (WCCs) that analyze workload information of the one or more storage systems, and control the modification of workloads thereon based on this analysis, through the WIS. The WIS also may serve one or more clients referred to herein as workload generation controllers (WGCs) that monitor workloads on the one or more storage systems, report workload information to the WIS and generate, modify or remove workloads on the one or more storage systems according to instructions received from the WIS in response to requests (e.g., hints) from the one or more WGCs.
1. A method comprising:
one or more workload generation tools reporting workload information corresponding to a plurality of workloads executing on the storage system, wherein the plurality of workloads includes a first controlled workload that is controlled by a set of components including the one or more workload generation tools, and wherein the plurality of workloads includes a first independent workload that is not controlled by the set of components;
determining from the workload information a resource utilization value indicative of an amount of resources of the storage system utilized by the plurality of workloads executing on the storage system;
comparing the determined utilization value to a predefined target utilization value;
determining, based at least in part on the comparison, whether to modify a cumulative workload on the storage system, wherein the cumulative workload includes the plurality of workloads having the corresponding workload information reported by the one or more workload generation tools; and
if it is determined to modify the cumulative workload, performing first processing including:
sending an instruction to at least one of the one or more workload generation tools to initiate a modification of the cumulative workload on the storage system; and
responsive to receiving the instruction, the at least one workload generation tool performing one or more tasks that modify the cumulative workload on the storage system, including modifying the first controlled workload.
2. The method of claim 1 , wherein the workload information includes performance information corresponding to the plurality of workloads, and the method further comprises:
selecting the target utilization value based at least in part on the performance information.
3. The method of claim 1 , wherein the workload information includes, for at least the first controlled workload of the plurality of workloads, a first resource utilization value indicative of a first amount of resources of the storage system utilized by the first controlled workload, wherein the method further comprises:
comparing the first utilization value to a predefined first utilization threshold of the first controlled workload, and
wherein determining whether to modify the cumulative workload on the storage system is based at least in part on the comparison of the first utilization value to the predefined first utilization threshold.
4. The method of claim 1 , wherein the instruction to at least one of the one or more workload generation tools indicates to perform any of: add a new controlled workload to the storage system, and remove a controlled workload from the storage system.
5. The method of claim 1 , wherein the instruction to at least one of the one or more workload generation tools indicates to increase and/or decrease an amount of I/O activity of one or more controlled workloads of the plurality of workloads.
6. The method of claim 1 , further comprising:
the one or more workload generation tools sending one or more requests or queries for workload tasks, wherein the instruction to the at least one of the one or more workload generation tools is sent in response to at least one of the one or more requests or queries.
7. The method of claim 1 , wherein the set of components further includes a first component and a second component, wherein the one or more workload generation tools report the workload information to the first component, wherein the method further comprises:
the second component sending a request for workload information to the first component; and
the first component sending at least a portion of the workload information to the second component in response to the request,
wherein the second component performs the comparing and the determining, wherein the method further comprises:
the second component sending a request to the first component to instruct the at least one workload generation tool to initiate the modification of the cumulative workload on the storage system, and
wherein the first component sends the instruction in response to the request.
8. A system comprising:
one or more workload generation tools reporting workload information corresponding to a plurality of workloads executing on the storage system, wherein the plurality of workloads includes a first controlled workload that is controlled by a set of components including the one or more workload generation tools, and wherein the plurality of workloads includes a first independent workload that is not controlled by the set of components; and
one or more memories comprising code stored therein that, when executed, performs a method including:
determining from the workload information a resource utilization value indicative of an amount of resources of the storage system utilized by the plurality of workloads executing on the storage system;
comparing the determined utilization value to a predefined target utilization value;
determining, based at least in part on the comparison, whether to modify a cumulative workload on the storage system, wherein the cumulative workload includes the plurality of workloads having the corresponding workload information reported by the one or more workload generation tools; and
if it is determined to modify the cumulative workload, performing first processing including:
sending an instruction to at least one of the one or more workload generation tools to initiate a modification of the cumulative workload on the storage system; and
responsive to receiving the instruction, the at least one workload generation tool performing one or more tasks that modify the cumulative workload on the storage system, including modifying the first controlled workload.
9. The system of claim 8 , wherein the workload information includes performance information corresponding to the plurality of workloads, and wherein the method further comprises:
selecting the target utilization value based at least in part on the performance information.
10. The system of claim 8 , wherein the workload information includes, for at least the first controlled workload of the plurality of workloads, a first resource utilization value indicative of a first amount of resources of the storage system utilized by the first controlled workload, wherein the method further comprises:
comparing the first utilization value to a predefined first utilization threshold of the first controlled workload, and
wherein determining whether to modify the cumulative workload on the storage system is based at least in part on the comparison of the first utilization value to the predefined first utilization threshold.
11. The system of claim 8 , wherein the instruction to at least one of the one or more workload generation tools indicates to perform any of: add a new controlled workload to the storage system, and remove a controlled workload from the storage system.
12. The system of claim 8 , wherein the instruction to at least one of the one or more workload generation tools indicates to increase and/or decrease an amount of I/O activity of one or more controlled workloads of the plurality of workloads.
13. One or more non-transitory computer readable media having code stored thereon that, when executed, performs a method comprising:
one or more workload generation tools reporting workload information corresponding to a plurality of workloads executing on the storage system, wherein the plurality of workloads includes a first controlled workload that is controlled by a set of components including the one or more workload generation tools, and wherein the plurality of workloads includes a first independent workload that is not controlled by the set of components;
determining from the workload information a resource utilization value indicative of an amount of resources of the storage system utilized by the plurality of workloads executing on the storage system;
comparing the determined utilization value to a predefined target utilization value;
determining, based at least in part on the comparison, whether to modify a cumulative workload on the storage system, wherein the cumulative workload includes the plurality of workloads having the corresponding workload information reported by the one or more workload generation tools; and
if it is determined to modify the cumulative workload, performing first processing including:
sending an instruction to at least one of the one or more workload generation tools to initiate a modification of the cumulative workload on the storage system; and
responsive to receiving the instruction, the at least one workload generation tool performing one or more tasks that modify the cumulative workload on the storage system, including modifying the first controlled workload.
14. The one or more non-transitory computer-readable media of claim 13 , wherein the workload information includes performance information corresponding to the plurality of workloads, and wherein the method further comprises:
selecting the target utilization value based at least in part on the performance information.
15. The one or more non-transitory computer-readable media of claim 13 , wherein the workload information includes, for at least a first workload of the one or more workloads, a first resource utilization value indicative of a first amount of resources of the first storage system utilized by the first workload, and wherein the method further comprises:
comparing the first utilization value to a predefined first utilization threshold of the first workload, and
wherein determining whether to modify the cumulative workload on the storage system is based at least in part on the comparison of the first utilization value to the predefined first utilization threshold.
16. The one or more non-transitory computer-readable media of claim 13 , wherein the method further comprises:
the one or more workload generation tools sending one or more requests or queries for workload tasks, wherein the instruction to the at least one of the one or more workload generation tools is sent in response to at least one of the one or more requests or queries.
17. The one or more non-transitory computer-readable media of claim 13 , wherein the set of components includes a first component and a second component, and wherein the one or more workload generation tools report the workload information to the first component, and wherein the method further comprises:
the second component sending a request for workload information to the first component; and
the first component sending at least a portion of the workload information to the second component in response to the request,
wherein the second component performs the comparing and the determining,
wherein the method further comprises:
sending a request to the first component to instruct the at least one workload generation tool to initiate the modification of the cumulative workload on the storage system, and wherein the first component sends the instruction in response to the request.