Profiling a job power and energy consumption for a data processing system
A process identifier for a job is collected. The job runs on a plurality of nodes. The job is identified using the process identifier. A node for the job is identified. An amount of power consumed by the node to run the job is determined.
1. A method to profile a job power performed on a data processing system comprising a plurality of shared nodes and a controller node, comprising:
collecting, using the controller node, a plurality of process identifiers sent from the plurality of shared nodes;
determining, using the controller node, identifiers for a plurality of jobs including a first job and a second job based on the plurality of process identifiers, wherein a job is a computer program that is executed on at least a portion of the plurality of shared nodes;
identifying, using the controller node, the shared nodes including a first shared node and a second shared node that are used to execute the plurality of jobs based on the job identifiers, wherein a table is generated that includes a number of accesses of the first shared node by the first job and a number of accesses of the first shared node by the second job;
and
calculating, using the controller node, an amount of power consumed by the first shared node to execute the first job, an amount of power consumed by the first shared node to execute the second job, and an amount of power consumed by the second shared node to execute the first job based on the number of accesses to control the job power to increase performance and throughput within power limits of the system.
2. The method of claim 1 , further comprising
determining a start time of the job; and
determining an end time of the job.
3. The method of claim 1 , further comprising
obtaining samples of a process associated with the job, wherein the samples comprise one or more power samples, one or more time samples, or any combination thereof.
4. The method of claim 1 , further comprising
generating a map between the process identifiers and the job identifiers.
5. The method of claim 1 , further comprising
identifying a shared resource used by the job.
6. The method of claim 1 , further comprising
measuring the amount of power consumed by each of the shared nodes.
7. A method to profile a job power for a data processing system comprising a plurality of shared nodes and a head node, comprising:
sampling a plurality of processes for a plurality of jobs including a first job and a second job running on at least a portion of the plurality of shared nodes including a first shared node and a second shared node to obtain one or more samples;
generating, a timestamp for each of the first job and the second job to create a log record comprising the one or more samples indicating a number of accesses of the first shared node by the first job and a number of accesses of the first shared node by the second job;
attaching a node identifier to the log record; and
sending the node identifier with the log record to the head node to calculate an amount of power consumed by the jobs to control the job power to increase performance and throughput within power limits of the system, wherein the sampling, generating, attaching and sending are performed by each of the first shared node and the second shared node.
8. The method of claim 7 , wherein the process is sampled at a predetermined time and wherein the one or more samples comprise one or more power samples, one or more time samples, or any combination thereof.
9. The method of claim 7 , wherein the node identifier attached to the log record is sent to the head node at a predetermined time.
10. The method of claim 7 , further comprising
intercepting an access request from the process of the job.
11. The method of claim 7 , further comprising
identifying a shared resource used by the process.
12. A non-transitory machine readable medium comprising
instructions that cause a data processing system comprising a plurality of shared nodes and a controller node to perform operations comprising:
collecting, using the controller node, a plurality of process identifiers sent from the plurality of shared nodes;
determining, using the controller node, identifiers for a plurality of jobs including a first job and a second job based on the plurality of process identifiers, wherein a job is a computer program that is executed on at least a portion of the plurality of shared nodes;
identifying, using the controller node, the shared nodes including a first shared node and a second shared node that are used to execute the plurality of jobs based on the job identifiers, wherein a table is generated that includes a number of accesses of the first shared node by the first job and a number of accesses of the first shared node by the second job; and
calculating, using the controller node, an amount of power consumed by the first shared node to execute the first job, an amount of power consumed by the first shared node to execute the second job, and an amount of power consumed by the second shared node to execute the first job based on the number of accesses to control a job power to increase performance and throughput within power limits of the system.
13. The non-transitory machine readable medium of claim 12 , further comprising instructions that cause the data processing system to perform operations comprising
determining a start time of the job; and
determining an end time of the job.
14. The non-transitory machine readable medium of claim 12 , further comprising instructions that cause the data processing system to perform operations comprising
obtaining samples of a process associated with the job, wherein the samples comprise one or power samples, one or more time samples, or any combination thereof.
15. The non-transitory machine readable medium of claim 12 , further comprising instructions that cause the data processing system to perform operations comprising
generating a map between the process identifiers and the job identifiers.
16. The non-transitory machine readable medium of claim 12 , further comprising instructions that cause the data processing system to perform operations comprising
identifying a shared resource used by the job.
17. The non-transitory machine readable medium of claim 12 , further comprising instructions that cause the data processing system to perform operations comprising
measuring an amount of power consumed by each of the shared nodes.
18. A data processing system, comprising:
a plurality of shared nodes;
a controller node coupled to the plurality of shared nodes;
a memory coupled to the controller node; and the controller node comprising a processor, wherein the processor is configured to collect a plurality of process identifiers sent from the plurality of shared nodes, wherein the processor is configured to determine identifiers for a plurality of jobs including a first job and a second job based on the plurality of process identifiers, wherein a job is a computer program that is executed on at least a portion of the plurality of shared nodes, wherein the processor is configured to identify the shared nodes including a first shared node and a second shared node that are used to execute the plurality of jobs based on the job identifiers, wherein a table is generated that includes a number of accesses of the first shared node by the first job and a number of accesses of the first shared node by the second job; and wherein the processor is configured to calculate an amount of power consumed by the first shared node to execute the first job, an amount of power consumed by the first shared node to execute the second job and an amount of power consumed by the second shared node to execute the first job based on the number of accesses to control a job power to increase performance and throughput within power limits of the system.
19. The data processing system of claim 18 , wherein the processor is further configured to determine a start time of the job; and wherein the processor is further configured to determine an end time of the job.
20. The data processing system of claim 18 , wherein the processor is further configured to obtain samples of a process associated with the job, wherein the samples comprise one or more power samples, one or more time samples, or any combination thereof.
21. The data processing system of claim 18 , wherein the processor is further configured to generate a map between the process identifiers and the job identifiers.
22. The data processing system of claim 18 , wherein the processor is further configured to identify a shared resource used by the job.