Health reporting for computing devices
Systems and methods for reporting health status for a plurality of computing devices such as within a data center are disclosed. A management device connected to the computing devices via a network executes a management application that periodically requests and receives status data from the computing devices. A pool checker may be used to track corresponding pool status data, and an environment checker may periodically request and receive environmental data from environmental sensors for temperature, humidity, and audio. A report generator creates health reports and assigns device health classifications based on the device status data, the environmental data, and the pool health data. The data may be associated with one or more locations and customers, permitting filtering of the report.
1. A method for reporting health of a plurality of computing devices, the method comprising:
periodically requesting device status data from one or more of the computing devices;
periodically requesting pool status data from a pool to which at least a subset of the computing devices are contributing work;
periodically requesting environmental status data from one or more environmental sensors;
receiving status data responses from the computing devices, the pool, and the one or more environmental sensors;
calculating a health status classification for each computing device based on the received status responses; and
creating a health report including the health status classifications, device error information based on the device status data, environmental health information based on the environmental status data, pool health information based on the pool status data, and luck information based on the pool status data; and
wherein calculating the health status classification for each computing device includes assigning a computing device:
an optimal classification if all of the received status responses are within a predetermined normal range;
a degraded classification if a plurality of the received status responses are out of the predetermined normal range; and
a repeat offender classification if (i) one or more of the received status responses is out of the predetermined normal range more than a predetermined number of times within a predetermined timeframe and/or (ii) the device status data indicates that more than a predetermined number of errors has occurred.
2. The method of claim 1 , wherein the device status data includes a computing device's hash rate, error list, fan speed, processor voltage, processor frequency, memory voltage, and memory operating frequency.
3. The method of claim 1 , wherein the pool status data includes a number of shares accepted, a number of shares rejected, a number of blocks solved, and difficulty.
4. The method of claim 1 , wherein the environmental sensors comprise temperature sensors, and wherein the environmental status data comprises temperature data, the method further comprising:
comparing temperature data from the temperature sensors to a predefined safe temperature range; and
adjusting the health classifications for computing devices associated with out of safe temperature range temperature data.
5. The method of claim 1 , wherein the environmental sensors comprise humidity sensors, and wherein the environmental status data comprises humidity data, method further comprising:
comparing humidity data from the humidity sensors to a predefined safe humidity range; and
adjusting the health classifications for computing devices associated with out of safe humidity range humidity data.
6. The method of claim 1 , wherein the environmental sensors comprise one or more microphones, and wherein the method further comprises:
capturing from the microphones a baseline known good audio signal representative of at least a subset of the computing devices operating in a known good state; and
periodically capturing real-time audio signals from the microphones; and
comparing the real-time audio signals with the baseline known good audio signal to determine a signal difference; and
adjusting the health classifications for computing devices associated with audio signals having a signal difference outside a predefined safe range.
7. The method of claim 1 , further comprising:
associating the received environmental status data with one or more of the computing devices based on the proximity of the location of the computing device to the environmental sensors;
associating the received pool status data with one or more of the computing devices contributing work to the pool originating the received pool status data; and
associating the received device status data with the device originating the received device status data.
8. The method of claim 6 , wherein:
the pool status data comprises a number of accepted shares and difficulty; and
the device status data comprises a hash rate of the computing device.
9. The method of claim 7 , wherein the health report is a presented via a page comprising controls permitting a user to filter the health report based on one or more locations.
10. A system for managing a plurality of networked computing devices in one or more data centers, the system comprising:
a management application executing on a management device connected to the plurality of networked computing devices, wherein the management application comprises:
a device status checker that periodically requests and receives status information from the computing devices,
an environment checker that periodically requests and receives environmental information from one or more environmental sensors within the data center, and
a report generator that calculates a health status classification for each computing device and creates a health report based on the received status information and received environmental information;
wherein the environmental sensors further comprise one or more microphones positioned to capture sound from the computing devices;
wherein the environment checker is configured to:
(i) periodically capture real-time audio signals from the microphones,
(ii) compare them with a known good baseline audio signal representative of normal operation of the computing devices, and
(iii) generate a warning when the difference is outside an expected range,
wherein the report generator is configured to incorporate the warning into the health reports and health status classifications.
11. The system of claim 10 , further comprising a pool checker that periodically requests and receives pool performance information from one or more pools associated with the computing devices, wherein the report generator incorporates the pool performance information into the health reports and health status classifications.
12. The system of claim 10 , wherein the computing devices are positioned in the one or more data centers to intake air from a cool aisle and exhaust air to a hot aisle, wherein the environmental sensors comprise at least one of:
temperature sensors positioned to measure hot aisle and cold aisle temperatures; and
humidity sensors positioned to measure hot aisle and cold aisle humidity.
13. The system of claim 10 , wherein the management application further comprises a luck checker, wherein the luck checker tracks luck information for each of the computing devices, wherein the luck information comprises a number of blocks solved, wherein the report generator incorporates the luck information into the health reports.
14. A non-transitory, computer-readable storage medium storing instructions executable by a processor of a computational device, which when executed cause the computational device to:
periodically request device health status data from one or more of a plurality of computing devices connected via a network;
periodically request pool health status data from a computing pool associated with at least a subset of the plurality of computing devices;
periodically request environmental health status data from one or more environmental sensors;
receive responses from the computing devices, the computing pool, and the one or more environmental sensors;
calculate health classifications for the computing devices based on the responses;
create a health report based on the received responses, wherein the health report comprises alerts for out of normal range computing device health status data, environmental health status data, and pool health status data; and
assign a computing device:
an optimal classification if all of the received responses are within a predetermined normal range;
a degraded classification if a plurality of the received responses are out of the predetermined normal range; and
a repeat offender classification if (i) one or more of the received responses is out of the predetermined normal range more than a predetermined number of times within a predetermined timeframe and/or (ii) the device health status data indicates that more than a predetermined number of errors has occurred;
wherein the pool health status data includes a number of shares rejected.
15. The medium of claim 14 , further comprising instructions which when executed cause the computational device to periodically request and receive pool performance information from one or more pools associated with the computing devices, wherein the health report incorporates the pool performance information.
16. The medium of claim 15 , further comprising instructions which when executed cause the computational device to track luck information for each of the computing devices, wherein the luck information comprises a number of blocks solved, wherein the health report incorporates the luck information.
17. The medium of claim 15 , wherein the environmental sensors comprise temperature sensors, wherein the health report incorporates temperature information from the temperature sensors.
18. The medium of claim 15 , further comprising instructions which when executed cause the computational device to associate location information with the environmental health status data.
19. The medium of claim 14 , further comprising instructions which when executed cause the computational device to:
periodically capture real-time audio signals from one or more microphones positioned to capture sound from the computing devices,
compare the captured real-time audio signals with a known good baseline audio signal representative of normal operation of the computing devices, and
generate a warning when a difference between the captured real-time audio signals and the baseline audio signal is outside an expected range.