Anomaly aware log retrieval from disk array enclosures (DAEs)
Topology and performance metrics of a storage system are monitored for anomalies. The storage system includes a set of disk array enclosures (DAEs) connected to a host server. Each DAE is chained to another DAE. Upon detecting an anomaly associated with a DAE, log collection is triggered to obtain logs from the DAE and logs in other DAEs upstream and downstream of the DAE.
1. A method comprising:
monitoring a topology of a storage system, the storage system comprising a plurality of disk array enclosures (DAEs) coupled to a host server, each DAE being chained to another DAE by first and second chains, and comprising a serial attached SCSI (SAS) expander and a peer SAS expander, the SAS expander in a DAE forming part of the first chain and the peer SAS expander in the DAE forming part of the second chain;
monitoring performance metrics of the DAEs;
detecting, from the monitoring of the topology and the performance metrics, an anomaly associated with one of a SAS expander or a peer SAS expander in a particular DAE, the one of the SAS expander or the peer SAS expander being a problematic SAS expander, the problematic SAS expander in the particular DAE being in one of the first or second chains; and
upon the detecting, triggering DAE log collections to obtain a plurality of logs, the plurality of logs comprising a log stored at the problematic SAS expander in the particular DAE, and logs stored at other SAS expanders in other DAEs upstream and downstream from the particular DAE and being in the one of the first or second chains.
2. The method of claim 1 further comprising:
tagging the plurality of logs with an anomaly class; and
storing the collected logs in a repository accessible by the host server and separate from the DAEs.
3. The method of claim 1 further comprising:
tagging the plurality of logs with an anomaly class; and
limiting, for each anomaly class, a number of logs stored.
4. The method of claim 1 wherein the DAEs are provided by a third party.
5. The method of claim 1 wherein the monitoring the topology of the storage system further comprises:
tracking a number of changes in the topology of the storage system over a predetermined period of time; and
when the number of changes over the predetermined period of time exceeds a threshold, determining that the DAE log collections should be triggered.
6. The method of claim 1 wherein the monitoring the performance metrics of the DAEs further comprises:
identifying a redundant array of inexpensive disks (RAID) group within the plurality of DAEs;
obtaining input/output (IO) latency for each drive in the RAID group;
calculating a population mean of the IO latency;
generating a z-score for each drive in the RAID group based on the population mean; and
determining that DAE log collections should be triggered when each of first and second conditions are satisfied, wherein the first condition is satisfied when a particular drive in the RAID group has an IO latency that exceeds a threshold percentage of the population mean for a threshold number of consecutive days, and
wherein the second condition is satisfied when the particular drive in the RAID group has a z-score that exceeds a threshold z-score for the threshold number of consecutive days.
7. A system comprising: a processor; and memory configured to store one or more sequences of instructions which, when executed by the processor, cause the processor to carry out the steps of:
monitoring a topology of a storage system, the storage system comprising a plurality of disk array enclosures (DAEs) coupled to a host server, each DAE being chained to another DAE by first and second chains, and comprising a serial attached SCSI (SAS) expander and a peer SAS expander, the SAS expander in a DAE forming part of the first chain and the peer SAS expander in the DAE forming part of the second chain;
monitoring performance metrics of the DAEs;
detecting, from the monitoring of the topology and the performance metrics, an anomaly associated with one of a SAS expander or a peer SAS expander in a particular DAE, the one of the SAS expander or the peer SAS expander being a problematic SAS expander, the problematic SAS expander in the particular DAE being in one of the first or second chains; and
upon the detecting, triggering DAE log collections to obtain a plurality of logs, the plurality of logs comprising a log stored at the problematic SAS expander in the particular DAE, and logs stored at other SAS expanders in other DAEs upstream and downstream from the particular DAE and being in the one of the first or second chains.
8. The system of claim 7 wherein the processor further carries out the steps of:
tagging the plurality of logs with an anomaly class; and
storing the collected logs in a repository accessible by the host server and separate from the DAEs.
9. The system of claim 7 wherein the processor further carries out the steps of:
tagging the plurality of logs with an anomaly class; and
limiting, for each anomaly class, a number of logs stored.
10. The system of claim 7 wherein the DAEs are provided by a third party.
11. The system of claim 7 wherein the monitoring the topology of the storage system further comprises:
tracking a number of changes in the topology of the storage system over a predetermined period of time; and
when the number of changes over the predetermined period of time exceeds a threshold, determining that the DAE log collections should be triggered.
12. The system of claim 7 wherein the monitoring the performance metrics of the DAEs further comprises:
identifying a redundant array of inexpensive disks (RAID) group within the plurality of DAEs;
obtaining input/output (TO) latency for each drive in the RAID group;
calculating a population mean of the IO latency;
generating a z-score for each drive in the RAID group based on the population mean; and
determining that DAE log collections should be triggered when each of first and second conditions are satisfied, wherein the first condition is satisfied when a particular drive in the RAID group has an IO latency that exceeds a threshold percentage of the population mean for a threshold number of consecutive days, and
wherein the second condition is satisfied when the particular drive in the RAID group has a z-score that exceeds a threshold z-score for the threshold number of consecutive days.
13. A computer program product, comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein, the computer-readable program code adapted to be executed by one or more processors to implement a method comprising:
monitoring a topology of a storage system, the storage system comprising a plurality of disk array enclosures (DAEs) coupled to a host server, coupled to the host server, each DAE being chained to another DAE by first and second chains, and comprising a serial attached SCSI (SAS) expander and a peer SAS expander, the SAS expander in a DAE forming part of the first chain and the peer SAS expander in the DAE forming part of the second chain;
monitoring performance metrics of the DAEs;
detecting, from the monitoring of the topology and the performance metrics, an anomaly associated with one of a SAS expander or a peer SAS expander in a particular DAE, the one of the SAS expander or the peer SAS expander being a problematic SAS expander, the problematic SAS expander in the particular DAE being in one of the first or second chains; and
upon the detecting, triggering DAE log collections to obtain a plurality of logs, the plurality of logs comprising a log stored at the problematic SAS expander in the particular DAE, and logs stored at other SAS expanders in other DAEs upstream and downstream from the particular DAE and being in the one of the first or second chains.
14. The computer program product of claim 13 wherein the method further comprises:
tagging the plurality of logs with an anomaly class; and
storing the collected logs in a repository accessible by the host server and separate from the DAEs.
15. The computer program product of claim 13 wherein the method further comprises:
tagging the plurality of logs with an anomaly class; and
limiting, for each anomaly class, a number of logs stored.
16. The computer program product of claim 13 wherein the DAEs are provided by a third party.
17. The computer program product of claim 13 wherein the monitoring the topology of the storage system further comprises:
tracking a number of changes in the topology of the storage system over a predetermined period of time; and
when the number of changes over the predetermined period of time exceeds a threshold, determining that the DAE log collections should be triggered.
18. The computer program product of claim 13 wherein the monitoring the performance metrics of the DAEs further comprises:
identifying a redundant array of inexpensive disks (RAID) group within the plurality of DAEs;
obtaining input/output (TO) latency for each drive in the RAID group;
calculating a population mean of the IO latency;
generating a z-score for each drive in the RAID group based on the population mean; and
determining that DAE log collections should be triggered when each of first and second conditions are satisfied, wherein the first condition is satisfied when a particular drive in the RAID group has an IO latency that exceeds a threshold percentage of the population mean for a threshold number of consecutive days, and
wherein the second condition is satisfied when the particular drive in the RAID group has a z-score that exceeds a threshold z-score for the threshold number of consecutive days.