Systems and methods for low-overhead and flexible trace capture triggered on-demand by hardware events under production load

1. A method comprising:

configuring a plurality of watcher processes for observing and logging performance of one or more storage devices, wherein each watcher process is configured with a trigger condition and a resource limit and organized into tiers based on resource limits for the trigger conditions;

initiating a first watcher process of a first tier to observe one of the one or more storage devices;

monitoring, with a watcher service, the first watcher process for the trigger condition of the first watcher process;

in response to detecting the trigger condition, processing an output of the first watcher process to determine a debug procedure;

selecting a second watcher process of a second tier indicated by the debug procedure, wherein the second tier corresponds to a higher resource limit for trigger conditions than the first tier; and

initiating the second watcher process.

2. The method of claim 1 , further comprising terminating the first watcher process in response to at least one of:

exceeding the resource limit of the first watcher process;

reaching a time threshold before detecting the trigger condition; or

detecting early termination of another watcher process.

3. The method of claim 1 , further comprising:

detecting the first watcher process exceeding the resource limit; and

throttling the first watcher process.

4. The method of claim 1 , wherein initiating the second watcher process further comprises:

selecting, based on the processed output, the second watcher process from the second tier; and

initiating the second watcher process in response to the selection.

5. The method of claim 1 , wherein initiating the second watcher process further comprises:

determining whether a counter associated with the second tier satisfies a threshold number;

initiating the second watcher process when the counter does not satisfy the threshold number; and

increasing the counter.

6. The method of claim 1 , further comprising:

initiating a third watcher process of the first tier to observe a second of the one or more storage devices;

wherein initiating the second watcher process is further based on detecting the trigger condition of the third watcher process.

7. The method of claim 1 , further comprising:

monitoring, with the watcher service, the second watcher process for the trigger condition of the second watcher process;

in response to detecting the trigger condition of the second watcher process, processing an output of the second watcher process; and

initiating, based on the processed output of the second watcher process, a diagnostic action for the one of the one or more storage devices.

8. The method of claim 1 , wherein the trigger condition includes detecting a threshold number of latency events within a time window.

9. A system comprising:

at least one physical processor; and

physical memory comprising computer-executable instructions that, when executed by the physical processor, cause the physical processor to:

configure a plurality of watcher processes for observing and logging performance of one or more storage devices, wherein each watcher process is configured with a trigger condition and a resource limit and organized into tiers based on resource limits for the trigger conditions;

initiate a first watcher process of a first tier to observe one of the one or more storage devices;

monitor, with a watcher service, the first watcher process for the trigger condition of the first watcher process;

in response to detecting the trigger condition, process an output of the first watcher process to determine a debug procedure;

select a second watcher process of a second tier indicated by the debug procedure, wherein the second tier corresponds to a higher resource limit for trigger conditions than the first tier; and

initiate the second watcher process.

10. The system of claim 9 , further comprising instructions for terminating the first watcher process in response to at least one of:

exceeding the resource limit of the first watcher process;

reaching a time threshold before detecting the trigger condition; or

detecting early termination of another watcher process.

11. The system of claim 9 , further comprising instructions for:

detecting the first watcher process exceeding the resource limit; and

throttling the first watcher process.

12. The system of claim 9 , wherein instructions for initiating the second watcher process further comprises instructions for:

selecting, based on the processed output, the second watcher process from the second tier; and

initiating the second watcher process in response to the selection.

13. The system of claim 9 , wherein instructions for initiating the second watcher process further comprises instructions for:

determining whether a counter associated with the second tier satisfies a threshold number;

initiating the second watcher process when the counter does not satisfy the threshold number; and

increasing the counter.

14. The system of claim 9 , further comprising instructions for:

initiating a third watcher process of the first tier to observe a second of the one or more storage devices;

wherein initiating the second watcher process is further based on detecting the trigger condition of the third watcher process.

15. The system of claim 9 , further comprising instructions for:

monitoring, with the watcher service, the second watcher process for the trigger condition of the second watcher process;

in response to detecting the trigger condition of the second watcher process, processing an output of the second watcher process; and

initiating, based on the processed output of the second watcher process, a diagnostic action for the one of the one or more storage devices.

16. A non-transitory computer-readable medium comprising one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to:

configure a plurality of watcher processes for observing and logging performance of one or more storage devices, wherein each watcher process is configured with a trigger condition and a resource limit and organized into tiers based on resource limits for the trigger conditions;

initiate a first watcher process of a first tier to observe one of the one or more storage devices;

monitor, with a watcher service, the first watcher process for the trigger condition of the first watcher process;

in response to detecting the trigger condition, process an output of the first watcher process to determine a debug procedure;

select a second watcher process of a second tier indicated by the debug procedure, wherein the second tier corresponds to a higher resource limit for trigger conditions than the first tier; and

initiate the second watcher process.

17. The non-transitory computer-readable medium of claim 16 , further comprising instructions for terminating the first watcher process in response to at least one of:

exceeding the resource limit of the first watcher process;

reaching a time threshold before detecting the trigger condition; or

detecting early termination of another watcher process.

18. The non-transitory computer-readable medium of claim 16 , wherein initiating the second watcher process further comprises instructions for:

selecting, based on the processed output, the second watcher process from the second tier; and

initiating the second watcher process in response to the selection.

19. The non-transitory computer-readable medium of claim 16 , wherein instructions for initiating the second watcher process further comprises instructions for:

determining whether a counter associated with the second tier satisfies a threshold number;

initiating the second watcher process when the counter does not satisfy the threshold number; and

increasing the counter.

20. The non-transitory computer-readable medium of claim 16 , further comprising instructions for:

monitoring, with the watcher service, the second watcher process for the trigger condition of the second watcher process;

in response to detecting the trigger condition of the second watcher process, processing an output of the second watcher process; and

initiating, based on the processed output of the second watcher process, a diagnostic action for the one of the one or more storage devices.