Ingest health monitoring with failure indicator
Ingest health monitoring includes receiving an event stream of events to store on at least one storage system and obtaining an event from the event stream. Ingest health monitoring further includes transmitting the event to a selected ingest module queue for the event, updating an output rate indicator counter for the selected ingest module queue when failure to store the event in the ingest module queue occurs, obtaining the event from the selected ingest module queue, processing the event to generate a file for the event, and transmitting the file to the at least one storage system. Ingest health monitoring further includes updating the write failure indicator counter for a storage system of the at least one storage system when failure to transmit to the storage system occurs and updating the user interface based on the output rate indicator counter and the write failure indicator counter.
1. A computer-implemented method comprising:
receiving an event stream of events in a data intake and query system to store on at least one storage system;
obtaining a first event from the event stream;
processing the first event to generate a first file for the event;
transmitting the first file to the at least one storage system;
updating a write failure indicator counter for a storage system of the at least one storage system when failure to transmit the first file to the storage system occurs;
generating a write failure status by performing a first comparison of the write failure indicator counter with a write failure indicator threshold; and
updating a user interface with the write failure status based on the first comparison.
2. The computer-implemented method of claim 1 , further comprising:
obtaining a second event from the event stream;
processing the second event to generate a second file for the event;
transmitting the second file to the at least one storage system;
updating a write failure indicator counter for a storage system of the at least one storage system when failure to transmit the second file to the storage system occurs; and
resetting the write failure indicator counter upon success of transmission of the second file to the storage system.
3. The computer-implemented method of claim 1 , wherein the write failure indicator threshold is a warning write failure indicator threshold, and wherein the method further comprises:
performing a second comparison of the write failure indicator counter with an error write failure indicator threshold, wherein the write failure status is further generated according to the second comparison; and
updating the user interface with the write failure status based on the second comparison.
4. The computer-implemented method of claim 1 , further comprising:
determining from the first comparison that the write failure indicator counter is above an error write failure indicator threshold, and from a second comparison that the write failure indicator counter is below a warning write failure indicator threshold; and
setting the write failure status as in warning status responsive to the determining.
5. The computer-implemented method of claim 1 , further comprising:
determining from the first comparison that the write failure indicator counter is above an error write failure indicator threshold; and
setting the write failure status as error mode status responsive to the determining.
6. The computer-implemented method of claim 1 , wherein the at least one storage system comprises a plurality of storage systems each related to a corresponding write failure indicator counter.
7. The computer-implemented method of claim 1 , further comprising:
displaying, in the user interface, a root cause of the write failure status comprising an explanation of a possible reason for the write failure status to exceed a write failure indicator threshold.
8. The computer-implemented method of claim 1 , further comprising:
displaying, in the user interface, a set of related messages from a file processor to a write failure tracker.
9. The computer-implemented method of claim 1 , further comprising:
storing a log entry comprising a date of the write failure indicator threshold being exceeded, a name of the write failure indicator counter that exceeded the write failure indicator threshold, a value of the write failure indicator counter, and the value of the write failure indicator threshold.
10. A computing device, comprising:
a processor; and
a non-transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform operations including:
receiving an event stream of events in a data intake and query system to store on at least one storage system,
obtaining a first event from the event stream,
processing the first event to generate a first file for the event,
transmitting the first file to the at least one storage system,
updating a write failure indicator counter for a storage system of the at least one storage system when failure to transmit the first file to the storage system occurs,
generating a write failure status by performing a first comparison of the write failure indicator counter with a write failure indicator threshold, and
updating a user interface with the write failure status based on the first comparison.
11. The computing device of claim 10 , wherein the operations further comprise:
obtaining a second event from the event stream;
processing the second event to generate a second file for the event;
transmitting the second file to the at least one storage system;
updating the write failure indicator counter for a storage system of the at least one storage system when failure to transmit the second file to the storage system occurs; and
resetting the write failure indicator counter upon success of transmission of the second file to the storage system.
12. The computing device of claim 10 , wherein the write failure indicator threshold is a warning write failure indicator threshold, and wherein the operations further comprise:
performing a second comparison of the write failure indicator counter with an error write failure indicator threshold, wherein the write failure status is further generated according to the second comparison; and
updating the user interface with the write failure status based on the second comparison.
13. The computing device of claim 10 , wherein the operations further comprise
determining from the first comparison that the write failure indicator counter is above an error write failure indicator threshold, and from a second comparison that the write failure indicator counter is below a warning write failure indicator threshold; and
setting the write failure status as in warning status responsive to the determining.
14. The computing device of claim 10 , wherein the operations further comprise:
determining from the first comparison that the write failure indicator counter is above an error write failure indicator threshold; and
setting the write failure status as error mode status responsive to the determining.
15. The computing device of claim 10 , wherein the at least one storage system comprises a plurality of storage systems each related to a corresponding write failure indicator counter.
16. The computing device of claim 10 , wherein the operations further comprise:
displaying, in the user interface, a root cause of the write failure status comprising an explanation of a possible reason for the write failure status to exceed a write failure indicator threshold.
17. The computing device of claim 10 , wherein the operations further comprise:
displaying, in the user interface, a set of related messages from a file processor to a write failure tracker.
18. The computing device of claim 10 , wherein the operations further comprise:
storing a log entry comprising a date of the write failure indicator threshold being exceeded, a name of the write failure indicator counter that exceeded the write failure indicator threshold, a value of the write failure indicator counter, and the value of the write failure indicator threshold.
19. A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processor to perform operations including:
receiving an event stream of events in a data intake and query system to store on at least one storage system;
obtaining a first event from the event stream;
processing the first event to generate a first file for the event;
transmitting the first file to the at least one storage system;
updating a write failure indicator counter for a storage system of the at least one storage system when failure to transmit the first file to the storage system occurs;
generating a write failure status by performing a first comparison of the write failure indicator counter with a write failure indicator threshold; and
updating a user interface with the write failure status based on the first comparison.
20. The non-transitory computer-readable medium of claim 19 , wherein the operations further comprise:
obtaining a second event from the event stream;
processing the second event to generate a second file for the event;
transmitting the second file to the at least one storage system;
updating the write failure indicator counter for a storage system of the at least one storage system when failure to transmit the second file to the storage system occurs; and
resetting the write failure indicator counter upon success of transmission of the second file to the storage system.