Edge-based data collection system for an observability pipeline system

1. A method performed by an edge agent of an observability pipeline system, the edge agent running on a computer node, the method comprising:

identifying processes running on the computer node;

identifying files on the computer node that the processes have opened for writing;

accessing log discovery parameters of the observability pipeline system;

selecting a plurality of files from the identified files according to the log discovery parameters;

generating a list of discovered log files, the list of discovered log files comprising a path and a name for each of the plurality of files;

adding the list of discovered log files to a list of monitored log files to be monitored by the observability pipeline system; and

after adding the list of discovered log files to the list of monitored log files, monitoring the plurality of files to generate input for the observability pipeline system.

2. The method of claim 1 , wherein monitoring the plurality of files to generate input for the observability pipeline system comprises:

extracting data from the one or more of the plurality of files;

generating observability pipeline input data by formatting the extracted data; and

processing the observability pipeline input data by operation of a data processing engine on the computer node.

3. The method of claim 2 , wherein processing the observability pipeline input data generates observability pipeline output data, and the method comprises:

collecting metadata from the computer node;

augmenting the observability pipeline output data to include the metadata; and

sending the augmented observability pipeline output data to a data destination.

4. The method of claim 1 , wherein the log discovery parameters comprise a file type parameter, a file path parameter, or a file name parameter.

5. The method of claim 1 , wherein the edge agent runs in a container on the computer node, and identifying processes running on the computer node comprises:

mounting a root file system of the computer node; and

scanning the root file system to identify processes running in other containers.

6. The method of claim 1 , wherein identifying files on the computer node comprises identifying a first file, and the method comprises:

identifying a container on the computer node where the first file is defined; and

identifying a modified path for the first file, wherein the modified path allows a process running outside the container to access the first file.

7. The method of claim 1 , comprising:

running a container discovery process to identify containers running on the computer node;

detecting console log files in each of the containers; and

adding a list of the console log files to the list of monitored log files.

8. The method of claim 1 , comprising:

running a container discovery process to identify containers running on the computer node;

detecting console logs in each of the containers; and

collecting the console logs through one or more sockets.

9. The method of claim 8 , comprising:

collecting container metrics for each of the containers;

generating observability pipeline input data by formatting the container metrics; and

processing the observability pipeline input data by operation of a data processing engine on the computer node.

10. The method of claim 1 , comprising modifying the list of discovered log files based on input received through a user interface in response to the list of discovered log files being presented to a user, wherein adding the list of discovered log files to the list of monitored log files comprises adding the modified list of discovered log files to the list of monitored log files.

11. The method of claim 1 , wherein the plurality of files is a first plurality of files, the list of discovered log files is a first list of discovered log files, and the method comprises:

modifying the log discovery parameters based on user input;

selecting a second plurality of files from the identified files according to the modified log discovery parameters;

generating a second list of discovered log files, the second list of discovered log files comprising a path and a name for each of the second plurality of files; and

adding the second list of discovered log files to the list of monitored log files to be monitored by the observability pipeline system.

12. The method of claim 1 , wherein the observability pipeline system comprises a leader role operating on a second computer node that communicates with the computer node over a network, and the method comprises displaying a list of the identified files on a user interface of the second computer node.

13. A computer node comprising:

one or more processors; and

memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations of an edge agent of an observability pipeline system, the operations comprising:

identifying processes running on the computer node;

identifying files on the computer node that the processes have opened for writing;

accessing log discovery parameters of the observability pipeline system;

selecting a plurality of files from the identified files according to the log discovery parameters;

generating a list of discovered log files, the list of discovered log files comprising a path and a name for each of the plurality of files;

adding the list of discovered log files to a list of monitored log files to be monitored by the observability pipeline system; and

after adding the list of discovered log files to the list of monitored log files, monitoring the plurality of files to generate input for the observability pipeline system.

14. The computer node of claim 13 , wherein monitoring the plurality of files to generate input for the observability pipeline system comprises:

extracting data from one or more of the plurality of files;

generating observability pipeline input data by formatting the extracted data; and

processing the observability pipeline input data by operation of a data processing engine on the computer node.

15. The computer node of claim 14 , wherein processing the observability pipeline input data generates observability pipeline output data, and the operations comprise:

collecting metadata from the computer node;

augmenting the observability pipeline output data to include the metadata; and

sending the augmented observability pipeline output data to a data destination.

16. The computer node of claim 13 , wherein the log discovery parameters comprise a file type parameter, a file path parameter, or a file name parameter.

17. The computer node of claim 13 , wherein the edge agent runs in a container on the computer node, and identifying processes running on the computer node comprises:

mounting a root file system of the computer node; and

scanning the root file system to identify processes running in other containers.

18. The computer node of claim 13 , wherein identifying files on the computer node comprises identifying a first file, and the operations comprise:

identifying a container on the computer node where the first file is defined; and

identifying a modified path for the first file, wherein the modified path allows a process running outside the container to access the first file.

19. The computer node of claim 13 , wherein the operations comprise:

running a container discovery process to identify containers running on the computer node;

detecting console log files in each of the containers; and

adding a list of the console log files to the list of monitored log files.

20. The computer node of claim 13 , wherein the operations comprise:

running a container discovery process to identify containers running on the computer node;

detecting console logs in each of the containers; and

collecting the console logs through one or more sockets.

21. The computer node of claim 20 , wherein the operations comprise:

collecting container metrics for each of the containers;

generating observability pipeline input data by formatting the container metrics; and

processing the observability pipeline input data by operation of a data processing engine on the computer node.

22. The computer node of claim 13 , wherein the operations comprise:

modifying the list of discovered log files based on input received through a user interface in response to the list of discovered log files being presented to a user, wherein adding the list of discovered log files to the list of monitored log files comprises adding the modified list of discovered log files to the list of monitored log files.

23. The computer node of claim 13 , wherein the plurality of files is a first plurality of files, the list of discovered log files is a first list of discovered log files, and the operations comprise:

modifying the log discovery parameters based on user input;

selecting a second plurality of files from the identified files according to the modified log discovery parameters;

generating a second list of discovered log files, the second list of discovered log files comprising a path and a name for each of the second plurality of files; and

adding the second list of discovered log files to the list of monitored log files to be monitored by the observability pipeline system.

24. A non-transitory computer-readable medium storing instructions that perform operations of an edge agent of an observability pipeline system when executed by data processing apparatus of a computer node, the operations comprising:

identifying processes running on the computer node;

identifying files on the computer node that the processes have opened for writing;

accessing log discovery parameters of the observability pipeline system;

selecting a plurality of files from the identified files according to the log discovery parameters;

generating a list of discovered log files, the list of discovered log files comprising a path and a name for each of the plurality of files;

adding the list of discovered log files to a list of monitored log files to be monitored by the observability pipeline system; and

after adding the list of discovered log files to the list of monitored log files, monitoring the plurality of files to generate input for the observability pipeline system.