Collecting and aggregating log data with fault tolerance
Systems and methods of collecting and aggregating log data with fault tolerance are disclosed. One embodiment includes, one or more devices that generate log data, the one or more machines each associated with an agent node to collect the log data, wherein, the agent node generates a batch comprising multiple messages from the log data and assigns a tag to the batch. In one embodiment, the agent node further computes a checksum for the batch of multiple messages. The system may further include a collector device, the collector device being associated with a collector tier having a collector node to which the agent sends the log data; wherein, the collector determines the checksum for the batch of multiple messages received from the agent node.
1. A method for collecting and aggregating datasets with fault tolerance, the method comprising:
collecting a dataset from a data source on a machine where the dataset is generated, wherein the dataset is collected by an agent node executed on the machine;
generating a batch comprising messages from the dataset;
assigning a tag to the batch and computing a checksum for the batch;
writing the tag, the batch comprising the messages, and the checksum to an entry in a write-ahead-log (WAL) in a storage;
sending the batch comprising the messages to a receiving location;
verifying the checksum of the batch comprising the messages at the receiving location; and
in response to determining that receiving location has failed, storing, by the agent node, the dataset in a persistent storage of the machine until the receiving location is repaired or until another destination is identified.
2. The method of claim 1 , further comprising:
writing a file to a destination location; and
identifying, in the file, tags associated with batches in the file that have been written to the destination location.
3. The method of claim 2 , wherein the destination location is a storage tier comprising a distributed file system.
4. The method of claim 1 , further comprising:
in response to verifying the checksum of the batch comprising messages at the receiving location, adding the tag to a map.
5. The method of claim 4 , wherein the map is associated with multiple tags assigned to multiple batches of messages from the dataset.
6. The method of claim 1 , further comprising querying for an acknowledgement message that the batch has been written to the destination location.
7. The method of claim 1 , further comprising, in response to determining that the batch has been written in the destination location, deleting the entry in the storage of the machine.
8. The method of claim 1 , further comprising, in response to determining that the batch has not been written to a file in the destination location, resending the batch.
9. The method of claim 1 , wherein the receiving location is a collector node in a collector tier other than the machine where the dataset is generated.
10. The method of claim 1 , wherein the receiving location is a file system in a storage tier comprising one or more storage devices.
11. The method of claim 1 , wherein the storage for the WAL is on the machine where the dataset is generated.
12. An apparatus for collecting and aggregating datasets for storage in a file system with fault tolerance, the apparatus including a memory storing instructions that, when executed by a processor of the apparatus, cause the apparatus to perform a method comprising:
collecting a dataset from a data source on a machine where the dataset is generated, wherein the dataset is collected by an agent node executed on the machine;
generating a batch comprising messages from the dataset;
assigning a tag to the batch and computing a checksum for the batch;
writing the tag, the batch comprising the messages, and the checksum to an entry in a write-ahead-log (WAL) in a storage;
sending the batch comprising the messages to a receiving location;
verifying the checksum of the batch comprising the messages at the receiving location; and
in response to determining that receiving location has failed, storing, by the agent node, the dataset in a persistent storage of the machine until the receiving location is repaired or until another destination is identified.
13. The apparatus of claim 12 , the method further comprising:
writing a file to a destination location; and
identifying, in the file, tags associated with batches in the file that have been written to the destination location.
14. The apparatus of claim 13 , wherein the destination location is a storage tier comprising a distributed file system.
15. The apparatus of claim 12 , the method further comprising:
in response to verifying the checksum of the batch comprising messages at the receiving location, adding the tag to a map.
16. The apparatus of claim 15 , wherein the map is associated with multiple tags assigned to multiple batches of messages from the dataset.
17. The apparatus of claim 12 , the method further comprising querying for an acknowledgement message that the batch has been written to the destination location.
18. The apparatus of claim 12 , the method further comprising, in response to determining that the batch has been written in the destination location, deleting the entry in the storage of the machine.
19. The apparatus of claim 12 , the method further comprising, in response to determining that the batch has not been written to a file in the destination location, resending the batch.
20. The apparatus of claim 12 , wherein the receiving location is a collector node in a collector tier other than the machine where the dataset is generated.
21. The apparatus of claim 12 , wherein the receiving location is a file system in a storage tier comprising one or more storage devices.
22. The apparatus of claim 12 , wherein the storage for the WAL is on the machine where the dataset is generated.