Collecting and aggregating log data with fault tolerance
View Patent ↗Systems and methods of collecting and aggregating log data with fault tolerance are disclosed. One embodiment includes, one or more devices that generate log data, the one or more machines each associated with an agent node to collect the log data, wherein, the agent node generates a batch comprising multiple messages from the log data and assigns a tag to the batch. In one embodiment, the agent node further computes a checksum for the batch of multiple messages. The system may further include a collector device, the collector device being associated with a collector tier having a collector node to which the agent sends the log data; wherein, the collector determines the checksum for the batch of multiple messages received from the agent node.
1. A method for collecting and aggregating datasets for storage in a file system with fault tolerance, the method comprising:
collecting datasets from a data source on a machine where the datasets are generated;
generating a batch comprising multiple messages from the datasets;
assigning a tag to the batch and computing a checksum for the batch;
writing the tag, the batch of multiple messages, and the checksum to an entry in a write-ahead-log (WAL) in storage;
sending the datasets to a receiving location;
in response to verifying the checksum of the batch of multiple messages at the receiving location, adding the tag to a map;
writing a file to destination location; and
identifying, in the file, tags associated with the batches in the file that have been written to the destination location.
2. The method of claim 1 , wherein the storage for the WAL is on the machine where the datasets are generated.
3. The method of claim 1 , wherein the map is associated with multiple tags assigned to multiple batches of messages from the datasets.
4. The method of claim 1 , wherein the destination location is a storage tier comprising a distributed file system.
5. The method of claim 1 , further comprising querying for an acknowledgement message that the batch has been written to the destination location.
6. The method of claim 1 , further comprising, in response to determining that the batch has been written in the destination location, deleting the entry in the storage of the machine.
7. The method of claim 1 , further comprising, in response to determining that the batch has not been written to a file in the destination location, resending the batch.
8. An apparatus for collecting and aggregating datasets for storage in a file system with fault tolerance, the apparatus including a memory storing instructions that, when executed by a processor of the apparatus, cause the apparatus to perform a method comprising:
collecting datasets from a data source on a machine where the datasets are generated;
generating a batch comprising multiple messages from the datasets;
assigning a tag to the batch and computing a checksum for the batch;
writing the tag, the batch of multiple messages, and the checksum to an entry in a write-ahead-log (WAL) in storage;
sending the datasets to a receiving location;
in response to verifying the checksum of the batch of multiple messages at he receiving location, adding the tag to a map;
writing a file to destination location; and
identifying, in the file, tags associated with the batches in the file that have been written to the destination location.
9. The apparatus of claim 8 , wherein the storage for the WAL is on the machine where the datasets are generated.
10. The apparatus of claim 8 , wherein the map is associated with multiple tags assigned to multiple batches of messages from the datasets.
11. The apparatus of claim 8 , wherein the destination location is a storage tier comprising a distributed file system.
12. The apparatus of claim 8 , further comprising querying for an acknowledgement message that the batch has been written to the destination location.
13. The apparatus of claim 8 , further comprising, in response to determining that the batch has been written in the destination location, deleting the entry in the storage of the machine.
14. The apparatus of claim 8 , further comprising, in response to determining that the batch has not been written to a file in the destination location, resending the batch.
15. A non-transitory computer-readable medium for collecting and aggregating datasets for storage in a file system with fault tolerance, the medium storing instructions that, when executed by a processor of an apparatus, cause the apparatus to perform a method comprising:
collecting datasets from a data source on a machine where the datasets are generated;
generating a batch comprising multiple messages from the datasets;
assigning a tag to the batch and computing a checksum for the batch;
writing the tag, the batch of multiple messages, and the checksum to an entry in a write-ahead-log (WAL) in storage;
sending the datasets to a receiving location;
in response to verifying the checksum of the batch of multiple messages at the receiving location, adding the tag to a map;
writing a file to destination location; and
identifying, in the file, tags associated with the batches in the file that have been written to the destination location.
16. The medium of claim 15 , wherein the storage for the WAL is on the machine where the datasets are generated.
17. The medium of claim 15 , wherein the map is associated with multiple tags assigned to multiple batches of messages from the datasets.
18. The medium of claim 15 , wherein the destination location is a storage tier comprising a distributed file system.
19. The medium of claim 15 , the method further comprising querying for an acknowledgement message that the batch has been written to the destination location.
20. The medium of claim 15 , the method further comprising, selectively performing deleting the entry in the storage of the machine or resending the batch based on whether the batch has been successfully written in the destination location.