Approach for a controllable trade-off between cost and availability of indexed data in a cloud log aggregation solution such as splunk or sumo
Systems and methods for classification of log data at the source into sets of critical and non-critical data. Critical data may be indexed and processed normally, while non-critical data may be provided to and stored by the cloud-based log aggregation system without indexing, at significantly lower cost in terms of processing and storage. In the event that non-critical data is required for troubleshooting or analysis, the non-critical data may be indexed dynamically on request. Because the non-critical data is stored at the cloud-based log aggregation system, it may be quickly indexed and added to the critical data, without additional consumption of bandwidth or delays due to transmission. Dynamic selection and classification of critical and non-critical data may thus allow an enterprise to balance costs and availability of indexed data.
1. A method for cloud-based log management, comprising:
retrieving, by a log manager of a computing device, a plurality of logs;
selecting, by the log manager, a first subset of the plurality of logs responsive to each of the logs of the first subset lacking an identifier of whether each of the logs should be indexed or not;
classifying, by the log manager, a first portion of the first subset of the plurality of logs as to be indexed and a second portion of the first subset of the plurality of logs as not to be indexed, responsive to characteristics of logs previously requested for review being respectively similar to the first portion and not similar to the second portion;
transmitting, by the log manager to a log aggregator, the first portion of the first subset of the plurality of logs, responsive to the classification of the first subset of the plurality of logs as to be indexed; and
transmitting, by the log manager to a storage device of the log aggregator, the second portion of the first subset of the plurality of logs.
2. The method of claim 1 , wherein retrieving the plurality of logs further comprises retrieving, by the log manager from a plurality of additional computing devices, the plurality of logs.
3. The method of claim 1 , wherein classifying the first subset of the plurality of logs as to be indexed further comprises appending, by the log manager, an indexing control identifier to each log of the first subset of the plurality of logs.
4. The method of claim 1 , wherein classifying the first subset of the plurality of logs as to be indexed further comprises further comprises classifying the first subset of the plurality of logs as critical or high priority.
5. The method of claim 1 , wherein the log aggregator stores the second portion of the first subset of the plurality of logs without indexing responsive to the second portion of the first subset of the plurality of logs not being classified as to be indexed.
6. The method of claim 1 , wherein selecting the first subset of the plurality of logs further comprises:
selecting a log of the plurality of logs, by a machine learning system of the computing device; and
adding the log to the first subset, by the machine learning system, responsive to the log having characteristics matching characteristics of logs previously requested for review.
7. The method of claim 6 , wherein the characteristics comprise routines that generated the logs previously requested for review, computing devices corresponding to the logs previously requested for review, generation times of logs previously requested for review, or types of logs previously requested for review.
8. The method of claim 1 , wherein selecting the first subset of the plurality of logs further comprises:
for each log of the first subset of the plurality of logs:
identifying, by the log manager, a routine that generated the log, the routine comprising an indexing command, and
adding the log to the first subset, by the log manager, responsive to the identified routine comprising the indexing command.
9. The method of claim 1 , further comprising transmitting, by the log manager to the log aggregator, a request to index at least one log of the second portion of the first subset of the plurality of logs, the log aggregator indexing the at least one log of the second portion of the first subset of the plurality of logs responsive to receipt of the request.
10. A system for cloud-based log management, comprising:
a computing device comprising a network interface and a processor executing a log manager;
wherein the log manager is configured to:
retrieve a plurality of logs,
select a first subset of the plurality of logs responsive to each of the logs of the first subset lacking an identifier of whether each of the logs should be indexed or not,
classify a first portion of the first subset of the plurality of logs as to be indexed and a second portion of the first subset of the plurality of logs as not to be indexed, responsive to characteristics of logs previously requested for review being respectively similar to the first portion and not similar to the second portion,
transmit, to a log aggregator for indexing, the first portion of the first subset of the plurality of logs, responsive to the classification of the first subset of the plurality of logs as to be indexed, and
transmit, to a storage device of the log aggregator, the second portion of the first subset of the plurality of logs.
11. The system of claim 10 , wherein the log manager is further configured to retrieve, from a plurality of additional computing devices, the plurality of logs.
12. The system of claim 10 , wherein the log manager is further configured to append an indexing control identifier to each log of the first subset of the plurality of logs.
13. The system of claim 12 , wherein the log manager is further configured to identify each log of the first subset of the plurality of logs as critical or high priority.
14. The system of claim 10 , wherein the log aggregator stores the second portion of the first subset of the plurality of logs without indexing responsive to the second portion of the first subset of the plurality of logs not being classified as to be indexed.
15. The system of claim 10 , wherein the log manager further comprises a machine learning system configured to:
select a log of the plurality of logs; and
add the log to the first subset, responsive to the log having characteristics matching characteristics of logs previously requested for review.
16. The system of claim 15 , wherein the characteristics comprise routines that generated the logs previously requested for review, computing devices corresponding to the logs previously requested for review, generation times of logs previously requested for review, or types of logs previously requested for review.
17. The system of claim 10 , wherein the log manager is further configured to:
for each log of the first subset of the plurality of logs:
identify a routine that generated the log, the routine comprising an indexing command, and
add the log to the first subset, by the log manager, responsive to the identified routine comprising the indexing command.
18. The system of claim 10 , wherein the log manager is further configured to transmit, to the log aggregator, a request to index at least one log of the second portion of the first subset of the plurality of logs, the log aggregator indexing the at least one log of the second portion of the first subset of the plurality of logs responsive to receipt of the request.
19. A tangible computer-readable storage medium comprising instructions that, when executed by a processor of a computing device, cause the processor to:
retrieve a plurality of logs;
select a first subset of the plurality of logs responsive to each of the logs of the first subset lacking an identifier of whether each of the logs should be indexed or not;
classify a first portion of the first subset of the plurality of logs as to be indexed and a second portion of the first subset of the plurality of logs as not to be indexed, responsive to characteristics of logs previously requested for review being respectively similar to the first portion and not similar to the second portion;
transmit, to a log aggregator for indexing, the first subset of the plurality of logs; responsive to the classification of the first subset of the plurality of logs as to be indexed; and
transmit, to a storage device of the log aggregator, the second portion of the first subset of the plurality of logs.
20. The computer-readable storage medium of claim 19 , further comprising instructions to append an indexing control identifier to each log of the first subset of the plurality of logs.