Method and system for implementing target model configuration metadata for a log analytics system
Disclosed is a system, method, and computer program product for implementing a log analytics method and system that can configure, collect, and analyze log records in an efficient manner. The log analytics system, method, and computer program product provide target-based configuration of log monitoring metadata. The log analytics system, method, and computer program product can be implemented in a cloud-based/SaaS-based architecture.
1. A method comprising:
receiving, via a log configuration interface of a log analytics system configured to perform log analytics for a plurality of targets in a network, (a) first user input indicating a target type, and (b) second user input indicating a log source;
wherein the log source represents a location of log data and comprises a file pattern including a fixed part and a variable part;
generating, by the log analytics system based on the first user input and the second user input, a log collection configuration that indicates the target type and the log source;
responsive to determining that a first target, of the plurality of targets, is associated with the target type: associating the log collection configuration with the first target;
responsive to associating the log collection configuration with the first target: replacing the variable part of the file pattern, associated with the log collection configuration, with first metadata associated with a first target property of the first target to generate a first file path indicating a first location of first log data associated with the first target;
generating first target-side configuration content comprising the first file path;
responsive to determining that a second target, of the plurality of targets, is associated with the target type: associating the log collection configuration with the second target;
responsive to associating the log collection configuration with the second target: replacing the variable part of the file pattern, associated with the log collection configuration, with second metadata associated with a second target property of the second target to generate a second file path indicating a second location of second log data associated with the second target;
generating second target-side configuration content comprising the second file path;
distributing, by the log analytics system, the first target-side configuration content to a first agent associated with the first target and the second target-side configuration content to a second agent associated with the second target;
receiving, by the log analytics system from the first agent, the first log data captured for the first target at the first location indicated by the first file path;
receiving, by the log analytics system from the second agent, the second log data captured for the second target at the second location indicated by the second file path; and
storing, by the log analytics system, the first log data and the second log data
wherein the method is performed by at least one device including a hardware processor.
2. The method of claim 1 , further comprising:
configuring log monitoring configuration metadata, wherein the log monitoring configuration metadata comprises at least one of a base parser, a log rule applicable to a particular target of the target type, and an identification of the particular log target.
3. The method of claim 1 , further comprising:
identifying a parser for the log source, wherein the parser comprises a regular expression to define how to parse the log data for a particular target of the target type.
4. The method of claim 1 , wherein the log collection configuration comprises a first portion applicable to the first target and a second portion applicable to the second target.
5. The method of claim 1 , wherein the log collection configuration corresponds to an XML file.
6. The method of claim 1 , wherein the log collection configuration is generated, modified, or deleted upon a change to an association between the first target and the log source, a change to the first target, or a change to a status of the first agent.
7. The method of claim 1 , further comprising:
periodically synchronizing associations between the log collection configuration and the plurality of targets.
8. The method of claim 1 , wherein the first target property of the first target comprises a first host of the first target and the second target property of the second target comprises a second host of the second target.
9. One or more machine readable media storing instructions which, when executed by one or more processors, cause:
receiving, via a log configuration interface of a log analytics system configured to perform log analytics for a plurality of targets in a network, (a) first user input indicating a target type, and (b) second user input indicating a log source;
wherein the log source represents a location of log data and comprises a file pattern including a fixed part and a variable part;
generating, by the log analytics system based on the first user input and the second user input, a log collection configuration that indicates the target type and the log source;
responsive to determining that a first target, of the plurality of targets, is associated with the target type: associating the log collection configuration with the first target;
responsive to associating the log collection configuration with the first target: replacing the variable part of the file pattern, associated with the log collection configuration, with first metadata associated with a first target property of the first target to generate a first file path indicating a first location of first log data associated with the first target;
generating first target-side configuration content comprising the first file path;
responsive to determining that a second target, of the plurality of targets, is associated with the target type: associating the log collection configuration with the second target;
responsive to associating the log collection configuration with the second target: replacing the variable part of the file pattern, associated with the log collection configuration, with second metadata associated with a second target property of the second target to generate a second file path indicating a second location of second log data associated with the second target;
generating second target-side configuration content comprising the second file path;
distributing, by the log analytics system, the first target-side configuration content to a first agent associated with the first target and the second target-side configuration content to a second agent associated with the second target;
receiving, by the log analytics system from the first agent, the first log data captured for the first target at the first location indicated by the first file path;
receiving, by the log analytics system from the second agent, the second log data captured for the second target at the second location indicated by the second file path; and
storing, by the log analytics system, the first log data and the second log data.
10. The one or more media of claim 9 , further storing instructions which, when executed by one or more processors, cause:
configuring log monitoring configuration metadata, wherein the log monitoring configuration metadata comprises at least one of a base parser, a log rule applicable to a particular target of the target type, and an identification of the particular log target.
11. The one or more media of claim 9 , further storing instructions which, when executed by one or more processors, cause:
identifying a parser for the log source, wherein the parser comprises a regular expression to define how to parse the log data for a particular target of the target type.
12. The one or more media of claim 9 , wherein the log collection configuration comprises a first portion applicable to the first target and a second portion applicable to the second target.
13. The one or more media of claim 9 , wherein the log collection configuration corresponds to an XML file.
14. The one or more media of claim 9 , wherein the log collection configuration is generated, modified, or deleted upon a change to an association between the first target and the log source, a change to the first target, or a change to a status of the first agent.
15. The one or more media of claim 9 , further storing instructions which, when executed by one or more processors, cause:
periodically synchronizing associations between the log collection configuration and the plurality of targets.
16. A system comprising:
at least one device including a hardware processor;
the system being configured to perform operations comprising:
receiving, via a log configuration interface of a log analytics system configured to perform log analytics for a plurality of targets in a network, (a) first user input indicating a target type, and (b) second user input indicating a log source;
wherein the log source represents a location of log data and comprises a file pattern including a fixed part and a variable part;
generating, by the log analytics system based on the first user input and the second user input, a log collection configuration that indicates the target type and the log source;
responsive to determining that a first target, of the plurality of targets, is associated with the target type: associating the log collection configuration with the first target;
responsive to associating the log collection configuration with the first target: replacing the variable part of the file pattern, associated with the log collection configuration, with first metadata associated with a first target property of the first target to generate a first file path indicating a first location of first log data associated with the first target;
generating first target-side configuration content comprising the first file path;
responsive to determining that a second target, of the plurality of targets, is associated with the target type: associating the log collection configuration with the second target;
responsive to associating the log collection configuration with the second target: replacing the variable part of the file pattern, associated with the log collection configuration, with second metadata associated with a second target property of the second target to generate a second file path indicating a second location of second log data associated with the second target;
generating second target-side configuration content comprising the second file path;
distributing, by the log analytics system, the first target-side configuration content to a first agent associated with the first target and the second target-side configuration content to a second agent associated with the second target;
receiving, by the log analytics system from the first agent, the first log data captured for the first target at the first location indicated by the first file path;
receiving, by the log analytics system from the second agent, the second log data captured for the second target at the second location indicated by the second file path; and
storing, by the log analytics system, the first log data and the second log data.
17. The system of claim 16 , the operations further comprising:
configuring log monitoring configuration metadata, wherein the log monitoring configuration metadata comprises at least one of a base parser, a log rule applicable to a particular target of the target type, and an identification of the particular log target.
18. The system of claim 16 , the operations further comprising:
identifying a parser for the log source, wherein the parser comprises a regular expression to define how to parse the log data for a particular target of the target type.
19. The system of claim 16 , wherein the log collection configuration corresponds to an XML file.
20. The system of claim 16 , wherein the log collection configuration is generated, modified, or deleted upon a change to an association between the first target and the log source, a change to the first target, or a change to a status of the first agent.
21. The system of claim 16 , the operations further comprising:
periodically synchronizing associations between the log collection configuration and the plurality of targets.