AUTOMATIC IDENTIFICATION AND TRACKING OF LOG ENTRY SCHEMAS CHANGES
A log analysis unit compares log entries describing an event to one or more schemas associated with the event. Each of the schemas describes a different log entry structure. When a log entry is determine to have a structure that does not match any of the structures defined by any of the schemas associated with a particular event, a new schema describing the structure of the log entry is generated. In response to the generation of the new schema, one or more entities are notified. Additionally, instructions for processing log entries adhering to the new schema are generated. A cumulative schema and an intersection schema corresponding to the event are also generated.
1 . A method comprising:
obtaining a first log entry in a log, wherein the first log entry describes a first occurrence of a particular event;
wherein data within the first log entry is organized according to a first structure;
in the absence of any schema that accurately describes the first structure, generating, based on the first log entry, a first schema describing the first structure;
storing the first schema;
obtaining a second log entry in the log, wherein the second log entry describes a second occurrence of the particular event;
wherein data within the second log entry is organized according to a second structure;
determining that the second structure does not match the first structure;
in response to determining that the second structure does not match the first structure, generating, based on the second log entry, a second schema describing the second structure;
storing the second schema;
generating, based on a plurality of schemas for the particular event, a cumulative schema corresponding to the particular event;
wherein the plurality of schemas includes at least the first schema and the second schema;
wherein the cumulative schema describes each field of each of the plurality of schemas; and
wherein the method is performed by one or more computing devices.
2 . The method of claim 1 further comprising:
generating, based on the plurality of schemas, an intersection schema describing only those fields that are common to every schema in the plurality of schemas.
3 . The method of claim 1 , wherein:
the step of determining that the second structure does not match the first structure includes determining that a value in a particular field of the second log entry is of a different type than a type identified in the first schema for the particular field.
4 . The method of claim 1 , wherein:
the step of determining that the second structure does not match the first structure includes determining that a value in a particular field of the second log entry is of a different length than a length identified in the first schema for the particular field.
5 . The method of claim 1 , wherein:
the cumulative schema identifies a plurality of fields of the plurality of schemas; and
for at least one field of the plurality of fields, the cumulative schema identifies:
a base type of the at least one field; and
an actual type of the at least one field.
6 . The method of claim 5 , wherein the base type is different than the actual type.
7 . The method of claim 1 further comprising:
in response to determining that the second log entry does not conform to the first schema, notifying a particular entity associated with development of an application that caused the log to be generated.
8 . The method of claim 1 further comprising:
wherein the step of notifying the particular entity includes sending a notification identifying a schema change relating to a particular field in a particular schema and that requests comments regarding the schema change;
receiving a comment relating to the schema change;
storing the comment in association with the particular field in the particular schema.
9 . The method of claim 1 further comprising:
in response to determining that the second log entry does not conform to the first schema, notifying a particular entity that uses data in the log.
10 . A method comprising:
obtaining a log entry in a log, wherein the log entry describes an occurrence of a particular event;
wherein data within the log entry is organized according to a particular structure;
determining that a base type of a value in a particular field in the log entry is a first type;
based on an analysis of the value, determining that the value has an actual type of a second type that is different than the first type;
in the absence of any schema that accurately describes the particular structure, generating, based on the log entry, a schema describing the particular structure;
wherein the schema indicates that the base type of the value in the particular field is the first type, and that an actual type of the value in the particular field is the second type;
storing the schema; and
wherein the method is performed by one or more computing devices.
11 . The method of claim 10 further comprising:
obtaining a second log entry in the log, wherein the second log entry describes a second occurrence of the particular event;
determining that the second log entry does not conform to the schema based on a determination that, within the second log entry, the particular field has a particular value that is of a different type than the second type;
in response to determining that the second log entry does not conform to the schema:
generating, based on the second log entry, a second schema describing a structure of the second log entry; and
sending a notification to an entity indicating that a schema change has occurred.
12 . A method comprising:
obtaining a first log entry in a log, wherein the first log entry describes a first occurrence of a particular event;
wherein data within the first log entry is organized according to a first structure;
in the absence of any schema that accurately describes the first structure, generating, based on the first log entry, a first schema describing the first structure;
storing the first schema;
determining a first set of log entry processing instructions, which when executed, automatically extract data from log entries adhering to the first structure;
obtaining a second log entry in the log, wherein the second log entry describes a second occurrence of the particular event;
wherein data within the second log entry is organized according to a second structure;
determining that the second structure does not match the first structure;
in response to determining that the second structure does not match the first structure:
generating, based on the second log entry, a second schema describing the second structure;
storing the second schema;
determining a second set of log entry processing instructions, which when executed, automatically extract data from log entries adhering to the second structure;
associating the second set of log entry processing instructions with the second schema; and
wherein the method is performed by one or more computing devices.
13 . The method of claim 12 , wherein the first set of log entry processing instructions and the second set of log entry processing instructions extract data using different techniques but both the first set of log entry processing instructions and the second set of log entry processing instructions provide information in a same format.
14 . One or more non-transitory computer-readable media storing instructions which, when executed by one or more processors, cause performance of a method comprising:
obtaining a first log entry in a log, wherein the first log entry describes a first occurrence of a particular event;
wherein data within the first log entry is organized according to a first structure;
in the absence of any schema that accurately describes the first structure, generating, based on the first log entry, a first schema describing the first structure;
storing the first schema;
obtaining a second log entry in the log, wherein the second log entry describes a second occurrence of the particular event;
wherein data within the second log entry is organized according to a second structure;
determining that the second structure does not match the first structure;
in response to determining that the second structure does not match the first structure, generating, based on the second log entry, a second schema describing the second structure;
storing the second schema;
generating, based on a plurality of schemas for the particular event, a cumulative schema corresponding to the particular event;
wherein the plurality of schemas includes at least the first schema and the second schema;
wherein the cumulative schema describes each field of each of the plurality of schemas.
15 . The one or more non-transitory computer-readable media of claim 14 , wherein the method further comprises:
generating, based on the plurality of schemas, an intersection schema describing only those fields that are common to every schema in the plurality of schemas.
16 . The one or more non-transitory computer-readable media of claim 14 , wherein:
the step of determining that the second structure does not match the first structure includes determining that a value in a particular field of the second log entry is of a different type than a type identified in the first schema for the particular field.
17 . The one or more non-transitory computer-readable media of claim 14 , wherein:
the step of determining that the second structure does not match the first structure includes determining that a value in a particular field of the second log entry is of a different length than a length identified in the first schema for the particular field.
18 . The one or more non-transitory computer-readable media of claim 14 , wherein:
the cumulative schema identifies a plurality of fields of the plurality of schemas; and
for at least one field of the plurality of fields, the cumulative schema identifies:
a base type of the at least one field; and
an actual type of the at least one field.
19 . The one or more non-transitory computer-readable media of claim 18 , wherein the base type is different than the actual type.
20 . The one or more non-transitory computer-readable media of claim 14 , wherein the method further comprises:
in response to determining that the second log entry does not conform to the first schema, notifying a particular entity associated with development of an application that caused the log to be generated.
21 . The one or more non-transitory computer-readable media of claim 14 , wherein the method further comprises:
wherein the step of notifying the particular entity includes sending a notification identifying a schema change relating to a particular field in a particular schema and that requests comments regarding the schema change;
receiving a comment relating to the schema change;
storing the comment in association with the particular field in the particular schema.
22 . The one or more non-transitory computer-readable media of claim 14 , wherein the method further comprises:
in response to determining that the second log entry does not conform to the first schema, notifying a particular entity that uses data in the log.
23 . One or more non-transitory computer-readable media storing instructions which, when executed by one or more processors, cause performance of a method comprising:
obtaining a log entry in a log, wherein the log entry describes an occurrence of a particular event;
wherein data within the log entry is organized according to a particular structure;
determining that a base type of a value in a particular field in the log entry is a first type;
based on an analysis of the value, determining that the value has an actual type of a second type that is different than the first type;
in the absence of any schema that accurately describes the particular structure, generating, based on the log entry, a schema describing the particular structure;
wherein the schema indicates that the base type of the value in the particular field is the first type, and that an actual type of the value in the particular field is the second type;
storing the schema.
24 . The one or more non-transitory computer-readable media of claim 23 , wherein the method further comprises:
obtaining a second log entry in the log, wherein the second log entry describes a second occurrence of the particular event;
determining that the second log entry does not conform to the schema based on a determination that, within the second log entry, the particular field has a particular value that is of a different type than the second type;
in response to determining that the second log entry does not conform to the schema:
generating, based on the second log entry, a second schema describing a structure of the second log entry; and
sending a notification to an entity indicating that a schema change has occurred.
25 . One or more non-transitory computer-readable media storing instructions which, when executed by one or more processors, cause performance of a method comprising:
obtaining a first log entry in a log, wherein the first log entry describes a first occurrence of a particular event;
wherein data within the first log entry is organized according to a first structure;
in the absence of any schema that accurately describes the first structure, generating, based on the first log entry, a first schema describing the first structure;
storing the first schema;
determining a first set of log entry processing instructions, which when executed, automatically extract data from log entries adhering to the first structure;
obtaining a second log entry in the log, wherein the second log entry describes a second occurrence of the particular event;
wherein data within the second log entry is organized according to a second structure;
determining that the second structure does not match the first structure;
in response to determining that the second structure does not match the first structure:
generating, based on the second log entry, a second schema describing the second structure;
storing the second schema;
determining a second set of log entry processing instructions, which when executed, automatically extract data from log entries adhering to the second structure;
associating the second set of log entry processing instructions with the second schema.
26 . The one or more non-transitory computer-readable media of claim 25 , wherein the first set of log entry processing instructions and the second set of log entry processing instructions extract data using different techniques but both the first set of log entry processing instructions and the second set of log entry processing instructions provide information in a same format.