Processing log files using a database system
View Patent ↗Disclosed are some examples of database systems, methods, and computer program products for processing log files. In some implementations, log files can be accessed and identifiers can be generated for the data entries of the log files. The generated identifiers can be used to determine whether data entries in the log files are new, and therefore, should be extracted for analysis.
1. A system comprising:
a database system implemented using a server system comprising one or more processors, the database system configurable to cause:
accessing a log file maintained using one or more data objects stored in a database, the log file comprising a plurality of data entries identifying a plurality of system events corresponding to user activity associated with a plurality of users, each data entry of the log file comprising one or more data items,
determining that one or more formats of one or more of the data entries is incompatible with an analytics application configured to analyze log files,
generating, responsive to determining that the one or more formats is incompatible with the analytics application, a respective one or more identifiers for each data entry of the one or more data entries,
determining that the respective one or more identifiers for each data entry have not been extracted for analysis by the analytics application,
extracting, responsive to determining that the respective one or more identifiers have not been extracted for analysis, a new one or more data entries based on the respective one or more identifiers, and
providing the new one or more data entries to the analytics application.
2. The system of claim 1 , wherein the respective one or more identifiers are generated based on the one or more data items of a corresponding data entry.
3. The system of claim 2 , wherein generating the respective one or more identifiers comprises:
extracting the one or more data items of the corresponding data entry; and
generating the identifier corresponding to the data entry using the extracted one or more data items.
4. The system of claim 3 , wherein generating the identifier comprises providing the one or more data items to a hashing function.
5. The system of claim 1 , wherein the new one or more data entries are extracted at a threshold time interval.
6. The system of claim 1 , wherein the database system is further configurable to cause:
determining that a processing time corresponding to extracting the new one or more data entries based on the respective one or more identifiers exceeds the threshold time interval; and
extracting a further one or more data entries based on the determination that the processing time exceeds the threshold time interval.
7. The system of claim 1 , wherein the log file is a customer-facing log file.
8. A computer program product comprising program code to be executed by one or more processors when retrieved from a non-transitory computer-readable medium, the program code comprising instructions configured to cause:
accessing a log file maintained using one or more data objects stored in a database, the log file comprising a plurality of data entries identifying a plurality of system events corresponding to user activity associated with a plurality of users, each data entry of the log file comprising one or more data items,
determining that one or more formats of one or more of the data entries is incompatible with an analytics application configured to analyze log files,
generating, responsive to determining that the one or more formats is incompatible with the analytics application, a respective one or more identifiers for each data entry of the one or more data entries,
determining that the respective one or more identifiers for each data entry have not been extracted for analysis by the analytics application,
extracting, responsive to determining, that the respective one or more identifiers have not been extracted for analysis, a new one or more data entries based on the respective one or more identifiers, and
providing the new one or more data entries to the analytics application.
9. The computer program product of claim 8 , wherein the respective one or more identifiers are generated based on the one or more data items of a corresponding data entry.
10. The computer program product of claim 9 , wherein generating the respective one or more identifiers comprises:
extracting the one or more data items of the corresponding data entry; and
generating the identifier corresponding to the data entry using the extracted one or more data items.
11. The computer program product of claim 10 , wherein generating the identifier comprises providing the one or more data items to a hashing function.
12. The computer program product of claim 8 , wherein the new one or more data entries are extracted at a threshold time interval.
13. The computer program product of claim 8 , wherein the instructions are further configured to cause:
determining that a processing time corresponding to extracting the new one or more data entries based on the respective one or more identifiers exceeds the threshold time interval; and
extracting a further one or more data entries based on the determination that the processing time exceeds the threshold time interval.
14. A method for processing log files, the method comprising:
accessing, by a system, a log file maintained using one or more data objects stored in a database, the log file comprising a plurality of data entries identifying a plurality of system events corresponding to user activity associated with a plurality of users, each data entry of the log file comprising one or more data items,
determining that one or more formats of one or more of the data entries is incompatible with an analytics application configured to analyze log files,
generating, responsive to determining that the one or more formats is incompatible with the analytics application, a respective one or more identifiers for each data entry of the one or more data entries,
determining that the respective one or more identifiers for each data entry have not been extracted for analysis by the analytics application,
extracting, responsive to determining that the respective one or more identifiers have not been extracted for analysis, a new one or more data entries based on the respective one or more identifiers, and
providing the new one or more data entries to the analytics application.
15. The method of claim 14 , wherein the respective one or more identifiers are generated based on the one or more data items of a corresponding data entry.
16. The method of claim 15 , wherein generating the respective one or more identifiers comprises:
extracting the one or more data items of the corresponding data entry, and
generating the identifier corresponding to the data entry using the extracted one or more data items.
17. The method of claim 16 , wherein generating the identifier comprises providing the one or more data items to a hashing function.
18. The method of claim 14 , wherein the new one or more data entries are extracted at a threshold time interval.
19. The method of claim 14 , further comprising:
determining that a processing time corresponding to extracting the new one or more data entries based on the respective one or more identifiers exceeds the threshold time interval; and
extracting a further one or more data entries based on the determination that the processing time exceeds the threshold time interval.
20. The method of claim 14 , wherein the log file is a customer-facing log file.