Importation, presentation, and persistent storage of data
Described are methods, systems and computer readable media for the importation, presentation, and persistent storage of data.
1. A computer data system comprising:
one or more processors;
computer readable storage coupled to the one or more processors, the computer readable storage having stored thereon instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including:
starting with a processor a log tailer process for reading data from one or more log files and transmitting the data read from the one or more log files;
the log tailer process reading the data from the one or more log files;
the log tailer process transmitting the data to one or more data import servers, the one or more data import servers being separate from the log tailer process;
at the one or more data import servers, writing the data to one or more columns of an associated table location in cache memory buffers;
confirming, at the one or more data import servers, a current number of rows of data written to the cache memory buffers;
synchronizing, at the one or more data import servers, the column data from the cache memory buffers into a persistent table data store;
updating, at the one or more data import servers, table location metadata;
writing formatted data into the one or more log files by applying schema-specified, generated code to format the data;
parsing at least a portion of the formatted data with a listener generated from a schema;
generating the schema-specified, generated code used to format the data, wherein the table location metadata comprises a version identifier of another schema used to generate the schema-specified, generated code;
and
writing a check point for persisted data from the persistent table data store for use in resuming import after a system fault.
2. The computer data system of claim 1 , the operations further including:
advertising an existence of imported table location data available in the cache memory buffers.
3. The computer data system of claim 2 , wherein the advertising is made without waiting for the check point to be written.
4. The computer data system of claim 1 , wherein writing a check point for the cache memory buffers includes at least one of:
a table location size in rows; and
a log file and a byte-offset to resume from a table location next row.
5. The computer data system of claim 1 , the operations further including:
writing state associated with each table location imported from one or more application process logs.
6. The computer data system of claim 5 , wherein the processor applies the state to validate data written to the persistent table data store.
7. The computer data system of claim 6 , wherein the state includes information on connections subscribed to advertisements from the one or more data import servers about one or more of existence of new locations of tables, size of individual table locations, schema-specific data used for the validation, actual column file data or the table location metadata.
8. The computer data system of claim 1 , wherein the one or more log files are row-oriented.
9. The system of claim 1 , wherein the one or more data import servers are remote from the log tailer process.
10. A method for importing and presenting data for computer system use, the method comprising:
Starting, with a processor, a log tailer process for reading data from one or more log files and transmitting the data read from the one or more log files;
the log tailer process reading the data from the one or more log files;
the log tailer process transmitting the data to one or more data import servers, the one or more data import servers being separate from the log tailer process;
at the one or more data import servers, writing the data to one or more columns of an associated table location in cache memory buffers;
confirming, at the one or more data import servers, a current number of rows of data written to the cache memory buffers;
writing formatted data into the one or more log files by applying schema-specified, generated code to format the data;
parsing at least a portion of the formatted data with a listener generated from a schema;
updating, at the one or more data import servers, table location metadata;
generating the schema-specified, generated code used to format the data, wherein the table location metadata comprises a version identifier of another schema used to generate the schema-specified, generated code;
and
writing a check point for persisted data from a persistent table data store for use in resuming import after a system fault.
11. The method of claim 10 , further comprising:
advertising an existence of imported table location data available in the cache memory buffers.
12. The method of claim 11 , wherein the advertising is made without waiting for the check point to be written.
13. The method of claim 10 , wherein writing a check point for the cache memory buffers includes at least one of:
a table location size in rows; and
a log file and a byte-offset to resume from for thea table location next row.
14. The method of claim 10 , further comprising:
writing state associated with each table location imported from one or more application process logs.
15. The method of claim 14 , further comprising:
synchronizing, at the one or more data import servers, the column data from the cache memory buffers into a persistent table data store,
wherein the processor applies the state to validate data written to the persistent table data store.
16. The method of claim 15 , wherein the state includes information on connections subscribed to advertisements from the one or more data import servers about one or more of existence of new locations of tables, size of individual table locations, schema-specific data used for the validation, actual column file data or the table location metadata.
17. The method of claim 10 , wherein the one or more log files are row-oriented.
18. The method of claim 10 , wherein the one or more data import servers are remote from the log tailer process.
19. A nontransitory computer readable medium having stored thereon software instructions that, when executed by one or more processors, cause the one or more processors to perform operations including:
starting with a processor a log tailer process for reading data from one or more log files and transmitting the data read from the one or more log files;
the log tailer process reading the data from the one or more log files;
the log tailer process transmitting the data to one or more data import servers, the one or more data import servers being separate from the log tailer process;
at the one or more data import servers, writing the data to one or more columns of an associated table location in cache memory buffers;
synchronizing, at the one or more data import servers, the column data from the cache memory buffers into a persistent table data store;
writing formatted data into the one or more log files by applying schema-specified, generated code to format the data;
parsing at least a portion of the formatted data with a listener generated from a schema;
updating, at the one or more data import servers, table location metadata;
generating the schema-specified, generated code used to format the data, wherein the table location metadata comprises a version identifier of another schema used to generate the schema-specified, generated code;
and
writing a check point for persisted data from the persistent table data store for use in resuming import after a system fault.
20. The nontransitory computer readable medium of claim 19 , the operations further including:
advertising an existence of imported table location data available in the cache memory buffers.
21. The nontransitory computer readable medium of 20 , wherein the advertising is made without waiting for the check point to be written.
22. The nontransitory computer readable medium of claim 19 , wherein writing a check point for the cache memory buffers includes at least one of:
a table location size in rows; and
a log file and a byte-offset to resume from a table location next row.
23. The nontransitory computer readable medium of claim 19 , the operations further including:
writing state associated with each table location imported from one or more application process logs.
24. The nontransitory computer readable medium of claim 23 , wherein the processor applies the state to validate data written to the persistent table data store.
25. The nontransitory computer readable medium of claim 19 , wherein the one or more log files are row-oriented.
26. The nontransitory computer readable medium of claim 19 , wherein the one or more data import servers are remote from the log tailer process.
27. A computer data system comprising:
one or more processors;
computer readable storage coupled to the one or more processors, the computer readable storage having stored thereon instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including:
starting a log tailer process for reading data from one or more log files and transmitting data read from log files;
the log tailer process reading data from the one or more log files;
the log tailer process transmitting the data to one or more data import servers;
at the one or more data import servers, writing the data to one or more columns of an associated table location in cache memory buffers;
confirming, at the one or more data import servers, a current number of rows of data written to the cache memory buffers;
synchronizing, at the one or more data import servers, the column data from the cache memory buffers into a persistent table data store;
updating, at the one or more data import servers, table location metadata;
writing a check point for the persisted data for use in resuming import after a system fault;
generating a listener based on a schema;
generating formatting code based on the same schema;
receiving data;
formatting the received data by executing the generated formatting code to format the data for output to the one or more log files;
writing the formatted data to the one or more log files; and
parsing, with the listener, at least a portion of the formatted data read from the one or more log files.
28. A method for importing and presenting data for computer system use, the method comprising:
starting with a processor a log tailer process for reading data from one or more log files and transmitting data read from log files;
the log tailer process reading data from the one or more log files;
the log tailer process transmitting the data to one or more data import servers;
at the one or more data import servers, writing the data to one or more columns of an associated table location in cache memory buffers;
confirming, at the one or more data import servers, a current number of rows of data written to the cache memory buffers;
writing a check point for the persisted data for use in resuming import after a system fault;
generating a listener based on a schema;
generating formatting code based on the same schema;
receiving data;
formatting the received data by executing the generated formatting code to format the data for output to the one or more log files;
writing the formatted data to the one or more log files; and
parsing, with the listener, at least a portion of the formatted data read from the one or more log files.
29. A nontransitory computer readable medium having stored thereon software instructions that, when executed by one or more processors, cause the one or more processors to perform operations including:
starting a log tailer process for reading data from one or more log files and transmitting data read from log files;
the log tailer process reading data from the one or more log files;
the log taller process transmitting the data to one or more data import servers;
at the one or more data import servers, writing the data to one or more columns of an associated table location in cache memory buffers;
synchronizing, at the one or more data import servers, the column data from the cache memory buffers into a persistent table data store;
writing a check point for the persisted data for use in resuming import after a system fault
generating a listener based on a schema;
generating formatting code based on the same schema;
receiving data;
formatting the received data by executing the generated formatting code to format the data for output to the one or more log files;
writing the formatted data to the one or more log files; and
parsing, with the listener, at least a portion of the formatted data read from the one or more log files.