Importation, presentation, and persistent storage of data
The disclosed subject matter includes systems, methods, and computer readable medium for importing/presenting data for computer system use. A logging process can receive data. Log file(s) can be created in memory. A listener can parse received log data to be appended to columns. A log tailer process can transmit data read from log file(s) to data import server(s). At data import server(s): the data can be written to column(s) of the associated table location in cache memory buffers, for immediate use by data consumers and for writing to persistent storage; a current number of rows of data written to the cache memory buffers can be confirmed; column data can be synchronized from the cache memory buffers into a persistent table data store; and table location metadata can be updated. A check point can be written for the persisted data for resuming import after a fault.
1. A computer data system for importing and presenting data for a computer system having cache memory buffers, persistent storage, and a listener device, the computer data system comprising
one or more processors;
computer readable storage coupled to the one or more processors, the computer readable storage having stored thereon instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including:
starting a logging process to receive a stream of data;
creating in a computer memory device one or more log files;
starting a log tailer process for reading data from the one or more log files and transmitting data read from log files;
starting a listener to parse received log data to be appended to columns;
receiving a stream of data by the logging process;
the logging process writing the data into the one or more log files in a computer memory;
the log tailer process reading data from the one or more log files;
the log tailer process transmitting the data to one or more data import servers;
parsing the data with the listener;
at the one or more data import servers, writing the data to one or more columns of the associated table location in cache memory buffers, for immediate use by data consumers and for writing to persistent storage;
confirming, at the one or more data import servers, a current number of rows of data written to the cache memory buffers;
synchronizing, at the one or more data import servers, the column data from the cache memory buffers into a persistent table data store;
updating, at the one or more data import servers, table location metadata; and
writing a check point for the persisted data for use in resuming import after a system fault.
2. The computer data system of claim 1 , the operations further including:
advertising an existence of imported table location data available in the cache memory buffers.
3. The computer data system of claim 1 , wherein writing a check point for the cache memory buffers includes at least one of:
a table location size in rows; and
a log file and byte-offset to resume from for the table location next row.
4. The computer data system of claim 1 , the operations further including:
writing state associated with each table location imported from the application process logs.
5. The computer data system of claim 4 , wherein the processor applies the state to validate data written to the persistent table data store.
6. The computer data system of claim 1 , wherein the logging process writes the data into the one or more log files in a computer memory by applying schema-specified, generated code to format the data; and wherein the listener is generated from a schema.
7. The system of claim 6 , wherein the table location metadata comprises a version identifier of a schema used to generate the schema-specified, generated code used to format the data.
8. The computer data system of claim 1 , wherein the one or more log files are row-oriented.
9. The system of claim 1 , wherein the one or more data import servers are remote from the log trailer process.
10. The system of claim 1 , the operations further including:
generating the listener based on a schema; and
generating formatting code based on the same schema,
wherein the logging process writing the data into the one or more log files in a computer memory comprises formatting the data by executing the generated formatting code to format the data for output to the one or more log files.
11. A method for importing and presenting data for computer system use, the method comprising:
starting with a processor a logging process to receive a stream of data;
creating in a computer memory device one or more log files;
starting with a processor a log tailer process for reading data from the one or more log files and transmitting data read from log files;
starting a listener to parse received log data to be appended to columns;
receiving a stream of data by the logging process;
the logging process writing the data into the one or more log files in a computer memory;
the log toiler process reading data from the one or more log files;
the log tailer process transmitting the data to one or more data import servers;
parsing the data with the listener;
at the one or more data import servers, writing the data to one or more columns of the associated table location in cache memory buffers, for immediate use by data consumers and for writing to persistent storage;
confirming, at the one or more data import servers, a current number of rows of data written to the cache memory buffers;
synchronizing, at the one or more data import servers, the column data from the cache memory buffers into a persistent table data store;
updating, at the one or more data import servers, table location metadata; and
writing a check point for the persisted data for use in resuming import after a system fault.
12. The method of claim 11 , further comprising:
advertising an existence of imported table location data available in the cache memory buffers.
13. The method of claim 11 , wherein writing a check point for the cache memory buffers includes at least one of:
a table location size in rows; and
a log file and byte-offset to resume from for the table location next row.
14. The method of claim 11 , further comprising:
writing state associated with each table location imported from the application process logs.
15. The method of claim 11 , wherein the processor applies the state to validate data written to the persistent table data store.
16. The method of claim 11 , wherein the logging process writes the data into the one or more log files in a computer memory by applying schema-specified, generated code to format the data; and wherein the listener is generated from a schema.
17. The method of claim 16 , wherein the table location metadata comprises a version identifier of a schema used to generate the schema-specified, generated code used to format the data.
18. The method of claim 11 , wherein the one or more log files are row-oriented.
19. The method of claim 11 , wherein the one or more data import servers are remote from the log tailer process.
20. The method of claim 11 , further comprising:
generating the listener based on a schema; and
generating formatting code based on the same schema,
wherein the logging process writing the data into the one or more log files in a computer memory comprises formatting the data by executing the generated formatting code to format the data for output to the one or more log files.
21. A nontransitory computer readable medium having stored thereon software instructions that, when executed by one or more processors, cause the one or more processors to perform operations including:
starting a logging process to receive a stream of data;
creating in a computer memory device one or more log files,
starting a log tailer process for reading data from the one or more log files and transmitting data read from log files;
starting a listener to parse received log data to be appended to columns;
receiving a stream of data by the logging process;
the logging process writing the data into the one or more log files in a computer memory;
the log tailer process reading data from the one or more log files;
the log tailer process transmitting the data to one or more data import servers;
parsing the data with the listener;
at the one or more data import servers, writing the data to one or more columns of the associated table location in cache memory buffers, for immediate use by data consumers and for writing to persistent storage;
confirming, at the one or more data import servers, a current number of rows of data written to the cache memory buffers;
synchronizing, at the one or more data import servers, the column data from the cache memory buffers into a persistent table data store;
updating, at the one or more data import servers, table location metadata; and
writing a check point for the persisted data for use in resuming import after a system fault.
22. The nontransitory computer readable medium of claim 21 , the operations further including:
advertising an existence of imported table location data available in the cache memory buffers.
23. The nontransitory computer readable medium of claim 21 , wherein writing a check point for the cache memory buffers includes at least one of:
a table location size in rows; and
a log file and byte-offset to resume from for the table location next row.
24. The nontransitory computer readable medium of claim 21 , the operations further including:
writing state associated with each table location imported from the application process logs.
25. The nontransitory computer readable medium of claim 21 , wherein the logging process writes the data into the one or more log files in a computer memory by applying schema-specified, generated code to format the data and wherein the listener is generated from a schema.
26. The nontransitory computer readable medium of claim 21 , wherein the table location metadata comprises a version identifier of a schema used to generate the schema-specified, generated code used to format the data.
27. The nontransitory computer readable medium of claim 21 , wherein the one or more log files are row-oriented.
28. The nontransitory computer readable medium of claim 21 , wherein the one or more data import servers are remote from the log tailer process.
29. The nontransitory computer readable medium of claim 21 , the operations further including:
generating the listener based on a schema; and
generating formatting code based on the same schema,
wherein the logging process writing the data into the one or more log files in a computer memory comprises formatting the data by executing the generated formatting code to format the data for output to the one or more log files.