Table data persistence
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for a distributed parallel processing database system that persists table data in memory to a distributed file system. A distributed parallel processing database system persists table data in memory to a distributed file system. A parameter of creating a database table specifies that data records in the database table and history of changes to the data records can be stored in memory as well as in the distributed file system. When the database table is populated or otherwise modified, data records and the history in memory are evicted to the distributed file system as log files and removed from memory. The log files can be designated as write-only, where the data records, once written, cannot be read by structured query language (SQL) queries, or as read-write, where the data records, once written, can be read by SQL queries.
1. A method, comprising:
receiving a command for creating a database table, the command including an identifier of a data store persisted on a distributed file system;
creating the database table in response to the command;
receiving and inserting updated data as one or more data records in the database table;
storing at least a portion of the data records as log files of the data store on the distributed file system, comprising:
determining that an amount of data records stored in a memory device satisfies a memory overflow condition; and
in response, moving one or more data records from the memory device to the log files; and
in response to receiving a request to access the data records stored on the distributed file system, retrieving the data records from the log files using a mapper procedure of a plurality of mapper procedures of a mapping process.
2. The method of claim 1 , wherein the command includes a data definition language (DDL) statement.
3. The method of claim 2 , wherein storing at least a portion of the data records as log files of the data store comprises:
determining that a data record satisfies a condition of evicting data records from a memory device to the distributed file system as specified in an eviction setting in the DDL statement; and
in response, moving the data record from the memory device to the log files.
4. The method of claim 1 , wherein the mapper is configured to filter and sort the log files.
5. The method of claim 1 comprising storing a data manipulation language (DML) statement defining a schema of the database table in a metadata file in the data store.
6. A non-transitory storage device storing instructions operable to cause one or more computers to perform operations comprising:
receiving a command for creating a database table, the command including an identifier of a data store persisted on a distributed file system;
creating the database table in response to the command;
receiving and inserting updated data as one or more data records in the database table;
storing at least a portion of the data records as log files of the data store on the distributed file system, comprising:
determining that an amount of data records stored in a memory device satisfies a memory overflow condition; and
in response, moving one or more data records from the memory device to the log files; and
in response to receiving a request to access the data records stored on the distributed file system, retrieving the data records from the log files using a mapper procedure of a plurality of mapper procedures of a mapping process.
7. The non-transitory storage device of claim 6 , wherein the command includes a data definition language (DDL) statement.
8. The non-transitory storage device of claim 7 , wherein storing at least a portion of the data records as log files of the data store comprises:
determining that a data record satisfies a condition of evicting data records from a memory device to the distributed file system as specified in an eviction setting in the DDL statement; and
in response, moving the data record from the memory device to the log files.
9. The non-transitory storage device of claim 6 , wherein the mapper is configured to filter and sort the log files.
10. The non-transitory storage device of claim 6 , the operations comprising storing a data manipulation language (DML) statement defining a schema of the database table in a metadata file in the data store.
11. A system comprising:
one or more computers; and
a non-transitory storage device storing instructions operable to cause the one or more computers to perform operations comprising:
receiving a command for creating a database table, the command including an identifier of a data store persisted on a distributed file system;
creating the database table in response to the command;
receiving and inserting updated data as one or more data records in the database table;
storing at least a portion of the data records as log files of the data store on the distributed file system, comprising:
determining that an amount of data records stored in a memory device satisfies a memory overflow condition; and
in response, moving one or more data records from the memory device to the log files; and
in response to receiving a request to access the data records stored on the distributed file system, retrieving the data records from the log files using a mapper procedure of a plurality of mapper procedures of a mapping process.
12. The system of claim 11 , wherein the command includes a data definition language (DDL) statement.
13. The system of claim 12 , wherein storing at least a portion of the data records as log files of the data store comprises:
determining that a data record satisfies a condition of evicting data records from a memory device to the distributed file system as specified in an eviction setting in the DDL statement; and
in response, moving the data record from the memory device to the log files.
14. The system of claim 11 , wherein the mapper is configured to filter and sort the log files.
15. The system of claim 11 , the operations comprising storing a data manipulation language (DML) statement defining a schema of the database table in a metadata file in the data store.