Incremental addition of data to partitions in database tables
View Patent ↗A method and system for accessing updated data from a database in response to a user query has been developed. First, multiple transaction logs are generated for a database. Each transaction log contains a record of actions executed by a database management system and referenced according to the specified date of the actions. Data updates are received and stored with the database. An incremental database partition is created for each data update. Each incremental database partition is stored with reference to a corresponding transaction log for the date of the data update. The updated data is accessed through the incremental database partition in response to an outdated user query. The outdated user query contains a data access request for a date earlier than the receipt of data updates.
1. A method for accessing updated data from a database in response to a user query, comprising:
generating multiple transaction logs for the database, where each transaction log contains a record of actions executed by a database management system and referenced according to the specified date of the actions, where the database is a Hive database;
receiving data updates that are stored with the database;
creating an incremental database partition for each data update, where a most recent incremental database partition is stored in addition to all previous incremental database partitions and stored with reference to a corresponding transaction log with a generation date of the data update so that no duplicate copies of the data update with an earlier generation date are stored in the database;
receiving a user query for accessing data in the database, where the user query is an outdated user query; and
accessing updated data with no downtime through the incremental database partition in response to the outdated user query without changing the content of the outdated user query, where the outdated user query contains a data access request for a date earlier than the receipt of data updates.
2. The method of claim 1 , where the user query uses HiveQL query language.
3. The method of claim 2 , where HiveQL supports ACID (atomicity, consistency, isolation and durability) properties of operations of the database.
4. The method of claim 1 , where the database management system is HDFS (Hadoop Distributed File System).
5. The method of claim 1 , where the database utilizes a YARN (Yet Another Resource Negotiator) execution engine.
6. A system for accessing updated data from a database in response to
a user query, comprising:
a microprocessor; and
a memory coupled to the processor, wherein the memory includes computer program instructions capable of,
generating multiple transaction logs for the database, where each transaction log contains a record of actions executed by a database management system and referenced according to the specified date of the actions, where the database is a Hive database,
receiving data updates that are stored with the database,
creating an incremental database partition for each data update, where a most recent incremental database partition is stored in addition to all previous incremental database partitions and stored with reference to a corresponding transaction log with a generation date of the data update so that no duplicate copies of the data update with an earlier generation date are stored in the database,
receiving a user query for accessing data in the database, where the user query is an outdated user query, and
accessing updated data with no downtime through the incremental database partition in response to the outdated user query without changing the content of the outdated user query, where the outdated user query contains a data access request for a date earlier than the receipt of data updates.
7. The system of claim 6 , where the user query uses HiveQL query language.
8. The system of claim 7 , where HiveQL supports ACID (atomicity, consistency, isolation and durability) properties of operations of the database.
9. The system of claim 6 , where the database management system is HDFS (Hadoop Distributed File System).
10. The system of claim 6 , where the database utilizes a YARN (Yet Another Resource Negotiator) execution engine.
11. A non-transitory computer readable storage media comprising:
computer instructions to implement,
generating multiple transaction logs for the database, where each transaction log contains a record of actions executed by a database management system and referenced according to the specified date of the actions, where the database is a Hive database,
receiving data updates that are stored with the database,
creating an incremental database partition for each data update, where a most recent incremental database partition is stored in addition to all previous incremental database partitions and stored with reference to a corresponding transaction log with a generation date of the data update so that no duplicate copies of the data update with an earlier generation date are stored in the database,
receiving a user query for accessing data in the database, where the user query is an outdated user query, and
accessing updated data with no downtime through the incremental database partition in response to the outdated user query without changing the content of the outdated user query, where the outdated user query contains a data access request for a date earlier than the receipt of data updates.
12. The storage media of claim 11 , where the user query uses HiveQL query language.
13. The storage media of claim 12 , where HiveQL supports ACID (atomicity, consistency, isolation and durability) properties of operations of the database.
14. The storage media of claim 11 , where the database management system is HDFS (Hadoop Distributed File System).
15. The storage media of claim 11 , where the database utilizes a YARN (Yet Another Resource Negotiator) execution engine.