Systems and methods of managing manifest refresh in a database
Systems and methods are provided for receiving, at a database system having a memory and at least one persistent storage device to store records, a query for a least one record, where the query uses a first version of a manifest, and where each version of the records that are stored in the at least one persistent storage device are represented by metadata that is part of the first version of the manifest. A first operation may be performed based on a scan operation. The database system may determine whether a purge of the memory has occurred after the first operation. When it is determined that the memory purge has occurred, the scan operation may be restarted from a last position of the scan operation prior to the memory purge using a second version of the manifest.
1. A method comprising:
receiving, at a database system having a memory and at least one persistent storage device to store records, a query for a least one record, wherein the query uses a first version of a manifest, and wherein each version of the records that are stored in the at least one persistent storage device are represented by metadata that is part of the first version of the manifest;
starting a scan operation of at least one of the records stored in the memory and the records that are stored in the at least one persistent storage device based on the received query;
performing a first operation based on the scan operation;
determining, at the database system, whether a purge of the memory has occurred after the first operation and while performing the query; and
when it is determined that the memory purge has occurred while performing the query, restarting the scan operation from a last position of the scan operation prior to the memory purge using a second version of the manifest.
2. The method of claim 1 , wherein the first operation is selected from the group consisting of: a retrieve next record operation, a sort operation, and a lock wait operation.
3. The method of claim 1 , wherein the restarting the scan operation comprises:
refreshing the manifest to generate the second version of the manifest.
4. The method of claim 1 , wherein the scan operation includes one or more sub-scan operations, based on the received query.
5. The method of claim 1 , further comprising:
creating, at the database system, new records that are stored in the memory;
flushing the records in the memory that are committed to the at least one persistent storage device,
wherein the purge of the memory is performed after the flushing.
6. The method of claim 1 , wherein when the query is using the first version of the manifest, the query reads the records up to a maximum commit number for the first version of the manifest, and determines that any newer records than the maximum commit number are stored in the memory.
7. The method of claim 1 , further comprising:
purging the memory, at the database system, without regard to the received query.
8. The method of claim 1 , further comprising:
managing, by the received query, the detection of the memory purge; and
using, by the received query, the second version of the manifest.
9. A database system comprising:
a memory and at least one persistent storage device to store records;
a processor to receive a query for a least one record, wherein the query uses a first version of a manifest, and wherein each version of the records that are stored in persistent storage are represented by metadata that is part of the first version of the manifest, to start a scan operation of at least one of the records that are stored in memory and the records stored in the at least one persistent storage device based on the received query;
performing a first operation based on the scan operation;
determining, at the database system, whether a purge of the memory has occurred after the first operation and while performing the query; and
when it is determined that the memory purge has occurred while performing the query, restarting the scan operation from a last position of the scan operation prior to the memory purge using a second version of the manifest.
10. The system of claim 9 , wherein the first operation is selected from the group consisting of: a retrieve next record operation, a sort operation, and a lock wait operation.
11. The system of claim 9 , wherein the processor restarts the scan operation by refreshing the manifest to generate the second version of the manifest.
12. The system of claim 9 , wherein the scan operation performed by the processor includes one or more sub-scan operations, based on the received query.
13. The system of claim 9 , wherein the processor creates new records that are stored in the memory, and flushes the records in the memory that are committed to the at least one persistent storage device,
wherein the purge of the memory is performed by the processor after the flushing.
14. The system of claim 9 , wherein when the query is using the first version of the manifest, the query reads the records up to a maximum commit number for the first version of the manifest, and the processor determines that any newer records than the maximum commit number are stored in the memory.
15. The system of claim 9 , wherein the processor purges the memory without regard to the received query.
16. The system of claim 9 , wherein the processor manages, by the received query, the detection of the memory purge, and the query uses the second version of the manifest.