Space management for transactional consistency of in-memory objects on a standby database
Embodiments store transaction metadata in dedicated pools of allocated memory chunks. Portions of the pools of allocated memory chunks are dedicated to the respective apply slave processes that mine and process change records. Also, the pools of allocated memory chunks are anchored within the structure of a transaction log such that buffering and application of metadata for one transaction does not block required buffering and application of metadata for other transactions. The standby database system pre-processes transaction metadata in preparation for application of the metadata to invalidate appropriate portions of MF data. Further, embodiments divide the work of pre-processing invalidation records among the many apply slave processes that record the invalidation records. A garbage collection selects memory chunks for garbage collection in reverse order of how the chunks were allocated. Also, a deduplication algorithm ensures that typically only a single invalidation message per block is applied to invalidate MF data.
1. A method comprising:
maintaining, on persistent storage, a first database that is accessible to a first database server;
wherein said first database includes a set of persistent format data (PF data), stored on said persistent storage, in a persistent format;
converting said set of PF data to a mirror format to produce a set of mirror format data (MF data);
storing said set of MF data within volatile memory;
wherein said mirror format is different from and independent of said persistent format;
maintaining a particular pool of allocated memory chunks comprising a plurality of allocated memory chunks that are allocated from a particular area of memory that is accessible to the first database server;
wherein the particular area of memory is dedicated to one or more pools of memory chunks including the particular pool of allocated memory chunks;
wherein a particular memory chunk, of the plurality of allocated memory chunks, comprises a plurality of fixed-length memory slabs;
wherein the plurality of fixed-length memory slabs comprises one or more vacant memory slabs;
receiving, by said first database server, one or more change records from a second database server, said one or more change records indicating a particular transaction performed against a second database;
applying a first change record of the one or more change records to said set of PF data;
wherein applying the first change record to said set of PF data comprises:
allocating a particular memory slab, from the one or more vacant memory slabs of the particular memory chunk, to store a transaction control structure for the particular transaction, and
storing transaction metadata derived from the first change record within the transaction control structure stored in the particular memory slab;
committing the particular transaction in the set of PF data; and
after committing the particular transaction in the set of PF data and based, at least in part, on the transaction metadata stored in the transaction control structure, invalidating any of the set of MF data that is changed by the particular transaction;
wherein said method is performed by one or more computing devices.
2. The method of claim 1 , further comprising:
hashing a transaction identifier that identifies the particular transaction to produce an index into a transaction log hash table;
wherein the index identifies a particular hash bucket of the transaction log hash table; and
identifying the particular pool of allocated memory chunks based on the particular pool being associated with the particular hash bucket;
wherein allocating the particular memory slab, from the particular memory chunk, is performed in response to identifying the particular pool of allocated memory chunks based on the particular pool being associated with the particular hash bucket.
3. The method of claim 1 , wherein allocating the particular memory slab, from the one or more vacant memory slabs of the particular memory chunk, is performed in response to determining that the particular memory chunk is the earliest allocated memory chunk, among the plurality of allocated memory chunks in the particular pool, that has a vacant slab of memory.
4. The method of claim 3 , wherein determining that the particular memory chunk is the earliest allocated memory chunk, among the plurality of allocated memory chunks in the particular pool, that has a vacant slab of memory is based on a bit vector, maintained for the particular memory chunk, that records which slabs of memory in the particular memory chunk are vacant.
5. The method of claim 1 , wherein applying the first change record to said set of PF data further comprises:
allocating a second memory slab from a second memory chunk of a second pool of allocated memory chunks;
wherein the second memory chunk comprises a second plurality of fixed-length memory slabs that includes the second memory slab;
associating the second memory slab with the transaction control structure as a location of an initial invalidation bundle for the particular transaction;
wherein the second memory slab is configured to store metadata for a fixed plural number of change records; and
storing an invalidation record with data from the first change record in the second memory slab.
6. The method of claim 1 , wherein the one or more change records comprise two or more change records, the method further comprising:
a particular apply slave applying a second change record of the two or more change records to said set of PF data;
wherein applying the second change record to said set of PF data comprises:
allocating, to the particular apply slave, a memory granule from a transaction-specific memory chunk that is associated with the transaction control structure for the particular transaction,
recording information identifying the memory granule in a slave memory context within the transaction control structure, and
the particular apply slave storing an invalidation record with data from the second change record in the memory granule.
7. The method of claim 6 , further comprising:
the particular apply slave filling the memory granule with a plurality of invalidation records for the particular transaction; and
in response to filling the memory granule with invalidation records for the particular transaction, the particular apply slave sorting, by relative data block address, the plurality of invalidation records within the memory granule.
8. The method of claim 6 , further comprising:
the particular apply slave filling the memory granule with a plurality of invalidation records for the particular transaction; and
after filling the memory granule with invalidation records for the particular transaction, and in response to the particular apply slave identifying a change record, for the particular transaction, for which an invalidation record has not yet been recorded, allocating a second memory granule from a transaction-specific memory chunk that is associated with the transaction control structure for the particular transaction.
9. The method of claim 6 , wherein the memory granule is dedicated to invalidation records generated by the particular apply slave.
10. The method of claim 1 , wherein:
applying the first change record to said set of PF data further comprises:
a particular apply slave determining whether a relative data block address from the first change record is stored in a buffer structure within a slave memory context, for the particular apply slave, within the transaction control structure; and
in response to determining that the relative data block address is not stored in the buffer structure:
storing an invalidation record with information from the first change record in connection with the transaction control structure, and
storing the relative data block address in the buffer structure;
the one or more change records comprise a plurality of change records; and
the method further comprises:
applying a second change record of the plurality of change records to said set of PF data;
wherein the second change record includes the same relative data block address as the first change record; and
wherein applying the second change record to said set of PF data comprises:
the particular apply slave determining whether the relative data block address from the second change record is stored in the buffer structure; and
in response to determining that the relative data block address is stored in the buffer structure, applying a third change record of the plurality of change records without recording an invalidation record for the second change record.
11. The method of claim 1 , further comprising:
wherein allocating the particular memory slab to store the transaction control structure for the particular transaction comprises changing a particular bit, which corresponds to the particular memory slab in a bit vector, to indicate that the particular memory slab is occupied; and
after invalidating any of the set of MF data that is changed by the particular transaction, changing the particular bit, in the bit vector, to indicate that the particular memory slab is vacant.
12. One or more non-transitory computer-readable media storing one or more sequences of instructions that, when executed by one or more processors, cause:
maintaining, on persistent storage, a first database that is accessible to a first database server;
wherein said first database includes a set of persistent format data (PF data), stored on said persistent storage, in a persistent format;
converting said set of PF data to a mirror format to produce a set of mirror format data (MF data);
storing said set of MF data within volatile memory;
wherein said mirror format is different from and independent of said persistent format;
maintaining a particular pool of allocated memory chunks comprising a plurality of allocated memory chunks that are allocated from a particular area of memory that is accessible to the first database server;
wherein the particular area of memory is dedicated to one or more pools of memory chunks including the particular pool of allocated memory chunks;
wherein a particular memory chunk, of the plurality of allocated memory chunks, comprises a plurality of fixed-length memory slabs;
wherein the plurality of fixed-length memory slabs comprises one or more vacant memory slabs;
receiving, by said first database server, one or more change records from a second database server, said one or more change records indicating a particular transaction performed against a second database;
applying a first change record of the one or more change records to said set of PF data;
wherein applying the first change record to said set of PF data comprises:
allocating a particular memory slab, from the one or more vacant memory slabs of the particular memory chunk, to store a transaction control structure for the particular transaction, and
storing transaction metadata derived from the first change record within the transaction control structure stored in the particular memory slab;
committing the particular transaction in the set of PF data; and
after committing the particular transaction in the set of PF data and based, at least in part, on the transaction metadata stored in the transaction control structure, invalidating any of the set of MF data that is changed by the particular transaction.
13. The one or more non-transitory computer-readable media of claim 12 , wherein the one or more sequences of instructions further comprise instructions that, when executed by one or more processors, cause:
hashing a transaction identifier that identifies the particular transaction to produce an index into a transaction log hash table;
wherein the index identifies a particular hash bucket of the transaction log hash table; and
identifying the particular pool of allocated memory chunks based on the particular pool being associated with the particular hash bucket;
wherein allocating the particular memory slab, from the particular memory chunk, is performed in response to identifying the particular pool of allocated memory chunks based on the particular pool being associated with the particular hash bucket.
14. The one or more non-transitory computer-readable media of claim 12 , wherein allocating the particular memory slab, from the one or more vacant memory slabs of the particular memory chunk, is performed in response to determining that the particular memory chunk is the earliest allocated memory chunk, among the plurality of allocated memory chunks in the particular pool, that has a vacant slab of memory.
15. The one or more non-transitory computer-readable media of claim 14 , wherein determining that the particular memory chunk is the earliest allocated memory chunk, among the plurality of allocated memory chunks in the particular pool, that has a vacant slab of memory is based on a bit vector, maintained for the particular memory chunk, that records which slabs of memory in the particular memory chunk are vacant.
16. The one or more non-transitory computer-readable media of claim 12 , wherein applying the first change record to said set of PF data further comprises:
allocating a second memory slab from a second memory chunk of a second pool of allocated memory chunks;
wherein the second memory chunk comprises a second plurality of fixed-length memory slabs that includes the second memory slab;
associating the second memory slab with the transaction control structure as a location of an initial invalidation bundle for the particular transaction;
wherein the second memory slab is configured to store metadata for a fixed plural number of change records; and
storing an invalidation record with data from the first change record in the second memory slab.
17. The one or more non-transitory computer-readable media of claim 12 , wherein the one or more change records comprise two or more change records, wherein the one or more sequences of instructions further comprise instructions that, when executed by one or more processors, cause:
a particular apply slave applying a second change record of the two or more change records to said set of PF data;
wherein applying the second change record to said set of PF data comprises:
allocating, to the particular apply slave, a memory granule from a transaction-specific memory chunk that is associated with the transaction control structure for the particular transaction,
recording information identifying the memory granule in a slave memory context within the transaction control structure, and
the particular apply slave storing an invalidation record with data from the second change record in the memory granule.
18. The one or more non-transitory computer-readable media of claim 17 , wherein the one or more sequences of instructions further comprise instructions that, when executed by one or more processors, cause:
the particular apply slave filling the memory granule with a plurality of invalidation records for the particular transaction; and
in response to filling the memory granule with invalidation records for the particular transaction, the particular apply slave sorting, by relative data block address, the plurality of invalidation records within the memory granule.
19. The one or more non-transitory computer-readable media of claim 17 , wherein the one or more sequences of instructions further comprise instructions that, when executed by one or more processors, cause:
the particular apply slave filling the memory granule with a plurality of invalidation records for the particular transaction; and
after filling the memory granule with invalidation records for the particular transaction, and in response to the particular apply slave identifying a change record, for the particular transaction, for which an invalidation record has not yet been recorded, allocating a second memory granule from a transaction-specific memory chunk that is associated with the transaction control structure for the particular transaction.
20. The one or more non-transitory computer-readable media of claim 17 , wherein the memory granule is dedicated to invalidation records generated by the particular apply slave.
21. The one or more non-transitory computer-readable media of claim 12 , wherein:
applying the first change record to said set of PF data further comprises:
a particular apply slave determining whether a relative data block address from the first change record is stored in a buffer structure within a slave memory context, for the particular apply slave, within the transaction control structure; and
in response to determining that the relative data block address is not stored in the buffer structure:
storing an invalidation record with information from the first change record in connection with the transaction control structure, and
storing the relative data block address in the buffer structure;
the one or more change records comprise a plurality of change records; and
the one or more sequences of instructions further comprise instructions that, when executed by one or more processors, cause:
applying a second change record of the plurality of change records to said set of PF data;
wherein the second change record includes the same relative data block address as the first change record; and
wherein applying the second change record to said set of PF data comprises:
the particular apply slave determining whether the relative data block address from the second change record is stored in the buffer structure; and
in response to determining that the relative data block address is stored in the buffer structure, applying a third change record of the plurality of change records without recording an invalidation record for the second change record.
22. The one or more non-transitory computer-readable media of claim 12 , wherein the one or more sequences of instructions further comprise instructions that, when executed by one or more processors, cause:
wherein allocating the particular memory slab to store the transaction control structure for the particular transaction comprises changing a particular bit, which corresponds to the particular memory slab in a bit vector, to indicate that the particular memory slab is occupied; and
after invalidating any of the set of MF data that is changed by the particular transaction, changing the particular bit, in the bit vector, to indicate that the particular memory slab is vacant.