Separation of storage and compute in database management systems

1 . A method for managing data in a database management system, the database management system comprising a first node, the first node being a master storage node for storing user data from a database, and a second node, the second node performing operations on user data stored in the database, wherein the operations include reading or writing the user data,

wherein the method comprises configuring the second node to:

write log data indicative of one or more operations that have been performed on user data in the database to a set of log files, the set of log files being used to record operations performed on user data in the database and being stored in non-volatile storage of the second node, wherein the one or more operations are made permanent in the database once the log data is written to the set of log files in the non-volatile storage of the second node;

segment the set of log files into a set of log file chunks, wherein each log file chunk is a portion of a log file in the set of log files; and

for a said log file chunk, transmit user data, corresponding to log data in the said log file chunk and comprising a binary large object, to the first node and thereafter transmit the said log file chunk to the first node,

wherein the second node segments the set of log files and transmits the log file chunks and corresponding user data to the first node asynchronously to its writing of log data to the set of log files.

2 . The method according to claim 1 , wherein each log file of the set of log files comprises a set of log records, each log record representing a single operation and being associated with a log sequence indicator representing a relative order of the single operation with respect to other operations.

3 . The method according to claim 2 , comprising configuring the first node to:

receive log file chunks from the second node; and

reconstruct the set of log files from the log file chunks.

4 . The method according to claim 3 , wherein respective file names for each log file chunk comprise metadata including:

a first log sequence indicator that is a log sequence indicator representing an oldest log record in the log file chunk; and

a second log sequence indicator that is a log sequence indicator representing a newest log record in the log file chunk,

wherein the first node identifies adjacent log file chunks based at least on the respective first log sequence indicators and the respective second log sequence indicators for each log file chunk, whereby to reconstruct the set of log files from the log file chunks.

5 . The method according to claim 2 , comprising configuring the second node to write a checkpoint log record to the log files confirming that all log records up to and including the log records in the transmitted log file chunk have been transmitted to the first node.

6 . The method according to claim 5 , wherein, on receipt of the log file chunk, the first node provides a signal to the second node confirming receipt of the log file chunk, and wherein updating the log files at the second node to include the checkpoint log record is dependent on the second node receiving the signal confirming receipt of the log file chunk.

7 . The method according to claim 5 , comprising configuring the second node to perform a snapshot backup process, the snapshot backup process comprising:

writing a snapshot log record in the log files;

generating a snapshot of the user data in the database; and

ending the snapshot backup process after a log file chunk comprising the snapshot log record is transmitted to the first node and the set of log files include a checkpoint log record corresponding to the log file chunk comprising the snapshot log record.

8 . The method according to claim 5 , further comprising provisioning a third node, the third node performing operations on user data stored in a database, wherein provisioning the third node includes:

downloading first log files from the first node, the first log files comprising log records that are older than a most recent checkpoint log record generated at the second node;

downloading second log files from the second node, the second log files comprising log records that are more recent than the most recent checkpoint log record generated at the second node; and

replicating at least part of the user data stored in the first and second nodes at the third node using the downloaded first and second log files.

9 . The method according to claim 8 , wherein the first log files comprise log records that are:

older than a first specified log record;

more recent than the first specified log record; or

older than the first specified log record and more recent than a second specified log record.

10 . The method according to claim 8 , wherein the second log files comprise log records that are:

older than a first specified log record;

more recent than the first specified log record; or

older than the first specified log record and more recent than a second specified log record.

11 . The method according to claim 8 , wherein the first log files comprise log records that are more recent than a first specified snapshot log record and less recent than a second specified snapshot.

12 . The method according to claim 8 , wherein replicating at least part of the user data is dependent on receiving an instruction to perform an operation on the at least part of the user data at the third node.

13 . The method according to claim 8 , wherein the third node downloads a snapshot from the first node and performs operations corresponding to log records which are more recent than a snapshot log record corresponding to the downloaded snapshot, whereby to replicate at least part of the user data stored in the first and second nodes.

14 . The method according to claim 8 , wherein each binary large object corresponds to a respective blob creation log record in the set of log files, and the third node processes a blob creation log record by reserving a binary large object page without requiring the corresponding binary large object to be downloaded to the third node.

15 . The method according to claim 8 , comprising configuring the third node to:

on receipt of an instruction to perform an operation on a binary large object, identify whether the binary large object is stored in the third node; and

if the binary large object is not stored in the third node, download the binary large object from either the first or second node.

16 . The method according to claim 1 , comprising configuring the second node to:

apply a selection algorithm to identify user data to be deleted from the database; and

delete the identified user data,

wherein user data in the database which is identified for deletion has been transmitted to the first node.

17 . The method according to claim 16 , comprising configuring the second node to:

obtain an instruction to perform an operation on a portion of user data of the database which has been deleted;

in response to the instruction, download the portion of user data from the first node; and

perform the operation on the portion of user data according to the instruction.

18 . A database management system comprising:

a first node, the first node comprising at least one processor and storage comprising first computer-executable instructions which, when executed by the at least one processor, cause the first node to implement a master storage node for storing user data from a database; and

a second node, the second node performing operations on user data stored in the database, the operations comprising reading or writing the user data, wherein the second node comprises at least one processor and storage comprising second computer-executable instructions which, when executed by the least one processor of the second node, cause the second node to:

write log data indicative of one or more operations that have been performed on user data in the database to a set of log files, the set of log files being used to record operations performed on user data in the database and being stored in the storage of the second node, wherein the one or more operations are made permanent in the database once the log data is written to the set of log files;

segment the set of log files into a set of log file chunks, wherein each log file chunk is a portion of a log file in the set of log files; and

for a said log file chunk, transmit user data, corresponding to log data in the said log file chunk and comprising a binary large object, from the second node to the first node and thereafter transmit the log file chunk to the first node,

wherein the second node segments the set of log files and transmits the log file chunks and corresponding user data to the first node asynchronously to its writing of log data to the set of log files.

19 . The database management system according to claim 18 , wherein each log file of the set of log files comprises a set of log records, each log record representing a single operation and being associated with a log sequence indicator representing a relative order of the single operation with respect to other operations.

20 . A non-transitory computer-readable storage medium comprising computer executable instructions which, when executed by at least one processor, cause the at least one processor to:

implement a first node, the first node being a master storage node for storing user data from a database;

implement a second node for performing operations on data stored in the database, the operations comprising reading or writing the user data, the second node being configured to:

write log data indicative of one or more operations that have been performed on user data in the database to a set of log files, the set of log files being used to record operations performed on user data in the database and being stored in the storage of the second node, wherein the one or more operations are made permanent in the database once the log data is written to the set of log files;

segment the set of log files into a set of log file chunks, wherein each log file chunk is a portion of a log file in the set of log files; and

for a said log file chunk, transmit user data, corresponding to log data in the said log file chunk and comprising a binary large object, from the second node to the first node and thereafter transmit the log file chunk to the first node,

wherein the second node segments the set of log files and transmits the log file chunks and corresponding user data to the first node asynchronously to its writing of log data to the set of log files.