Optimizations to read and write transactions for large values in distributed databases
The subject technology receives a key-value pair. The subject technology determines a value size of the key-value pair is greater than a threshold size for an out of line blob (OLB) key space. The subject technology performs a write operation of a key-pointer pair into a regular space. The subject technology performs a plurality of write operations for multiple key-value parts of the key-value pair into the OLB key space. The subject technology performs conflict checking of the regular space for the key-value pair.
1 . A system comprising:
at least one hardware processor; and
a memory storing instructions that cause the at least one hardware processor to perform operations comprising:
receiving a statement including a key-value pair, the statement comprising a command to perform a write operation on a linearizable database based on the key-value pair;
determining a value size of the key-value pair is greater than a threshold size for an out of line blob (OLB) key space, the OLB key space being provided by a cloud storage platform;
performing, by a transaction manager, a first write operation of a key-pointer pair into a regular space, the transaction manager being separate from the cloud storage platform, the cloud storage platform providing the regular space, the key-pointer pair comprising a key, and a stamp, the stamp comprising a transaction identifier assigned to an SQL-level transaction that persists across multiple statement executions and restart attempts within a same transaction, a statement number identifying a SQL statement within the SQL-level transaction, and a restart count, the restart count tracking which statement restart attempt generated the first write operation, the restart count being incremented for each restart of the statement while the stamp maintains a same transaction identifier as the transaction identifier;
performing, using the stamp, a plurality of second write operations for multiple key-value parts of the key-value pair into the OLB key space; and
performing conflict checking of the regular space for the key-value pair.
2 . The system of claim 1 , wherein the OLB key space and the regular space are different storage spaces in the cloud storage platform, the statement comprises an SQL statement, and the write operation comprising a particular SQL statement to perform an insert or an update to the linearizable database, the transaction manager is included in a compute service manager, the compute service manager executes in a network-based data warehouse system, the compute service manager receiving the SQL statement from a client device, the client device being separate from the compute service manager and the network-based data warehouse system.
3 . The system of claim 1 , wherein the linearizable database comprises a distributed database, the distributed database is stored in the cloud storage platform, the cloud storage platform is a remote system from a compute service manager that has received the statement, the compute service manager executes in a network-based data warehouse system, the compute service manager receives the statement from a client device, the client device being separate from the compute service manager and the network-based data warehouse system, and the statement comprises an SQL statement.
4 . The system of claim 1 , wherein the key-pointer pair further comprises a set of control bits, a value indicating a number of parts of the key-value pair, a pointer, the pointer being based on the stamp, the set of control bits including one bit to indicate whether the key corresponds to the pointer to the OLB key space, a worker identifier, and a write number.
5 . The system of claim 4 , wherein performing the plurality of second write operations for multiple key-value parts into the OLB key space comprises:
performing a particular write operation of first data in the OLB key space, the first data comprising the key, a first pointer, and a first part from the multiple key-value parts of the key-value pair.
6 . The system of claim 5 , wherein the first pointer comprises the stamp and a value indicating a part number from the multiple key-value parts of the key-value pair.
7 . The system of claim 1 , wherein the multiple key-value parts are generated by splitting a large value from the key-value pair into different parts, the large value comprising a particular value greater than the threshold size.
8 . The system of claim 1 , wherein the operations further comprise:
receiving a particular request to perform a read operation on a key;
locating a visible version of the key in the regular space;
determining that a tombstone bit has not been set of the key;
determining that an OLB bit is set for the visible version of the key in the regular space;
performing a read operation based on a value of the visible version of the key in the OLB key space;
reassembling a large value based on the read operation of the OLB key space; and
providing the reassembled large value in response to the particular request.
9 . A method comprising:
receiving a statement including a key-value pair, the statement comprising a command to perform a write operation on a linearizable database based on the key-value pair;
determining a value size of the key-value pair is greater than a threshold size for an out of line blob (OLB) key space, the OLB key space being provided by a cloud storage platform;
performing, by a transaction manager, a first write operation of a key-pointer pair into a regular space, the transaction manager being separate from the cloud storage platform, the cloud storage platform providing the regular space, the key-pointer pair comprising a key, and a stamp, the stamp comprising a transaction identifier assigned to an SQL-level transaction that persists across multiple statement executions and restart attempts within a same transaction, a statement number identifying a SQL statement within the SQL-level transaction, and a restart count, the restart count tracking which statement restart attempt generated the first write operation, the restart count being incremented for each restart of the statement while the stamp maintains a same transaction identifier as the transaction identifier;
performing, using the stamp, a plurality of second write operations for multiple key-value parts of the key-value pair into the OLB key space; and
performing conflict checking of the regular space for the key-value pair.
10 . The method of claim 9 , wherein the OLB key space and the regular space are different storage spaces in the cloud storage platform.
11 . The method of claim 9 , wherein the linearizable database comprises a distributed database, the distributed database is stored in the cloud storage platform, and the cloud storage platform is a remote system from a compute service manager that has received the statement.
12 . The method of claim 9 , wherein the key-pointer pair further comprises a set of control bits, a value indicating a number of parts of the key-value pair, a pointer, the pointer being based on the stamp, the set of control bits including one bit to indicate whether the key corresponds to the pointer to the OLB key space, a worker identifier, and a write number.
13 . The method of claim 12 , wherein performing the plurality of second write operations for multiple key-value parts into the OLB key space comprises:
performing a particular write operation of first data in the OLB key space, the first data comprising the key, a first pointer, and a first part from the multiple key-value parts of the key-value pair.
14 . The method of claim 13 , wherein the first pointer comprises the stamp and a value indicating a part number from the multiple key-value parts of the key-value pair.
15 . The method of claim 9 , wherein the multiple key-value parts are generated by splitting a large value from the key-value pair into different parts, the large value comprising a particular value greater than the threshold size.
16 . A non-transitory computer-storage medium comprising instructions that, when executed by one or more processors of a machine, configure the machine to perform operations comprising:
receiving a statement including a key-value pair, the statement comprising a command to perform a write operation on a linearizable database based on the key-value pair;
determining a value size of the key-value pair is greater than a threshold size for an out of line blob (OLB) key space, the OLB key space being provided by a cloud storage platform;
performing, by a transaction manager, a first write operation of a key-pointer pair into a regular space, the transaction manager being separate from the cloud storage platform, the cloud storage platform providing the regular space, the key-pointer pair comprising a key, and a stamp, the stamp comprising a transaction identifier assigned to an SQL-level transaction that persists across multiple statement executions and restart attempts within a same transaction, a statement number identifying a SQL statement within the SQL-level transaction, and a restart count, the restart count tracking which statement restart attempt generated the first write operation, the restart count being incremented for each restart of the statement while the stamp maintains a same transaction identifier as the transaction identifier;
performing, using the stamp, a plurality of second write operations for multiple key-value parts of the key-value pair into the OLB key space; and
performing conflict checking of the regular space for the key-value pair.