Storing an ordered associative array of pairs using an append-only storage medium
The described technology is generally directed towards efficiently organizing data in a single, non-distributed database as an ordered associated array of key-value pairs implemented on top of a storage medium that allows only data appends. A table segment in an append-only storage medium is accessed by table keys. The table keys are hashed into key values used to access information in an attribute index (a search tree) that contains offsets to the table segment entries. Hashing the table keys can result in a series of hash parts, including a primary hash part corresponding to a node in the attribute index, and as needed in the event of hash collisions, secondary hash parts that map to child nodes in the attribute index. The table keys are additionally organized in a sorted data structure which is stored within the table segment along with other keys.
1. A method, comprising:
to organize data for storage via an append-only storage medium, processing, by a system comprising a hardware processor, a group of updates directed to a table segment implemented in the append-only storage medium, the processing comprising:
receiving the group of updates,
wherein an update in the group of updates comprises a key and a value, and
wherein the update in the group of updates comprises a property to indicate whether the update is a conditional update;
in response to the property indicating that the update in the group of updates is a conditional update, validating the update and determining whether to allow the update based on a result of the validating, wherein validating the update comprises querying the tail portion of the table segment index to determine whether the tail portion of the table segment index comprises an expected value;
serializing the group of updates;
appending the group of updates to the table segment;
updating a tail portion of a table segment index to include an offset corresponding to the key; and
updating a tail portion of a sorted key index to include the key.
2. The method of claim 1 , wherein updating the tail portion of the sorted key index comprises updating a volatile memory comprising the tail portion of the sorted key index.
3. The method of claim 1 , further comprising, in response to the property indicating that the update in the group of updates is the conditional update, collecting all conditional updates in the group of updates.
4. The method of claim 1 , further comprising, in response to the property indicating that the update in the group of updates is the conditional update, waiting for completion of another update.
5. The method of claim 2 , further comprising maintaining a persisted portion of the sorted key index in a non-volatile memory, wherein the persisted portion of the sorted key index comprises a tree data structure comprising leaf nodes, the leaf nodes comprising keys organized in a sorted order.
6. The method of claim 1 , wherein the tail portion of the sorted key index comprises keys organized in a sorted array.
7. The method of claim 1 , wherein validating the update further comprises, in response to the tail portion of the table segment index not comprising the expected value, querying a persisted portion of the table segment index to determine whether the persisted portion of the table segment index comprises the expected value.
8. A system, comprising:
a hardware processor; and
a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations that organize data for storage via an append-only storage medium, the operations comprising:
processing a group of updates directed to a table segment, the processing comprising:
receiving the group of updates,
wherein an update in the group of updates comprises a key and a value, and
wherein the update in the group of updates comprises a property to indicate whether the update is a conditional update;
validating the update and determining whether to allow the update based on a result of the validating, wherein validating the update comprises querying the tail portion of the table segment index to determine whether the tail portion of the table segment index comprises an expected value;
serializing the group of updates;
appending the group of updates to the table segment;
updating a tail portion of a table segment index to include an offset corresponding to the key; and
updating a tail portion of a sorted key index to include the key.
9. The system of claim 8 , wherein the table segment is implemented in the append-only storage medium.
10. The system of claim 8 , wherein validating the update and determining whether to allow the update based on a result of the validating is in response to the property indicating that the update in the group of updates is the conditional update.
11. The system of claim 8 , wherein the tail portion of the sorted key index comprises a flag associated with the key, and wherein the flag indicates whether the key has been deleted.
12. The system of claim 8 , wherein the operations further comprise, in response to a range request for keys in a range [K1 . . . K2], generating a list comprising keys between K1 and K2.
13. The system of claim 8 , wherein validating the update further comprises querying a persisted portion of the table segment index to determine whether the persisted portion of the table segment index comprises the expected value.
14. A non-transitory machine-readable medium comprising executable instructions to organize data for storage via an append-only storage medium, wherein when executed by a hardware processor, the executable instructions facilitate performance of operations, the operations comprising:
processing a group of updates directed to a table segment, the processing comprising:
receiving the group of updates,
wherein an update in the group of updates comprises a key and a value, and
wherein the update in the group of updates comprises a property to indicate whether the update is a conditional update;
validating the update and determining whether to allow the update based on a result of the validating, wherein validating the update comprises querying a persisted portion of the table segment index to determine whether the persisted portion of the table segment index comprises the expected value;
serializing the group of updates;
appending the group of updates to the table segment;
updating a tail portion of a table segment index to include an offset corresponding to the key; and
updating a tail portion of a sorted key index to include the key.
15. The non-transitory machine-readable medium of claim 14 , wherein validating the update further comprises querying the tail portion of the table segment index to determine whether the tail portion of the table segment index comprises an expected value.
16. The non-transitory machine-readable medium of claim 14 , wherein the operations further comprise receiving a request for a list of keys in the table segment index.
17. The non-transitory machine-readable medium of claim 16 , wherein the request for the list of keys in the table segment comprises a lower bound value and an upper bound value.
18. The non-transitory machine-readable medium of claim 16 , wherein the operations further comprise querying the tail portion of the sorted key index for a first list of keys that satisfy the request.
19. The non-transitory machine-readable medium of claim 18 , wherein the operations further comprise querying a persisted portion of the sorted key index for a second list of keys that satisfy the request.
20. The non-transitory machine-readable medium of claim 19 , wherein the operations further comprise merging the first list of keys and the second list of keys into a merged list of keys and returning the merged list of keys in response to the request.