Compound partition and clustering keys
A data storage service implements encoding techniques for generating compound partition keys, compound clustering keys, or a combination for more flexibility in storing data in a key-value database system. In some aspects a request to store data with multiple partition keys and/or multiple clustering keys may be received. Length information may be added to the partition keys, and the results concatenated together to form a compound partition key used to store the data in the database. Ordering information may be added to the clustering keys, and the results concatenated together to form a compound clustering key, to be used in storing and sorting data in the database. In some cases, the clustering keys may be made a uniform length, such as by padding keys, as necessary, to have a length that is a multiple of an arbitrary number, and then dividing the clustering keys into chunks of that arbitrary number.
1. A system comprising
a front end host comprising a customer interface;
a data storage service accessible via the front end host, wherein the data storage service comprises a plurality of nodes of a table-oriented database, and wherein the front end host includes instructions that as a result of being executed by the system, cause the system to:
receive a request to store data in a data table with the data storage service, wherein the request specifies a compound clustering key comprising at least two key values;
generate a compound clustering key for the data by:
padding each of the key values to produce padded key values that have a length that is a multiple of an arbitrary number;
dividing the padded key values into a number of chunks having a size equal to the arbitrary number;
appending metadata to each of the number of chunks to produce modified chunks, wherein the metadata indicates an order of individual chunks of the number of chunks; and
concatenating the modified chunks together; and
cause the data to be stored in the data table with the data storage service according to the compound clustering key.
2. The system of claim 1 , wherein the request specifies a compound partition key comprising at least two partition values, and wherein the instructions that as a result of being executed by the system, further cause the system to:
generate a compound partition key for the data by:
converting each of the at least two partition values to binary partition values;
adding metadata to each of the binary partition values to produce modified partition values, wherein the metadata indicates a length of individual binary partition values of the binary partition values; and
concatenate the modified partition values together to generate the compound partition key; and
cause the data of the data table to be stored by a node of the plurality of nodes of the data storage service according to the compound partition key and the compound clustering key.
3. The system of claim 1 , wherein the instructions that as a result of being executed by the system, further cause the system to:
append first metadata to at least a beginning chunk of the number of chunks; and
append second metadata to the last chunk of the number of chunks, the second metadata indicating a number of non-padded bytes in the chunk.
4. The system of claim 1 , wherein the instructions that as a result of being executed by the system, further cause the system to:
receive a request to perform an operation on the data of the data table, the response specifying at least one of the at least two keys values;
responsive to receiving the request, encode the at least one of the at least two key values to generate an access key, the access key comprising at least part of the compound clustering key; and
instruct a node of the plurality of nodes of the data storage service to perform the operation on the data of the data table using the access key.
5. A computer-implemented method comprising:
receiving a request to store data in a data table with a data storage service, wherein the request specifies a compound partition key comprising at least two partition key values and a compound clustering key comprising at least two clustering key values;
concatenating the at least two partition key values together to generate the compound partition key;
generating a compound clustering key for the data by:
appending metadata to each of the at least two clustering key values to produce at least two modified clustering key values, wherein the metadata preserves an order of portions of the at least two clustering keys; and
concatenating the at least two modified clustering key values together; and
causing the data to be stored in the data table with the data storage service according to the compound partition key and the compound clustering key.
6. The computer-implemented method of claim 5 , wherein generating the compound clustering key further comprises dividing the at least two clustering key values into a number of chunks having a uniform length.
7. The computer-implemented method of claim 6 , wherein generating the compound clustering key further comprises generating a complement of at least one of the number of chunks to produce reverse order chunks responsive to an instruction to store or access at least part of the data in reverse sort order.
8. The computer-implemented method of claim 5 , wherein generating the compound clustering key further comprises:
padding each of the at least two clustering key values to produce padded clustering key values that have a length that is a multiple of an arbitrary number; and
dividing the padded clustering key values into a number of chunks having a size equal to the arbitrary number.
9. The computer-implemented method of claim 8 , wherein appending the metadata to each of the at least two clustering key values further comprises:
appending first metadata to at least a beginning chunk of the number of chunks; and
appending second metadata to the last chunk of the number of chunks, the second metadata indicating a terminal chunk.
10. The computer-implemented method of claim 5 , further comprising:
appending length metadata to each of the at least two partition key values to produce modified partition key values, wherein the length metadata indicates a length of individual partition key values of the at least two partition key values; and
concatenating the modified partition key values to generate the compound partition key.
11. The computer-implemented method of claim 5 , further comprising:
performing a hash function on the compound partition key to generated a hashed compound partition key; and
causing the data to be stored in the data table with the data storage service according to the hashed compound partition key and the compound clustering key.
12. The computer-implemented method of claim 5 , wherein the compound partition key has a predefined length.
13. A non-transitory computer-readable storage medium storing thereon executable instructions that, as a result of being executed by one or more processors of a computer system, cause the computer system to at least:
receive a request to store data in a data table with a data storage service, wherein the request specifies a compound clustering key comprising at least two key values;
generate a compound clustering key for the data by:
dividing the key values into a number of chunks having a size equal to an arbitrary number; and
appending metadata to each of the number of chunks to produce modified chunks, wherein the metadata indicates an order of individual chunks of the number of chunks; and
concatenating the modified chunks together; and
cause the data to be stored in the data table with the data storage service according to the compound clustering key.
14. The non-transitory computer-readable storage medium of claim 13 , wherein the request specifies a partition key, and wherein the instructions that, as a result of being executed by the one or more processors, further cause the computer system to:
causing the data of the data table to be stored with the data storage service according to the partition key and the compound clustering key.
15. The non-transitory computer-readable storage medium of claim 13 , wherein the request specifies a compound partition key comprising at least two partition values, and wherein the instructions that, as a result of being executed by the one or more processors, further cause the computer system to:
generate a compound partition key for the data by:
converting each of the at least two partition values to binary partition values;
prepending metadata to each of the binary partition values to produce modified partition values, wherein the metadata indicates a length of individual binary partition values of the binary partition values; and
concatenating the modified partition values together to generate the compound partition key; and
cause the data of the data table to be stored with the data storage service according to the compound partition key and the compound clustering key.
16. The non-transitory computer-readable storage medium of claim 13 , further comprising:
converting each of the at least two key values to binary key values; and
padding each of the key values to produce padded key values that are a multiple of the arbitrary number prior to dividing the key values into the number of chunks.
17. The non-transitory computer-readable storage medium of claim 13 , wherein the instructions that, as a result of being executed by the one or more processors, further cause the computer system to generate a compliment of at least one of the modified chunks to produce reverse order modified chunks responsive to an instruction to store at least one of the key values in reverse order.
18. The non-transitory computer-readable storage medium of claim 13 , wherein the instructions that, as a result of being executed by the one or more processors, further cause the computer system to:
append first metadata to at least a beginning chunk of the number of chunks; and
append second metadata to the last chunk of the number of chunks, the second metadata indicating a number of non-padded bytes in the chunk.
19. The non-transitory computer-readable storage medium of claim 13 , wherein the instructions that, as a result of being executed by the one or more processors, further cause the computer system to append first metadata to the last chunk of the number of chunks, the first metadata indicating a terminating chunk.
20. The non-transitory computer-readable storage medium of claim 13 , wherein the instructions that, as a result of being executed by the one or more processors, further cause the computer system to:
receive a request to perform an operation on the data of the data table, the response specifying at least one of the at least two keys values;
responsive to receiving the request, encode the at least one of the at least two key values to generate an access key, the access key comprising at least part of the compound clustering key; and
instruct a storage node of the data storage service to perform the operation on the data of the data table using the access key.