Dynamically partition data
Techniques and systems can partition data with a partition key to provide data partitioned based on the partition key. Data writes to the data partitioned based on the partition key can be analyzed to identify a predicate shared by at least a plurality of the data writes. That predicate can be used to reparation the data to generate other partitioned data or to replace the data partitioned based on the partition key.
1 . A computer-implemented method, comprising:
storing, by a computer-implemented system, data comprising columns of data, each column of data having an associated column identifier;
generating, by the computer-implemented system, a partition key based on a first column identifier of the column identifiers associated with the columns of data;
partitioning, by the computer-implemented system, the data based on the partition key;
analyzing, by the computer-implemented system, aggregated data write information associated with data writes to the partitioned data based on the partition key to identify a write predicate common to at least a plurality of the data writes;
determining, by the computer-implemented system, a percentage based on a number of times the write predicate was determined to be associated with the data write information within a predetermined duration;
determining, by the computer-implemented system, the percentage satisfies a percentage threshold;
determining, by the computer-implemented system, the write predicate corresponds to a second column identifier of the column identifiers associated with the columns of data;
generating, by the computer-implemented system, another partition key based on the second column identifier;
based on determining the percentage satisfies the percentage threshold, partitioning, by the computer-implemented system, the data based on the generated another partition key to generate partitioned data based on the generated another partition key; and
writing, by the computer-implemented system, data to the data partitioned based on the generated another partition key.
2 . The computer-implemented method of claim 1 , further comprising:
storing, by the computer-implemented system, data write information over a predetermined duration to generate the aggregated data write information.
3 . The computer-implemented method of claim 1 , further comprising:
processing, by the computer-implemented system, data writes to the partitioned data based on the generated another ether partition key for a duration corresponding to a predetermined duration;
determining, by the computer-implemented system, the duration has lapsed; and
based on determining the duration has lapsed, processing, by the computer-implemented system, data writes to the partitioned data based on the partition key.
4 . The computer-implemented method of claim 1 , further comprising deleting, by the computer-implemented system, the partitioned data based on the partition key, wherein deleting the partitioned data based on the partition key is conditioned on a data retention parameter usable to determine when to retain or delete the partitioned data based on the partition key.
5 . A system, comprising:
one or more processors; and
memory that stores computer-executable instructions that are executable by the one or more processors to cause the system to:
analyze data write information, stored in computer-implemented storage, associated with data written to data partitioned based on a first partition key to identify a predicate common among at least a plurality of data writes to the partitioned data;
analyze the data write information to determine a percentage corresponding to a determined number of times the predicate was found to be associated with the plurality of data writes to computer-implemented storage over a predetermined duration; and
based on determining the percentage satisfies a percentage threshold:
generate a second partition key using the predicate, and
partition the data using the second partition key;
write data to the data portioned using the second partition key.
6 . The system of claim 5 , wherein the memory that stores the computer-executable instructions that are executable by the one or more processors are further to cause the system to:
generate the first partition key based on an identifier associated with a first structured data set of the data; and
generate the second partition key based on an identifier associated a second structured data set of the data,
wherein the second partition key corresponds to the predicate.
7 . The system of claim 6 , wherein the first structured data set is a first column of data comprised in the data and the second structured data set is a second column of data comprised in the data.
8 . The system of claim 5 , wherein the memory that stores the computer-executable instructions that are executable by the one or more processors are further to cause the system to:
process first real-time data writes to data partitioned based on the second partition key for a duration corresponding to a predetermined duration;
determine the duration has lapsed; and
based on determining the duration has lapsed, process second real-time data writes against the data partitioned based on the first partition key.
9 . The system of claim 8 , wherein the memory that stores the computer-executable instructions that are executable by the one or more processors are further to cause the system to:
determine the predetermined duration from a set of historical data writes to the data partitioned based on the first partition key.
10 . A system, comprising:
one or more processors; and
memory that stores computer-executable instructions that are executable by the one or more processors to cause the system to:
partition data, associated with computer-implemented storage, with a partition key to provide data partitioned based on the partition key;
analyze data writes to the data partitioned based on the partition key to determine a predicate common to least a plurality of the data writes;
determine a percentage based on a number of times the predicate was found to be associated with the data writes over a predetermined duration;
determine the percentage satisfies a percentage threshold;
based on determining the percentage satisfies the percentage threshold, partition the data based on the predicate; and
write data to the data partitioned based on the predicate.
11 . The system of claim 10 , wherein partitioning the data based on the predicate provides data partitioned based on another partition key associated with the predicate, the other partition key derived from an identifier for columnar data comprised in the data.
12 . The system of claim 10 , wherein the memory that stores the computer-executable instructions that are executable by the one or more processors are further to cause the system to:
analyze metadata based on the predicate to select another partition key, the metadata associated with the data,
wherein partitioning the data based on the predicate comprises partitioning the data using the selected another partition key.
13 . The system of claim 12 , wherein the metadata comprises the partition key and a plurality of identifiers associated with structured data comprised in the data, and wherein the other partition key is determined from one of the plurality of identifiers.
14 . The system of claim 10 , wherein the memory that stores the computer-executable instructions that are executable by the one or more processors are further to cause the system to:
process first real-time data writes to the data partitioned based on the predicate for a duration corresponding to a predetermined duration.
15 . The system of claim 14 , wherein the memory that stores the computer-executable instructions that are executable by the one or more processors are further to cause the system to:
subsequent to termination of the duration, process second real-time data writes to the data partitioned based on the partition key.
16 . The system of claim 10 , wherein the partition key is a randomly selected partition key corresponding to at least one identifier comprised in a logical schema for the data.