Partitioned backing store implemented in a distributed database
Techniques for providing batch performance using a stream processor are described. In one technique, a method includes receiving an event that includes an event key and a timestamp. The method then generates a partition value and time bucket value, the partition value generated based on the event key and the time bucket value generated based on the timestamp. The event key, partition value, and time bucket value are then combined to form a database key and used in a command to a database.
1. A method comprising:
receiving, by a processor, an event, the event including an event key comprising a unique identifier and a timestamp;
generating, by the processor, a partition value and time bucket value, the partition value generated based on the event key and the time bucket value generated based on the timestamp;
combining, by the processor, the event key, partition value, and time bucket value to form a database key; and
issuing, by the processor, a command to a database using the database key, the command affecting a row corresponding to the unique identifier.
2. The method of claim 1 , the issuing command comprising issuing a fetch command to the database.
3. The method of claim 1 , the generating the partition value comprising mapping the event key to one or more database partitions.
4. The method of claim 1 , the combining the event key, partition value, and time bucket value comprising:
generating a first segment by prepending the time bucket value to the event key; and
generating the database key by prepending the partition value to the first segment.
5. The method of claim 4 , further comprising reversing bits of the partition value and time bucket value prior to generating the first segment.
6. The method of claim 1 , the generating the time bucket value comprising identifying a batch interval based on the timestamp and using the batch interval as the time bucket value.
7. The method of claim 1 , the issuing a command to the database comprising issuing the command to a worker thread associated with a database partition associated with the partition value.
8. A non-transitory computer readable storage medium for tangibly storing computer program instructions capable of being executed by a computer processor, the computer program instructions defining the steps of:
receiving an event, the event including an event key comprising a unique identifier and a timestamp; generating a partition value and time bucket value, the partition value generated based on the event key and the time bucket value generated based on the timestamp;
combining the event key, partition value, and time bucket value to form a database key; and
issuing a command to a database using the database key, the command affecting a row corresponding to the unique identifier.
9. The computer-readable storage medium of claim 8 , the issuing command comprising issuing a fetch command to the database.
10. The computer-readable storage medium of claim 8 , the generating the partition value comprising mapping the event key to one or more database partitions.
11. The computer-readable storage medium of claim 8 , the combining the event key, partition value, and time bucket value comprising:
generating a first segment by prepending the time bucket value to the event key; and
generating the database key by prepending the partition value to the first segment.
12. The computer-readable storage medium of claim 11 , the instructions further defining the step of reversing bits of the partition value and time bucket value prior to generating the first segment.
13. The computer-readable storage medium of claim 8 , the generating the time bucket value comprising identifying a batch interval based on the timestamp and using the batch interval as the time bucket value.
14. The computer-readable storage medium of claim 8 , the issuing a command to the database comprising issuing the command to a worker thread associated with a database partition associated with the partition value.
15. An apparatus comprising:
a processor; and
a storage medium for tangibly storing thereon program logic for execution by the processor, the stored program logic causing the processor to perform the operations of:
receiving an event, the event including an event key comprising a unique identifier and a timestamp;
generating a partition value and time bucket value, the partition value generated based on the event key and the time bucket value generated based on the timestamp;
combining the event key, partition value, and time bucket value to form a database key; and
issuing a command to a database using the database key, the command affecting a row corresponding to the unique identifier.
16. The apparatus of claim 15 , the issuing command comprising issuing a fetch command to the database.
17. The apparatus of claim 15 , the generating the partition value comprising mapping the event key to one or more database partitions.
18. The apparatus of claim 15 , the combining the event key, partition value, and time bucket value comprising:
generating a first segment by prepending the time bucket value to the event key; and
generating the database key by prepending the partition value to the first segment.
19. The apparatus of claim 15 , the generating the time bucket value comprising identifying a batch interval based on the timestamp and using the batch interval as the time bucket value.
20. The apparatus of claim 15 , the issuing a command to the database comprising issuing the command to a worker thread associated with a database partition associated with the partition value.