Decentralized record expiry
A technique is described that reduces the complexity and resource consumption associated with performing record expiry in a distributed database system. In accordance with the technique, a record is checked to see if it has expired only when it has been accessed for a read or a write. If at the time of a read a record is determined to have expired, then it is not served. If at the time of a write a record is determined to have expired, then the write is treated as an insertion of a new record, and steps are taken to treat the insertion consistently with regard to the previous expired version. A background process is used to delete records that have not been written to or actively deleted by a client after expiration.
1. A method for automatically deleting an expired record in a distributed database system comprising a plurality of nodes, wherein each node comprises one or more processor-based systems and is configured to manage a respective one of a plurality of databases, the method comprising:
receiving a write request for a record at the one of the plurality of nodes, wherein the record is one of a plurality of records within a partition stored in the database managed by the one of the plurality of nodes and wherein the partition comprises a distinct portion of a logical database;
determining whether the record is expired responsive to receiving the write request;
determining if an incarnation number associated with the partition exceeds an incarnation number stored in the record responsive to determining that the record is expired;
responsive to determining that the incarnation number associated with the partition exceeds the incarnation number stored in the record,
generating an updated record based on the write request, wherein generating the updated record comprises replacing the incarnation number stored in the record with the incarnation number associated with the partition and
overwriting the expired record with the updated record; and
responsive to determining that the incarnation number associated with the partition does not exceed the incarnation number stored in the record,
deleting the expired record, wherein deleting the expired record comprises sending a message to a remote node to increment an incarnation number associated with a partition stored in the database managed by the remote node,
generating an updated record based on the write request, and
inserting the updated record.
2. The method of claim 1 , wherein generating an updated record based on the write request further comprises setting a sequence number stored in the updated record to an initial value.
3. The method of claim 1 , further comprising:
sending a message to the other nodes regarding the overwriting of the expired record with the updated record.
4. The method of claim 3 , wherein generating the updated record comprises:
obtaining an incarnation number from a remote node, wherein the incarnation number obtained from the remote node is associated with a partition stored in the database managed by the remote node, and
replacing the incarnation number stored in the record with the incarnation number obtained from the remote node.
5. A distributed database system, comprising:
a plurality of nodes, wherein the plurality of nodes are interconnected via a communication system and wherein each node comprises one or more processors and memory; and
a plurality of databases, wherein each node in the plurality of nodes is configured to manage a respective one of a plurality of databases;
wherein each node in the plurality of nodes is further configured to receive a write request for a record, wherein the record is one of a plurality of records within a partition stored in the database managed by the node and wherein the partition comprises a distinct portion of a logical database, to determine whether the record is expired responsive to receiving the write request, to determine if an incarnation number associated with the partition exceeds an incarnation number stored in the record responsive to determining that the record is expired, and, responsive to determining that the incarnation number associated with the partition exceeds the incarnation number stored in the record, to generate an updated record based on the write request, wherein generating the updated record comprises replacing the incarnation number stored in the record with the incarnation number associated with the partition, and to overwrite the expired record with the updated record, and
wherein each node in the plurality of nodes is further configured to, responsive to determining that the incarnation number associated with the partition does not exceed the incarnation number stored in the record, delete the expired record, wherein deleting the expired record comprises sending a message to a remote node to increment an incarnation number associated with a partition stored in the database managed by the remote node, to generate an updated record based on the write request, and to insert the updated record.
6. The system of claim 5 , wherein each node in the plurality of nodes is configured to generate an updated record based on the write request by setting a sequence number stored in the updated record to an initial value.
7. The system of claim 5 , wherein each node in the plurality of nodes is further configured to send a message to the other nodes via the communication system regarding the overwriting of the expired record with the updated record.
8. The system of claim 5 , wherein each node in the plurality of nodes is configured to generate the updated record by obtaining an incarnation number from a remote node, wherein the incarnation number obtained from the remote node is associated with a partition stored in the database managed by the remote node, and replacing the incarnation number stored in the record with the incarnation number obtained from the remote node.
9. A method for automatically deleting an expired record in a distributed database system comprising a plurality of nodes, wherein each node comprises one or more processor-based systems and is configured to manage a respective one of a plurality of databases, the method being performed by one of the plurality of nodes and the method comprising:
reading a record at the one of the plurality of nodes, wherein the record is one of a plurality of records within a partition stored in the database managed by the one of the plurality of nodes and wherein the partition comprises a distinct portion of a logical database;
determining whether the record is expired;
determining if an incarnation number associated with the partition exceeds an incarnation number stored in the record responsive to determining that the record is expired;
purging the record responsive to determining that the incarnation number associated with the partition exceeds the incarnation number stored in the record; and
deleting the expired record responsive to determining that the incarnation number associated with the partition does not exceed the incarnation number stored in the record, wherein deleting the expired record comprises sending a message to a remote node to increment an incarnation number associated with a partition stored in the database managed by the remote node.
10. The method of claim 9 , further comprising:
sending a message to the other nodes regarding the purging of the record.
11. A distributed database system, comprising:
a plurality of nodes, wherein the plurality of nodes are interconnected via a communication system and wherein each node comprises one or more processors and memory; and
a plurality of databases, wherein each node in the plurality of nodes is configured to manage a respective one of a plurality of databases;
wherein each node in the plurality of nodes is further configured to read a record, wherein the record is one of a plurality of records within a partition stored in the database managed by the node and wherein the partition comprises a distinct portion of a logical database, to determine whether the record is expired, to determine if an incarnation number associated with the partition exceeds an incarnation number stored in the record responsive to determining that the record is expired, and to purge the record responsive to determining that the incarnation number associated with the partition exceeds the incarnation number stored in the record, and
wherein each node in the plurality of nodes is further configured to, responsive to determining that the incarnation number associated with the partition does not exceed the incarnation number stored in the record, delete the expired record, wherein deleting the expired record comprises sending a message to a remote node to increment an incarnation number associated with a partition stored in the database managed by the remote node.
12. The system of claim 11 , wherein each node in the plurality of nodes is further configured to send a message to the other nodes regarding the purging of the record.