Maintaining retention policies in a block storage, multi-client dedup domain
One example method includes maintaining a deduplication data structure including one or more entries that each identify a respective fingerprint, and pointer, and also maintaining a ClientBlockList data structure comprising one or more entries that each identify a respective handle, retention date, and block, receiving a write request that identifies a handle, retention date, and block, computing a fingerprint of the block identified in the write request, determining, by the server, whether the fingerprint is in the deduplication data structure, and when the fingerprint is not in the deduplication data structure, storing the block identified in the write request at location ‘L’ in the deduplication data structure, and adding, to the deduplication data structure, an entry that identifies the fingerprint and the location ‘L,’ and adding, to the ClientBlockList data structure, an entry that identifies the handle, retention date, and fingerprint.
1. A method, comprising:
maintaining, at a server, a deduplication data structure comprising one or more entries, and each of the one or more entries comprises a respective fingerprint, and a respective pointer that points to a physical location on a disk where data associated with that fingerprint is stored;
maintaining, at the server, a ClientBlockList data structure comprising one or more entries, and each of the entries comprises a respective handle, a respective retention date, and a respective block;
receiving, at the server, a write request for writing a block of data, wherein the write request identifies a handle, retention date, and the block;
computing, at the server, a fingerprint of the block identified in the write request;
determining, by the server, whether the fingerprint of the block is in the deduplication data structure, and when the fingerprint of the block is not in the deduplication data structure, storing the block identified in the write request at a location in the deduplication data structure, and adding, to the deduplication data structure, an entry that includes the fingerprint of the block and a pointer that points to the location; and
adding, to the ClientBlockList data structure, an entry that identifies the handle, the retention date, and the fingerprint of the block,
wherein, when the handle is already included in an entry of the ClientBlockList data structure, the entry of the ClientBlockList data structure is overwritten by the entry that identifies the handle, the retention date, and the fingerprint of the block,
wherein the deduplication data structure spans multiple client domains, and
wherein a deduplication process performed with respect to the block identified in the write request is performed without reference to the retention date of that block.
2. The method as recited in claim 1 , wherein the ClientBlockList data structure is specific to a particular client, and a respective ClientBlockList data structure is maintained at the server for each of a plurality of clients.
3. The method as recited in claim 1 , further comprising scanning the ClientBlockList data structure and removing any entries whose retention date has passed.
4. The method as recited in claim 3 , further comprising, for a particular fingerprint ‘F,’ decrementing a reference count in the deduplication data structure when an entry that includes fingerprint ‘F’ is removed from the ClientBlockList data structure.
5. The method as recited in claim 1 , wherein the method further comprises providing a block storage API that enables a client to specify a retention date on each block identified in a write request.
6. A non-transitory computer readable storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
maintaining, at a server, a deduplication data structure comprising one or more entries, and each of the one or more entries comprises a respective fingerprint, and a respective pointer that points to a physical location on a disk where data associated with that fingerprint is stored;
maintaining, at the server, a ClientBlockList data structure comprising one or more entries, and each of the entries comprises a respective handle, a respective retention date, and a respective block;
receiving, at the server, a write request for writing a block of data, wherein the write request identifies a handle, retention date, and the block;
computing, at the server, a fingerprint of the block identified in the write request;
determining, by the server, whether the fingerprint of the block is in the deduplication data structure, and when the fingerprint of the block is not in the deduplication data structure, storing the block identified in the write request at a location in the deduplication data structure, and adding, to the deduplication data structure, an entry that includes the fingerprint of the block and a pointer that points to the location; and
adding, to the ClientBlockList data structure, an entry that identifies the handle, the retention date, and the fingerprint of the block,
wherein, when the handle is already included in an entry of the ClientBlockList data structure, the entry of the ClientBlockList data structure is overwritten by the entry that identifies the handle, the retention date, and the fingerprint of the block,
wherein the deduplication data structure spans multiple client domains, and
wherein a deduplication process performed with respect to the block identified in the write request is performed without reference to the retention date of that block.
7. The non-transitory computer readable storage medium as recited in claim 6 , wherein the ClientBlockList data structure is specific to a particular client, and a respective ClientBlockList data structure is maintained at the server for each of a plurality of clients.
8. The non-transitory computer readable storage medium as recited in claim 6 , wherein the operations further comprise scanning the ClientBlockList data structure and removing any entries whose retention date has passed.
9. The non-transitory computer readable storage medium as recited in claim 8 , wherein the operations further comprise, for a particular fingerprint ‘F,’ decrementing a reference count in the deduplication data structure when an entry that includes fingerprint ‘F’ is removed from the ClientBlockList data structure.
10. The non-transitory computer readable storage medium as recited in claim 6 , wherein the operations further comprise providing a block storage API that enables a client to specify a retention date on each block identified in a write request.
11. A system, comprising:
one or more hardware processors; and
a non-transitory computer readable storage medium having stored therein instructions that are executable by the one or more hardware processors to perform operations comprising:
maintaining, at a server, a deduplication data structure comprising one or more entries, and each of the one or more entries comprises a respective fingerprint, and a respective pointer that points to a physical location on a disk where data associated with that fingerprint is stored;
maintaining, at the server, a ClientBlockList data structure comprising one or more entries, and each of the entries comprises a respective handle, a respective retention date, and a respective block;
receiving, at the server, a write request for writing a block of data, wherein the write request identifies a handle, retention date, and the block;
computing, at the server, a fingerprint of the block identified in the write request;
determining, by the server, whether the fingerprint of the block is in the deduplication data structure, and when the fingerprint of the block is not in the deduplication data structure, storing the block identified in the write request at a location in the deduplication data structure, and adding, to the deduplication data structure, an entry that includes the fingerprint of the block and a pointer that points to the location; and
adding, to the ClientBlockList data structure, an entry that identifies the handle, the retention date, and the fingerprint,
wherein, when the handle is already included in an entry of the ClientBlockList data structure, the entry of the ClientBlockList data structure is overwritten by the entry that identifies the handle, the retention date, and the fingerprint of the block,
wherein the deduplication data structure spans multiple client domains, and
wherein a deduplication process performed with respect to the block identified in the write request is performed without reference to the retention date of that block.
12. The system as recited in claim 11 , wherein the ClientBlockList data structure is specific to a particular client, and a respective ClientBlockList data structure is maintained at the server for each of a plurality of clients.
13. The system as recited in claim 11 , wherein the operations further comprise scanning the ClientBlockList data structure and removing any entries whose retention date has passed.
14. The system as recited in claim 13 , wherein the operations further comprise, for a particular fingerprint ‘F,’ decrementing a reference count in the deduplication data structure when an entry that includes fingerprint ‘F’ is removed from the ClientBlockList data structure.
15. The system as recited in claim 11 , wherein the operations further comprise providing a block storage API that enables a client to specify a retention date on each block identified in a write request.