Synchronized data deduplication
A system and method for data deduplication is presented. Data received from one or more computing systems is deduplicated, and the results of the deduplication process stored in a reference table. A representative subset of the reference table is shared among a plurality of systems that utilize the data deduplication repository. This representative subset of the reference table can be used by the computing systems to deduplicate data locally before it is sent to the repository for storage. Likewise, it can be used to allow deduplicated data to be returned from the repository to the computing systems. In some cases, the representative subset can be a proper subset wherein a portion of the referenced table is identified shared among the computing systems to reduce bandwidth requirements for reference-table synchronization.
1. A computer-implemented data deduplication method, the method comprising:
with one or more computing systems of a shared storage system that maintains a deduplicated data store accessible by a plurality of client computing systems:
maintaining a first reference table including a plurality of data segment references corresponding to a plurality of data segments stored in the deduplicated data store;
based on one or more of data segment size information and data segment utilization frequency information, determining a first subset of the data segment references in the first reference table to transmit for inclusion in a second reference table accessible by a first client computing system of the plurality of client computing systems, the first subset including a reference to the first data segment;
transmitting the first subset for inclusion in the second reference table accessible by the first client computing system of the plurality of client computing systems;
based on one or more of data segment size information and data segment utilization frequency information, determining a second subset of the data segment references in the first reference table to transmit for inclusion in a third reference table accessible by a second client computing system of the plurality of client computing systems, the second subset different than the first subset; and
transmitting the second subset for inclusion in the third reference table accessible by the second client computing system of the plurality of client computing systems.
2. The method of claim 1 wherein the data segment references are hash signatures calculated based on the corresponding data segments.
3. The method of claim 1 , wherein the second and third reference tables are different partial versions of the first reference table.
4. The method of claim 1 wherein the second reference table is local to the first client computing system and the third reference table is local to the second client computing system, and the first reference table is remote from the first and second client computing systems.
5. The method of claim 1 , wherein said determining the first subset and said determining the second subset are based on data segment utilization frequency information and not data segment size information.
6. The method of claim 1 , wherein said determining the first subset and said determining the second subset are based on data segment size information and not data segment utilization frequency information.
7. The method of claim 1 , wherein said determining the first subset is based on a weighted combination of data segment utilization frequency information and data segment size information.
8. The method of claim 1 wherein said determining the first subset is in response to receiving the first data segment at the shared storage system from the first client computing system.
9. The method of claim 1 further comprising, subsequent to said the second subset for inclusion in the third reference table, receiving a signature corresponding to the first data segment from the first client computing system, without receiving the first data segment itself.
10. A non-transitory computer readable-medium comprising computer program code that, when executed by a computing device of a shared storage system that maintains a deduplicated data store accessible by a plurality of client computing systems, causes the computing device to perform operations comprising:
maintaining a first reference table including a plurality of data segment references corresponding to a plurality of data segments stored in the deduplicated data store;
based on one or more of data segment size information and data segment utilization frequency information, determining a first subset of the data segment references in the first reference table to transmit for inclusion in a second reference table accessible by a first client computing system of the plurality of client computing systems, the first subset including a reference to the first data segment;
transmitting the first subset for inclusion in the second reference table accessible by the first client computing system of the plurality of client computing systems;
based on one or more of data segment size information and data segment utilization frequency information, determining a second subset of the data segment references in the first reference table to transmit for inclusion in a third reference table accessible by a second client computing system of the plurality of client computing systems, the second subset different than the first subset; and
transmitting the second subset for inclusion in the third reference table accessible by the second client computing system of the plurality of client computing systems.
11. The non-transitory computer readable-medium of claim 10 , wherein the data segment references are hash signatures calculated based on the corresponding data segments.
12. The non-transitory computer readable-medium of claim 10 , wherein the second and third reference tables are different partial versions of the first reference table.
13. The non-transitory computer readable-medium of claim 10 , wherein the second reference table is local to the first client computing system and the third reference table is local to the second client computing system, and the first reference table is remote from the first and second client computing systems.
14. The non-transitory computer readable-medium of claim 10 , where said determining the first subset is based on a weighted combination of data segment utilization frequency information and data segment size information.
15. A system comprising:
one or more memory devices containing a deduplicated data store accessible by a plurality of client computing systems; and
one or more physical processors configured to execute instructions to cause a computing system to:
maintain a first reference table including a plurality of data segment references corresponding to a plurality of data segments stored in the deduplicated data store;
based on one or more of data segment size information and data segment utilization frequency information, determine a first subset of the data segment references in the first reference table to transmit for inclusion in a second reference table accessible by a first client computing system of the plurality of client computing systems, the first subset including a reference to the first data segment;
transmit the first subset for inclusion in the second reference table accessible by the first client computing system of the plurality of client computing systems;
based on one or more of data segment size information and data segment utilization frequency information, determine a second subset of the data segment references in the first reference table to transmit for inclusion in a third reference table accessible by a second client computing system of the plurality of client computing systems, the second subset different than the first subset; and
transmit the second subset for inclusion in the third reference table accessible by the second client computing system of the plurality of client computing systems.
16. The system of claim 15 , wherein the data segment references are hash signatures calculated based on the corresponding data segments.
17. The system of claim 15 , wherein the second and third reference tables are different partial versions of the first reference table.
18. The system of claim 15 , wherein the second reference table is local to the first client computing system and the third reference table is local to the second client computing system, and the first reference table is remote from the first and second client computing system.
19. The system of claim 15 , wherein the determination of the first subset and the determination of the second subset are based on data segment utilization frequency information and not data segment size information.
20. The system of claim 15 , wherein the determination of the first subset and the determination of the second subset are based on data segment size information and not data segment utilization frequency information.