Adaptive collaborative de-duplication
View Patent ↗Example apparatus, methods, and computers support adaptive, collaborative deduplication. One example method includes controlling a computer to compute a correlation between a processing state associated with collaborative data deduplication performed by the computer and an environmental state associated with a collaborative data deduplication environment with which the computer interacts. With the correlation in hand, the method then includes controlling the computer to adapt how the computer participates in collaborative data deduplication based, at least in part, on the correlation. The adaptations can include changing what processing is performed on the computer and what data is stored on the computer.
1. A data deduplication client apparatus, comprising:
a processor;
a memory; and
an interface to connect the processor, the memory, and a set of logics, the set of logics comprising:
a conditions logic configured to control the data deduplication client apparatus to identify a current condition in a collaborative data deduplication environment, where the current condition concerns a current condition in a first deduplication server with which the client apparatus is performing collaborative deduplication, a current condition in a second deduplication server with which the client apparatus is considering performing collaborative deduplication, a current condition in a first communication path between the client apparatus and the first server, a current condition in a second communication path between the client apparatus and the second server, and randomness of data being received at the client apparatus;
a performance logic configured to control the client apparatus to identify a data deduplication performance criterion for the client apparatus; and
an adaptation logic configured to selectively reconfigure one or more of, collaborative data deduplication processing performed by the client apparatus, and deduplication related storage performed by the client apparatus, where the selective reconfiguring occurs upon determining that the current condition and the performance criterion do not match a desired deduplication performance profile for the client apparatus.
2. The apparatus of claim 1 , where the current condition in the collaborative data deduplication environment concerns
a current condition in the client apparatus, and
a current condition in a third communication path between the client apparatus and the first server.
3. The apparatus of claim 2 , where the current condition in the client apparatus concerns processing cycles available on the client apparatus, memory available on the client apparatus, disk space available to the client apparatus, randomness of the workload at the client apparatus, and entropy of data to be collaboratively deduplicated by the client apparatus and a collaborating server.
4. The apparatus of claim 2 , where the current condition in the first deduplication server concerns a first deduplication approach available on the first server, the randomness of data being ingested at a client computer, the randomness of a workload at the client, and a first deduplication reduction factor achieved by the first server during collaborative deduplication performed with the client apparatus, where the first deduplication reduction factor describes the reduction of stored data, and the reduction of transmitted data.
5. The apparatus of claim 2 , where the current condition in the second deduplication server concerns a second deduplication approach available on the second server, and a second deduplication reduction factor achieved by the second server during collaborative deduplication performed with a different client apparatus, and where the second deduplication reduction factor describes the reduction of stored data, and the reduction of transmitted data.
6. The apparatus of claim 2 , where the current condition in the first communication path concerns cost per byte to transmit data over the first communication path, cost per packet to transmit data over the first communication path, latency on the first communication path, error rate on the first communication path, and security on the first communication path,
where the current condition in the second communication path concerns cost per byte to transmit data over the second communication path, cost per packet to transmit data over the second communication path, latency on the second communication path, error rate on the second communication path, and security on the second communication path, and
where the current condition in the third communication path concerns cost per byte to transmit data over the third communication path, cost per packet to transmit data over the third communication path, latency on the third communication path, error rate on the third communication path, and security on the third communication path.
7. The apparatus of claim 1 , where the data deduplication performance criterion concerns one or more of, cost to deduplicate data, time to deduplicate data, security of deduplicated data, resources consumed at the client apparatus, and a deduplication reduction factor.
8. The apparatus of claim 1 , where the desired deduplication performance profile is a function of cost compared to deduplication reduction achieved, time compared to deduplication reduction achieved, and resources compared to deduplication reduction achieved.
9. The apparatus of claim 8 , where there are separate data deduplication performance criteria for different workloads associated with archive based deduplication, with nearline based deduplication, and with online based deduplication.
10. The apparatus of claim 1 , where the collaborative data deduplication processing performed by the client apparatus includes sampling, making a duplicate determination, making a similarity determination, and preparing server hints.
11. The apparatus of claim 10 , where reconfiguring the client apparatus by adapting the collaborative deduplication processing performed by the client apparatus comprises changing what processing is performed on the client apparatus, to which server raw data is pushed, from which server data is pulled, the granularity of server hints, and what server hints are provided to a server.
12. The apparatus of claim 1 , where deduplication related storage performed by the client apparatus comprises storing, in one or more permuted data structures on the client apparatus, a sample for a chunk of data, information concerning a deduplication decision for a chunk of data, and a server hint for a chunk of data, where the server hint facilitates a server performing deduplication processing for the chunk of data.
13. The apparatus of claim 12 , where reconfiguring the client apparatus by adapting deduplication related storage performed by the client apparatus comprises changing one or more of, the size of a permuted data structure on the client apparatus, the ordering of a permuted data structure on the client apparatus, the type of data stored on the client apparatus, how frequently data stored on the client apparatus is accessed, what event triggers accessing data stored on the client apparatus, what event triggers manipulating data stored on the client apparatus, and the granularity of items for which data is stored.
14. The apparatus of claim 1 , comprising:
a deduplication logic configured to perform the collaborative deduplication processing performed by the client apparatus.
15. A computer-implemented method, comprising:
controlling a computer to compute a correlation between a processing state associated with collaborative data deduplication performed by the computer and an environmental state associated with a collaborative data deduplication environment with which the computer interacts; and
controlling the computer to adapt how the computer participates in collaborative data deduplication based, at least in part, on the correlation, where adapting how the computer participates in collaborative data deduplication comprises adapting how boundary identification is performed, how boundary placement is performed, how chunking is performed, how hashing is performed, how sampling is performed, how a duplicate determination is made, how a similarity determination is made, how server hints are prepared, what deduplication processing is performed on the client, to which server in the collaborative data deduplication environment raw data is pushed, from which server in the collaborative data deduplication environment data is pulled, and what server hints are provided to a server in the collaborative data deduplication environment.
16. The computer-implemented method of claim 15 , where adapting how the computer participates in collaborative data deduplication comprises adapting how much raw data is stored on the computer, how much chunked data is stored on the computer, whether a hash for a chunk of data is stored on the computer, whether a sample for a chunk of data is stored on the computer, how much information concerning a deduplication decision for a chunk of data is stored on the computer, how many server hints for a chunk of data are stored on the computer, the size of a permuted deduplication data structure on the client, the ordering of a permuted deduplication data structure on the client, the type of data stored on the client, the frequency of how often data stored on the client is accessed, what event triggers accessing data stored on the client, what even triggers manipulating data stored on the client, and the granularity of items for which data is stored.
17. The computer-implemented method of claim 16 , where the collaborative data deduplication is one or more of, cloud based, and client/server based.