Collaborative, distributed, data de-duplication
View Patent ↗Example apparatus, methods, and computers participate in collaborative, distributed, data de-duplication. One example method includes initializing a layered parser in a first node in a collaborative distributed data de-duplication (CDDD) topology with a first set of de-duplication control parameters. After transmitting some information to another node in the CDDD topology, the method includes selectively reconfiguring the layered parser in response to feedback acquired from the second node in the CDDD topology. The feedback concerns the data provided by the layered parser.
1. A method, comprising:
upon determining that a first node configured with a parameterized reactive agent is going to participate in a collaborative, distributed data de-duplication operation in a collaborative, distributed, data de-duplication (CDDD) topology,
controlling the first node, in accordance with a set of parameters associated with the parameterized reactive agent, to create a de-duplication negotiation packet (DNPkt) associated with processing raw data available to the first node;
controlling the first node to transmit the DNPkt to a second node in the CDDD topology;
receiving, from the second node, a reactive control packet (RCPkt) responsive to the DNPkt; and
selectively reconfiguring the parameterized reactive agent by modifying the set of parameters in response to information in the RCPkt.
2. The method of claim 1 , where the parameterized reactive agent is configured to control the first node to perform, in accordance with the set of parameters:
selectively producing a chunk of raw data from the raw data available at the first node,
selectively producing a hash of the chunk of raw data,
selectively indexing the chunk based on the hash,
selectively transmitting the chunk to the second node, and selectively transmitting the hash to the second node.
3. The method of claim 2 , comprising:
controlling the parameterized reactive agent to control the first node to perform, in accordance with the set of parameters:
selectively producing the chunk of raw data,
selectively producing the hash of the chunk of raw data,
selectively indexing the chunk based on the hash,
selectively transmitting the chunk to the second node, and
selectively transmitting the hash to the second node.
4. The method of claim 1 , comprising:
receiving the parameterized reactive agent at the first node.
5. The method of claim 4 , comprising:
establishing an initial set of parameters for a newly received parameterized reactive agent in the first node.
6. The method of claim 1 , where the DNPkt comprises:
a portion of the raw data;
a hash of the portion of the raw data; and
context data associated with the raw data.
7. The method of claim 6 , where the DNPkt comprises:
two or more different portions of the raw data, where the two or more different portions have different lengths; and
hashes associated with the two or more different portions.
8. The method of claim 6 , where the context data identifies one or more of, a file type associated with the raw data, whether the raw data has been previously de-duplicated, a source associated with the raw data, a destination associated with the raw data, whether the data is associated with a replication event, whether the raw data is associated with a rule for parsing data, and whether the raw data is associated with configuring a layered parser.
9. The method of claim 8 , where the set of parameters are determined based, at least in part, on the context data.
10. The method of claim 8 , where the composition of the DNPkt is determined, at least in part, by the context data.
11. The method of claim 1 , where the RCPkt includes one or more of:
information identifying a desired de-duplication node for the raw data;
information identifying a desired de-duplication time for the raw data;
information identifying a desired rule for de-duplicating the raw data;
information identifying a desired chunk size for chunking the raw data;
a statistic from processing data in the DNPkt; and
a request from the second node for a subsequent DNPkt.
12. The method of claim 1 , where determining that the first node is going to participate in the collaborative, distributed, data de-duplication operation is performed on-the-fly, just-in-time, during collaborative de-duplication actions involving two or more nodes in the CDDD topology.
13. An apparatus, comprising:
a data store to store:
raw data to be de-duplicated as part of a collaborative, distributed process involving two or more apparatus;
processed data de-duplicated as part of the collaborative, distributed process;
indexing data for identifying and retrieving the processed data; and
configuration information for controlling how the apparatus performs its portion of the collaborative, distributed process;
a processor;
a memory; and
an interface to connect the processor, the memory, and a set of logics, the set of logics comprising:
a chunk logic configured to participate in the collaborative, distributed process by creating a chunk of data from the raw data, where the chunk of data is a subset of the raw data;
a hash logic configured to participate in the collaborative, distributed process by computing a hash for the chunk of data;
an index logic configured to participate in the collaborative, distributed process by selectively storing the chunk of data in the data store and by manipulating the indexing data based, at least in part, on the hash;
a chunk transmission logic configured to transmit the chunk to a second apparatus involved in the collaborative, distributed process;
a hash transmission logic configured to transmit the hash to the second apparatus; and
a control logic configured to control the chunk logic, the hash logic, the index logic, the chunk transmission logic, and the hash transmission logic based on the configuration information and on feedback data from the second apparatus.
14. The apparatus of claim 13 , where the chunk logic selectively produces chunks of different sizes using different chunking criteria based on control information provided by the control logic as determined by the configuration information and the feedback data.
15. The apparatus of claim 13 , where the hash logic selectively produces different types of hashes based on control information provided by the control logic as determined by the configuration information and the feedback data.
16. The apparatus of claim 13 , where the index logic is configured to control the second apparatus to store one or more of, the chunk, and the hash on a data store associated with the second apparatus.
17. The apparatus of claim 13 , where the chunk transmission logic and the hash transmission logic are configured to produce a de-duplication negotiation packet that identifies how the control logic is configured based on the results produced by the chunk logic and the hash logic and to provide the de-duplication negotiation packet to the second apparatus as a solicitation for the feedback data.
18. The apparatus of claim 13 , where the de-duplication negotiation packet comprises:
one or more chunks produced by the chunk logic;
one or more related hashes produced by the hash logic; and
context data associated with the raw data to be de-duplicated,
where the context data identifies one or more of, a file type associated with the raw data, whether the raw data has been previously de-duplicated, a source associated with the raw data, a destination associated with the raw data, whether the data is associated with a replication event, whether the raw data is associated with a rule for parsing data, and whether the raw data is associated with configuring a layered parser.
19. The apparatus of claim 13 , where the feedback data comprises:
information identifying additional nodes to participate in the collaborative, distributed process;
information identifying when the raw data is to be de-duplicated; and
information identifying how the raw data is to be de-duplicated.