Multi-deduplication
View Patent ↗Example apparatus and methods improve deduplication efficiency for a deduplication application or process. A first blocklet repository may have been created according to a first deduplication approach that was optimized for a first set of conditions. Example apparatus and methods create a second blocklet repository from the first blocklet repository by deduplicating the first blocklet repository using a second deduplication approach that is optimized for a second set of conditions. While the first blocklet repository may have been appropriate for the first set of conditions, the second blocklet repository may be appropriate for the second set of conditions. For example, conditions that exist for an immature repository or during ingest may be different than conditions that exist for a mature repository or for applications that use a repository rather than build a repository. The first and second repositories may reside on separate deduplication apparatus.
1. A non-transitory computer-readable device storing computer-executable instructions that when executed by a computer cause the computer to perform a method, the method comprising:
accessing a first deduplication blocklet repository that was created by deduplicating a first data stream as controlled by a first set of deduplication parameters;
accessing a third deduplication blocklet repository that was created by deduplicating a second, different data stream as controlled by a third set of deduplication parameters; and
creating a second, different deduplication blocklet repository by deduplicating the first blocklet repository and the third blocklet repository, where deduplicating the first blocklet repository and the third blocklet repository are controlled by a second, different set of deduplication parameters.
2. The non-transitory computer-readable device of claim 1 , where the first set of deduplication parameters include a maximum blocklet size, a minimum blocklet size, a desired blocklet size distribution, a desired ingest speed, a boundary placing condition, a hash function, a duplicate determination condition, an amount of space reduction to achieve, or an item recreation time to achieve,
where the second set of deduplication parameters include a maximum blocklet size, a minimum blocklet size, a desired blocklet size distribution, a desired ingest speed, a boundary placing condition, a hash function, a duplicate determination condition, an amount of space reduction to achieve, or an item recreation time to achieve,
where the third set of deduplication parameters include a maximum blocklet size, a minimum blocklet size, a desired blocklet size distribution, a desired ingest speed, a boundary placing condition, a hash function, a duplicate determination condition, an amount of space reduction to achieve, or an item recreation time to achieve, and
where the second set of deduplication parameters differs from the first set of deduplication parameters in at least one way, or where the second set of deduplication parameters differs from the third set of deduplication parameters in at least one way.
3. The non-transitory computer-readable device of claim 1 , the method comprising: accessing an item recreation recipe associated with the first blocklet repository; and selectively updating the item recreation recipe to replace a reference to one or more blocklets in the first blocklet repository with a reference to one or more blocklets in the second blocklet repository.
4. The non-transitory computer-readable device of claim 3 , the method comprising: selectively deleting a chosen blocklet from the first blocklet repository upon determining that data from the chosen blocklet is present in one or more replacement blocklets in the second blocklet repository and that all references to the chosen blocklet have been replaced by references to the one or more replacement blocklets.
5. The non-transitory computer-readable device of claim 1 , where the first blocklet repository has a first index and a first set of recipes for recreating items from the first blocklet repository, and where creating the second deduplication blocklet repository includes creating a second index and a second set of recipes for recreating items from the second deduplication blocklet repository.
6. The non-transitory computer-readable device of claim 5 , the method comprising controlling a recreation logic to recreate an item using the second deduplication blocklet repository, the second index, or the second set of recipes without using the first deduplication blocklet repository, without using the first index, and without using the first set of recipes.
7. The non-transitory computer-readable device of claim 5 , the method comprising controlling a recreation logic to recreate a file using both the first deduplication blocklet repository and the second deduplication blocklet repository.
8. The non-transitory computer-readable device of claim 1 , the method comprising:
selectively deleting a chosen blocklet from the third deduplication blocklet repository upon determining that data from the chosen blocklet is present in one or more replacement blocklets in the second deduplication blocklet repository and that all references to the chosen blocklet have been replaced by references to the one or more replacement blocklets, and
controlling a recreation logic to recreate an item using the second deduplication blocklet repository without using the first deduplication blocklet repository and without using the third deduplication blocklet repository.
9. The non-transitory computer-readable medium device of claim 1 , where the first deduplication blocklet repository and the second deduplication blocklet repository reside on separate deduplication apparatus.
10. An apparatus, comprising:
a processor;
a memory configured to store a blocklet produced by a deduplication apparatus or process;
a set of logics; and
an interface that connects the processor, the memory, and the set of logics; the set of logics comprising:
a first logic configured to access data in a first existing blocklet data store and a second, different blocklet data store, where the first existing blocklet data store was populated by a first deduplication apparatus or a first process using a first set of deduplication constraints, and where the second, different blocklet data store was populated by a second deduplication apparatus or a second process using a second set of deduplication constraints;
a second logic configured to create blocklets for a new blocklet data store by deduplicating, using a third, different set of deduplication constraints, the first existing blocklet data store and the second existing blocklet data store, and
a third logic configured to selectively reconfigure a data structure associated with the first existing blocklet data store and the second existing blocklet data store.
11. The apparatus of claim 10 , where the deduplication application or process was controlled by one or more first deduplication constraints or one or more second deduplication constraints, and where second logic is controlled by one or more third deduplication constraints that differ in at least one way from the one or more first deduplication constraints or the one or more second deduplication constraints.
12. The apparatus of claim 11 , where the one or more third deduplication constraints cause the average blocklet size in the new blocklet data store to be smaller than the average blocklet size in the first existing blocklet data store or the second existing blocklet data store.
13. The apparatus of claim 11 , where the one or more third deduplication constraints cause the new blocklet data store to have fewer blocklets than the combined total number of blocklets in the first existing blocklet data store and the second existing blocklet data store while providing one hundred percent coverage of data stored in the first existing blocklet data store and the second existing blocklet data store.
14. The apparatus of claim 10 , where the first existing blocklet data store and the second existing blocklet data store and the new blocklet data store reside on separate deduplication apparatus.
15. The apparatus of claim 11 , the third logic being configured to delete one or more chosen blocklets from the first existing blocklet data store or the second existing blocklet data store upon determining that data covering the one or more chosen blocklets has been placed in the new blocklet data store.
16. The apparatus of claim 11 , the third logic being configured to delete all blocklets from the first existing blocklet data store or the second existing blocklet data store upon determining that data covering all the blocklets has been placed in the new blocklet data store.
17. The apparatus of claim 11 , the third logic being configured to reconfigure an index associated with an existing blocklet data store and to selectively reconfigure a recipe associated with recreating a file from an existing blocklet data store.
18. The apparatus of claim 11 , the third logic being configured to selectively control an apparatus or process that was interacting with the first existing blocklet data store or the second existing blocklet data store to interact instead with the new blocklet data store and to not interact with first existing blocklet data store and the second existing blocklet data store.