Similarity based deduplication for secondary storage

1. An apparatus comprising:

one or more processors; and

one or more non-transitory computer readable storage media, the one or more non-transitory computer readable storage media comprising executable code, that when executed by the one or more processors, causes the one or more processors to:

generate a simplified rolling hash value, using a first hashing algorithm, based on a portion of an incoming stream of backup data over a network;

compare the simplified rolling hash value with entries stored in a rolling hash index, and in response to matching the simplified rolling hash value with at least one entry in the rolling hash index, generate a strong hash value, using a second hashing algorithm, and determine if a match of the strong hash value exists in a first strong hash index of one of either a first public cloud or a first private cloud, and in response to a determination that the match does not exist in the first strong hash index, further compare the strong hash value with entries in a second strong hash index in one of either a second public cloud or a second private cloud; and

store, in response to a determination that the strong hash value does not match an entry in the first strong hash index or the second strong hash index, the portion of the incoming stream of backup data as new data in at least one of the first public cloud, first private cloud, second public cloud, or second private cloud.

2. The apparatus of claim 1 , where the executable code causes the one or more processors to import data from one of either the second public cloud or the second private cloud.

3. The apparatus of claim 2 , where the executable code causes the one or more processors to import the second strong hash index into one of either the first public cloud or the first private cloud.

4. The apparatus of claim 3 , where the executable code causes the one or more processors to only import entries that correspond to the portion of the incoming stream of backup data from the second strong hash index.

5. The apparatus of claim 4 , where the executable code causes the one or more processors to only import entries that are cached in one of either the first public cloud or the first private cloud.

6. The apparatus of claim 1 , where the executable code causes the one or more processors to retrieve from one of either the second public cloud or the second private cloud, in response to the determination that the strong hash value matches an entry in the second strong hash index, data corresponding to the entry in the second strong hash index.

7. The apparatus of claim 1 , where the executable code causes the one or more processors to query, in response to a determination that the simplified rolling hash value does not match an entry in the rolling hash index, one of the second public cloud or the second private cloud with the strong hash value to determine if one of the second public cloud or the second private cloud contains a copy of the portion of the incoming stream of backup data.

8. A method comprising:

generating, by use of a processor, a simplified rolling hash value, using a first hashing algorithm, based on a portion of an incoming stream of backup data over a network;

comparing the simplified rolling hash value with entries stored in a rolling hash index, and in response to matching the simplified rolling hash value with at least one entry in the rolling hash index, generating a strong hash value, using a second hashing algorithm, and determining if a match of the strong hash value exists in a first strong hash index of one of either a first public cloud or a first private cloud, and where the comparison module, in response to a determination that the match does not exist in the first strong hash index, further comparing the strong hash value with entries in a second strong hash index in one of either a second public cloud or a second private cloud; and

storing, in response to a determination that the strong hash value does not match an entry in the first strong hash index or the second strong hash index, the portion of the incoming stream of backup data as new data in at least one of the first public cloud, first private cloud, second public cloud, or second private cloud.

9. The method of claim 8 , further comprising importing data from one of either the second public cloud or the second private cloud.

10. The method of claim 8 , further comprising importing the second strong hash index into one of either the first public cloud or the first private cloud.

11. The method of claim 10 , further comprising importing only the entries that correspond to the portion of the incoming stream of backup data from the second strong hash index.

12. The method of claim 11 , further comprising importing only entries that are cached in one of either the first public cloud or the first private cloud.

13. The method of claim 8 , further comprising retrieving from one of either the second public cloud or the second private cloud, in response to the determination that the strong hash value matches an entry in the second strong hash index, data corresponding to the entry in the second strong hash index.

14. The method of claim 8 , further comprising querying, in response to a determination that the simplified rolling hash value does not match an entry in the rolling hash index, one of the second public cloud or the second private cloud with the strong hash value to determine if one of the second public cloud or the second private cloud contains a copy of the portion of the incoming stream of backup data.

15. A computer program product, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions readable/executable by a processor to cause the processor to execute steps comprising:

generating, by processor, a simplified rolling hash value, using a first hashing algorithm, based on a portion of an incoming stream of backup data over a network;

comparing the simplified rolling hash value with entries stored in a rolling hash index, and in response to matching the simplified rolling hash value with at least one entry in the rolling hash index, generating a strong hash value, using a second hashing algorithm, and determining if a match of the strong hash value exists in a first strong hash index of one of either a first public cloud or a first private cloud, and where the comparison module, in response to a determination that the match does not exist in the first strong hash index, further comparing the strong hash value with entries in a second strong hash index in one of either a second public cloud or a second private cloud; and

storing, in response to a determination that the strong hash value does not match an entry in the first strong hash index or the second strong hash index, the portion of the incoming stream of backup data as new data in at least one of the first public cloud, first private cloud, second public cloud, or second private cloud.

16. The computer program product of claim 15 , where the steps further comprise importing data from one of either the second public cloud or the second private cloud.

17. The computer program product of claim 16 , where the steps further comprise importing the second strong hash index into one of either the first public cloud or the first private cloud.

18. The computer program product of claim 17 , further comprising importing only the entries that correspond to the portion of the incoming stream of backup data from the second strong hash index.

19. The computer program product of claim 18 , further comprising importing only entries that are cached in one of either the first public cloud or the first private cloud.

20. The computer program product of claim 15 , further comprising querying, in response to a determination that the simplified rolling hash value does not match an entry in the rolling hash index, one of the second public cloud or the second private cloud with the strong hash value to determine if one of the second public cloud or the second private cloud contains a copy of the portion of the incoming stream of backup data.