Scalable de-duplication mechanism

1. A computerized method for classifying data from a backup application for de-duplication of redundant data based on one or more characteristics of the data, the method comprising:

receiving, by a computing device, a backup data object from a backup application, wherein the backup application created the backup data object, and a data object characteristic of the backup data object is not known to the computing device;

determining the data object characteristic of the backup data object based on metadata associated with the backup data object;

selecting a de-duplication domain from a plurality of de-duplication domains based at least in part on the data object characteristic of the backup data object, wherein each de-duplication domain from the plurality of de-duplication domains is associated with:

a data object characteristic that defines a data object type;

a de-duplication method selected from a plurality of de-duplication methods that is tailored to de-duplicate redundant data among backup data objects of the data object type, wherein backup data objects of the data object type have redundant data because they share the same data object characteristic; and

a data object type that is different than each data object type associated with the remaining de-duplication domains from the plurality of de-duplication domains; and

directing the backup data object to the selected de-duplication domain for de-duplication, wherein the selected de-duplication domain executes the de-duplication method that is associated with the selected de-duplication domain to de-duplicate redundant data among the backup data object, and one or more other backup data objects also directed to the selected de-duplication domain, based on the data object characteristic associated with the selected de-duplication domain.

2. The method according to claim 1 , wherein receiving the backup data object comprises:

receiving a data stream; and

identifying the backup data object using metadata included in the data stream.

3. The method according to claim 2 , wherein receiving the data stream includes receiving a multiplexed data stream.

4. The method according to claim 2 , further comprising extracting metadata included in the data stream with the backup data object.

5. The method according to claim 4 , wherein selecting the de-duplication domain from the plurality of de-duplication domains includes comparing the extracted metadata associated with the backup data object to each set of data object characteristics associated with each de-duplication domain.

6. The method according to claim 4 , wherein extracting the metadata included in the data stream includes extracting at least one of a back-up policy name, a data source type, a data source name, a back-up application name, an operating system type, a data type, a back-up type, a filename, a directory structure and chronological information.

7. The method according to claim 1 , further comprising configuring each of the plurality of de-duplication domains to use one of a plurality of de-duplication methods.

8. The method according to claim 7 , wherein configuring each of the plurality of de-duplication domains includes configuring each of the plurality of de-duplication domains to use one de-duplication method selected from the group comprising hash-fingerprinting, pattern recognition and content aware de-duplication.

9. The method according to claim 1 , further comprising:

de-duplicating, within the selected de-duplication domain, the backup data object; and

adjusting the data object characteristic associated with at least one of the plurality of de-duplication domains based on a result of the act of de-duplicating.

10. The method according to claim 9 , wherein adjusting the data object characteristic includes storing data in a de-duplication domain database.

11. The method according to claim 1 , wherein the method is executed in a grid computing environment.

12. The method according to claim 1 , wherein the method is executed on a back-up storage system while data is not being backed-up to the back-up storage system.

13. The method according to claim 1 , wherein the method is executed on a back-up storage system while data is being backed-up to the back-up storage system.

14. The method of claim 1 , wherein directing the backup data object to the selected de-duplication domain for de-duplication comprises providing a reference to the backup data object to the selected de-duplication domain instead of providing a copy of the data object to the de-duplication domain.

15. A computer program product, tangibly embodied in a non-transitory computer readable medium, the computer program product including instructions being configured to cause a data processing apparatus to:

receive a backup data object from a backup application, wherein the backup application created the backup data object, and a data object characteristic of the backup data object is not known to the computing device;

determine the data object characteristic of the backup data object based on metadata associated with the backup data object;

select a de-duplication domain from a plurality of de-duplication domains based at least in part on the data object characteristic of the backup data object, wherein each de-duplication domain from the plurality of de-duplication domains is associated with

a data object characteristic that defines a data object type;

a de-duplication method selected from a plurality of de-duplication methods that is tailored to de-duplicate redundant data among backup data objects of the data object type, wherein backup data objects of the data object type have redundant data because they share the same data object characteristic; and

a data object type that is different than each data object type associated with the remaining de-duplication domains from the plurality of de-duplication domains; and

direct the backup data object to the selected de-duplication domain for de-duplication, wherein the selected de-duplication domain executes the de-duplication method that is associated with the selected de-duplication domain to de-duplicate redundant data among the backup data object, and one or more other backup data objects also directed to the selected de-duplication domain, based on the data object characteristic associated with the selected de-duplication domain.

16. A system for classifying data from a backup application for de-duplication of redundant data based on one or more characteristics of the data, the system comprising:

a plurality of de-duplication domains, wherein each de-duplication domain from the plurality of de-duplication domains is associated with

a data object characteristic that defines a data object type;

a de-duplication method selected from a plurality of de-duplication methods that is tailored to de-duplicate redundant data among backup data objects of the data object type, wherein backup data objects of the data object type have redundant data because they share the same data object characteristic; and

a data object type that is different than each data object type associated with the remaining de-duplication domains from the plurality of de-duplication domains; and

a controller coupled to the plurality of de-duplication domains and configured to:

receive a backup data object from a backup application, wherein the backup application created the backup data object, and a data object characteristic of the backup data object is not known to the computing device;

determine the data object characteristic of the backup data object based on metadata associated with the backup data object;

select a de-duplication domain from a plurality of de-duplication domains based at least in part on the data object characteristic of the backup data object; and

direct the backup data object to the selected de-duplication domain for de-duplication, wherein the selected de-duplication domain executes the de-duplication method that is associated with the selected de-duplication domain to de-duplicate redundant data among the backup data object, and one or more other backup data objects also directed to the selected de-duplication domain, based on the data object characteristic associated with the selected de-duplication domain.

17. The system according to claim 16 , wherein the controller is further configured to:

receive a data stream; and

identify the backup data object using metadata included in the data stream.

18. The system according to claim 17 , wherein the data stream is multiplexed.

19. The system according to claim 17 , wherein the controller is further configured to extract metadata included in the data stream with the backup data object.

20. The system according to claim 19 , wherein the controller is further configured to determine that the backup data object has the at least one characteristic associated with the de-duplication domain by comparing the extracted metadata associated with the backup data object to each set of data object characteristics associated with each de-duplication domain.

21. The system according to claim 19 , wherein the controller is further configured to extract at least one of a back-up policy name, a data source type, a data source name, a back-up application name, an operating system type, a data type, a back-up type, a filename, a directory structure and chronological information.

22. The system according to claim 16 , wherein the controller is further arranged to configure each of the plurality of de-duplication domains to use one of a plurality of de-duplication methods.

23. The system according to claim 22 , wherein the controller is further arranged to configure each of the plurality of de-duplication domains to use one de-duplication method selected from the group comprising hash-fingerprinting, pattern recognition and content aware de-duplication.

24. The system according to claim 16 , wherein the controller is further configured to associate each of the plurality of de-duplication domains with a set of one or more data object characteristics.

25. The system according to claim 16 , wherein the controller is further configured to:

cause, within the selected de-duplication domain, de-duplication of the backup data object; and

adjust the data object characteristic associated with at least one of the plurality of de-duplication domains based on a result of the act of de-duplicating.

26. The system according to claim 25 , wherein the controller is further configured to store data in a de-duplication domain database.

27. The system according to claim 16 , wherein the system is included in a grid computing environment.

28. The system according to claim 16 , wherein the controller is further configured to receive the backup data object, determine that the backup data object has the at least one characteristic associated with a de-duplication domain and direct the backup data object to the de-duplication domain while data is not being backed-up to the system.

29. The system according to claim 16 , wherein the controller is further configured to receive the backup data object, determine that the backup data object has the at least one characteristic associated with a de-duplication domain and direct the backup data object to the de-duplication domain while data is being backed-up to the system.