De-duplication systems and methods for application-specific data
Content-aware systems and methods for improving de-duplication, or single instancing, in storage operations. In certain examples, backup agents on client devices parse application-specific data to identify data objects that are candidates for de-duplication. The backup agents can then insert markers or other indictors in the data that identify the location(s) of the particular data objects. Such markers can, in turn, assist a de-duplication manager to perform object-based de-duplication and increase the likelihood that like blocks within the data are identified and single instanced. In other examples, the agents can further determine if a data object of one file type can or should be single-instanced with a data object of a different file type. Such processing of data on the client side can provide for more efficient storage and back-end processing.
1. A system for creating a backup copy of application-generated data objects, the system comprising:
computer readable memory comprising at least a first de-duplication database, the first de-duplication database associated with first application-specific data, the first application-specific data comprising a first plurality of data objects generated by at least first and second clients;
at least first and second data agents, the first data agent in communication with at least the first client, and the second data agent in communication with at least the second client, the first and second data agents comprising one or more computer processors comprising computer hardware, the first and second data agents monitor the first application-specific data created by the at least first and second clients and insert headers comprising de-duplication indicators in the first application-specific data to identify portions of the first plurality of data objects to be processed for duplicate data, the first application-specific data further comprising metadata associated with one or more of the first plurality of data objects, wherein the metadata further identifies at least the first and second clients,
wherein one or more of the de-duplication indicators comprises an offset value identifying a beginning of at least the portions of the first plurality of data objects within the first application-specific data;
the at least one media agent executing on one or more computer processors, the at least one media agent manages a backup of the first application-specific data, the headers comprising the de-duplication indicators, and the metadata received from the first and second data agents;
a de-duplication module that communicates with the media agent, the de-duplication module executing on one or more computer processors comprising computer hardware, the de-duplication module receives the application-specific data from the media agent and performs block-level de-duplication on the first application-specific data as part of the backup of the first application-specific data, the de-duplication module further configured to:
use the inserted headers comprising the de-duplication indicators to identify the beginning of at least one of a first portion of the first plurality of data objects from the first client based on the offset value; and
determine if a duplicate copy of blocks associated with the first portion of the first plurality of data objects from the first client exist in the first de-duplication database; and
if a duplicate copy does not exist in the first de-duplication database, storing the first portion of the blocks in the first de-duplication database and storing first metadata that identifies the first client in associated with the duplicate copy;
determine if a duplicate copy of any of blocks associated with a second portion of the first plurality of data objects from the second client exist in the first de-duplication database;
if a duplicate copy exists in the first de-duplication database, removing the duplicate data in the second portion the first application-specific data associated with the second client;
determining whether second metadata in the second portion of the first application-specific data that identifies the second client is unique; and
if the second metadata is unique, creating a backup copy of the second metadata in association with the duplicate copy in the first de-duplication database, wherein the backup copy stores multiple instances of differing metadata associated with the duplicate copy wherein the first metadata that identifies the first client and the second metadata that identifies the second client are stored in association with the duplicate copy.
2. The system of claim 1 , further comprising a third module executing on one or more computer processors configured to:
parse second application-specific data that is different in format than the first application-specific data, the second application-specific data comprising a second plurality of data objects;
identify portions within the second plurality of data objects to be considered for de-duplication; and
insert at least one de-duplication indicator in the second application-specific data that identifies at least one location of the identified portions in the second plurality of data objects to be considered for de-duplication.
3. The system of claim 2 , wherein the de-duplication module is further configured to determine whether a duplicate copy of any of the blocks associated with the second plurality of data objects exist in a second de-duplication database.
4. The system of claim 2 , wherein the inserted de-duplication indicators in the second application-specific data further identify that the second de-duplication database is to be used in de-duplicating the application-specific data.
5. The system of claim 2 , wherein:
the first de-duplication database is configured to store unique blocks of the identified portions of the first portion of the first plurality of data objects associated with the first application-specific data; and
the second de-duplication database is configured to store unique blocks of the identified portions of the second plurality of data objects associated with the second application-specific data, wherein the first de-duplication database is separate and different from the second de-duplication database.
6. The system of claim 2 , wherein the de-duplication indicators indicate at least one of the first de-duplication database and the second de-duplication database.
7. The system of claim 1 , wherein the first application-specific data is associated with an electronic mail server application.
8. A method for creating a backup copy of application-generated data objects, the method comprising:
storing a first de-duplication database associated with first application-specific data, the first application-specific data comprising a first plurality of data objects;
inserting with at least first and second data agents de-duplication indicators in the first application-specific data generated by first and second clients, wherein the at least first and second data agents monitor the first application-specific data generated by the first and second clients and insert headers comprising one or more de-duplication indicators, wherein the one or more de-duplication indicators comprises an offset value identifying a beginning of at least portions of the first plurality of data objects within the first application-specific data to be processed for duplicate data, the first application-specific data further comprising metadata associated with one or more of the first plurality of data objects, and wherein the metadata further identifies at least the first and second clients;
sending to at least one media agent, the first application-specific data, the headers comprising the de-duplication indicators, and the metadata, wherein the media agent manages a backup of the first application-specific data received from the first and second data agents;
using the inserted headers comprising de-duplication indicators as part of the backup of the first application-specific data to identify the beginning of at least one of a first portion of the first plurality of data objects from the first client based on the offset value;
determining if a duplicate copy of blocks associated with the first portion of the first plurality of data objects from the first client exist in the first de-duplication database;
if a duplicate copy does not exist in the first de-duplication database, storing the first portion of the first plurality of data objects in the first de-duplication database;
determining if a duplicate copy of blocks associated with a second portion of the first plurality of data objects from the second client exist in the first de-duplication database;
if a duplicate copy exists in the first de-duplication database, removing the duplicate data in the second portion the first application-specific data associated with the second client;
determining whether the metadata in the second portion of the first application-specific data that identifies the second client is unique; and
if the metadata is unique, creating a backup copy of the metadata in association with the duplicate copy in the first de-duplication database, wherein the backup copy stores multiple instances of differing metadata associated with the duplicate copy wherein a first instance of the duplicate copy is stored in association with the first client and a second instance of the duplicate copy is stored in association with the second client.
9. The method of claim 8 , further comprising identifying portions within the first portion of the first plurality of data objects to be considered for de-duplication.
10. The method of claim 8 , further comprising:
parsing second application-specific data that is in a different format than the first application-specific data, the second application-specific data comprising a second plurality of second data objects;
identifying portions within the second plurality of second data objects to be considered for de-duplication; and
inserting at least one de-duplication indicator in the second application-specific data that identifies at least one location of the identified portions in the second plurality of data objects to be considered for de-duplication.
11. The method of claim 10 , further comprising determining whether a duplicate copy of any of the blocks associated with the second plurality of data objects exist in a second de-duplication database.
12. The method of claim 10 , wherein the inserted de-duplication indicators in the second application-specific data further identify that the second de-duplication database is to be used in de-duplicating the application-specific data.
13. The method of claim 10 , wherein:
the first de-duplication database is configured to store unique blocks of the first portion of the first plurality of data objects associated with the first application-specific data; and
the second de-duplication database is configured to store unique blocks of the identified portions of the second plurality of data objects associated with the second application-specific data, wherein the first de-duplication database is separate and different from the second de-duplication database.
14. The method of claim 10 , wherein the de-duplication indicators indicate at least one of the first de-duplication database and the second de-duplication database.
15. The method of claim 8 , wherein the first application-specific data is associated with an electronic mail server application.