System and method for data de-duplication
View Patent ↗A method and system can be used to read and obtain data from a variety of media, regardless of the application used to generate the backup media. The component parts of a file may be read from a medium, including content and metadata pertaining to a file. These pieces of content and metadata may then be stored and associated. To avoid duplication of data, pieces of content and metadata may be compared to previously stored content and metadata. Furthermore, using these same methods and systems the content and metadata of a file may be associated with a location where the file resided. A database which stores these components and allows linking between the various stored components may be particularly useful in implementing embodiments of these methods and systems.
1. A method utilizing one or more computer systems for extracting data directly from various distributed media and storing the data at a target location, the target location comprising one or more computer memory devices, without unnecessary duplication of identical data, comprising the steps of:
reading components of data in a plurality of files stored in a plurality of data storage devices, wherein the components of data include a content for each file, a location of the file as to where it is stored in the plurality of data storage devices, and a metadata for each file;
comparing the content with previously stored content, the previously stored content stored in a target location, the target location comprising one or more data storage devices, wherein comparing the content comprises generating a digital signature from the content and comparing the digital signature of the content with a digital signature of previously stored content;
storing the content at the target location only if substantially identical content is not found;
comparing the metadata to previously stored metadata, the previously stored metadata stored in a target location, the target location comprising one or more data storage devices;
storing the metadata at the target location only if substantially identical metadata is not found, wherein the stored metadata is associated with the substantially identical content if it is found, and the stored content if substantially identical content is not found;
associating the substantially identical metadata with the stored content if substantially identical metadata is found and substantially identical content is not found;
comparing the location of the file to a previously stored location, the previously stored location stored in a target location and storing the location if a substantially identical location is not found, wherein the location is associated with the metadata or content of the file; and
applying retention criteria to insure that any data stored meets the criteria for the retention criteria.
2. The method of claim 1 , wherein comparing the content comprises generating a digital signature from the content and comparing the digital signature of the content with digital signatures of previously stored content.
3. The method of claim 2 , wherein generating a digital signature from the content comprises hashing the stream of bytes which comprises the content.
4. The method of claim 3 , wherein the hashing algorithm is the SHA1 160 bit secure hash algorithm number one.
5. The method of claim 3 , wherein comparing the metadata comprises generating a digital signature from the metadata and comparing the digital signature of the metadata with digital signatures of previously stored metadata.
6. The method of claim 5 , wherein generating a digital signature from the metadata comprises hashing an ASCII string assembled from the metadata.
7. The method of claim 6 , wherein the algorithm used to hash the metadata is the same algorithm used to hash the content.
8. The method of claim 7 , wherein the hashing algorithm is the SHA1 160 bit secure hash algorithm number one.
9. The method of claim 5 , wherein the content is associated with the metadata using the digital signatures of the metadata and content.
10. The method of claim 9 , and comprising storing the content and metadata in a single relational database.