System and method for storing redundant information
A method and system for reducing storage requirements and speeding up storage operations by reducing the storage of redundant data includes receiving a request that identifies one or more data objects to which to apply a storage operation. For each data object, the storage system determines if the data object contains data that matches another data object to which the storage operation was previously applied. If the data objects do not match, then the storage system performs the storage operation in a usual manner. However, if the data objects do match, then the storage system may avoid performing the storage operation.
1. A method performed by a computer system of storing a single-instance copy of multiple data objects on a sequential storage medium, the method comprising:
storing, on a random-access storage medium, a single-instance copy of the multiple, substantially identical, data objects, wherein:
the single-instance copy comprises a copy of only one of the multiple data objects,
the single-instance copy is associated with a first logical location in the random-access storage medium of a first data object selected from the multiple, substantially identical, data objects;
determining whether a data object to be stored on the random-access storage medium is duplicative of the single-instance copy;
creating a first reference to the single-instance copy based on a determination that the data object is duplicative of the single-instance copy, the first reference associated with a second logical location in the random-access storage medium, wherein the second logical location is different from the first logical location;
transferring the single-instance copy from the random-access storage medium to a sequential storage medium; and
storing a second reference to the single-instance copy on the sequential storage medium after transferring the single-instance copy to the sequential storage medium.
2. The method of claim 1 , further comprising:
receiving or accessing the multiple data objects from multiple, different logical locations within a computer network,
wherein the single-instance copy contains information identifying the logical locations, and
wherein storing the second reference to the single-instance copy includes storing the information identifying the logical locations.
3. The method of claim 1 , wherein storing the second reference to the single-instance copy includes storing a reference count to track a number of references that refer to the single-instance copy.
4. The method of claim 1 , wherein the second reference to the single-instance copy includes a media identifier identifying a storage medium on which the single-instance copy is stored and an offset within the identified storage medium to the single-instance copy.
5. The method of claim 1 , further comprising:
maintaining an index on the random-access storage medium, wherein the index includes, for each of the multiple data objects:
an identifier for the data object;
information indicating whether the data object is stored as a copy or a reference to a copy; and
an identifier of a source copy when the data object is stored as a reference to the source copy.
6. The method of claim 1 , wherein storing, on the random-access storage medium, the single-instance copy of the multiple data objects includes storing the single-instance copy of the multiple data objects using an index that includes, for each of the multiple data objects:
an identifier of the data object;
information indicating whether the data object is stored as a copy or a reference to a copy; and
an identifier of a source copy when the data object is stored as a reference to the source copy.
7. The method of claim 1 , further comprising:
receiving or accessing the multiple data objects from multiple, different logical locations within a computer network, wherein the single-instance copy contains information identifying the logical locations;
storing an index that includes, for each of the multiple data objects:
an identifier of the data object;
information indicating whether the data object is stored as a copy or a reference to a copy; and
an identifier of a source copy when the data object is stored as a reference to the source copy; and
storing the information identifying the logical locations on the sequential storage medium.
8. The method of claim 1 , wherein at least some of the multiple data objects are of different types or formats, and wherein the different types or formats correspond to documents, email messages, and configuration settings.
9. A method performed by a computer system of storing a de-duplicated copy of data objects on a sequential storage medium, comprising:
storing, on a random-access storage medium, a de-duplicated copy of one or more data objects, wherein the de-duplicated copy comprises:
a first instance of a first data object selected from the one or more data objects, and
one or more references to other data objects selected from the one or more data objects stored on the random-access storage medium; and
transferring the de-duplicated copy of the one or more data objects from the random-access storage medium to a sequential storage medium.
10. The method of claim 9 , wherein the data objects are of at least two different object types, and wherein the different types or formats correspond to documents, email messages, or configuration settings.
11. The method of claim 9 , wherein at least one of the data objects has an archive file format.
12. The method of claim 9 , wherein the sequential storage medium is one or more magnetic tapes.
13. A non-transitory, computer-readable medium containing instructions for controlling a computer system to execute a method of storing a copy on a sequential storage medium, the method comprising:
receiving or accessing multiple data objects from a computer network, wherein some of the multiple data objects are identified as identical based on hashing;
storing a de-duplicated copy of the multiple data objects to a random-access storage medium, wherein the de-duplicated copy is associated with a first network location on the computer network;
creating a reference in the random-access storage medium to the de-duplicated copy, wherein the reference is associated with a second network location on the computer network; and
transferring the de-duplicated copy of the multiple data objects from the random-access storage medium to a sequential storage medium.
14. The non-transitory, computer-readable medium of claim 13 , further comprising:
storing a reference to the de-duplicated copy of the multiple data objects to the sequential storage medium after transferring the de-duplicated copy of the multiple data objects from the random-access storage medium to the sequential storage medium.
15. The non-transitory, computer-readable medium of claim 13 , wherein the deduplicated copy contains a copy of only one of the multiple, substantially identical, data objects.
16. The non-transitory, computer-readable medium of claim 13 , wherein receiving or accessing multiple data objects from a computer network includes receiving or accessing multiple data objects from multiple, different logical locations within a computer network; and
wherein the de-duplicated copy contains information identifying the logical locations.
17. The non-transitory, computer-readable medium of claim 13 , further comprising maintaining an index on the random-access storage medium, wherein the index comprises, for each of the multiple data objects:
an identifier of the data object;
information indicating whether the data object is stored as a copy or a reference to a copy; and
an identifier of a source copy when the data object is stored as a reference to the source copy.
18. The non-transitory, computer-readable medium of claim 13 , wherein at least some of the multiple data objects are of different types or formats.
19. The non-transitory, computer-readable medium of claim 13 , wherein at least some of the multiple data objects include documents and email messages.
20. The non-transitory, computer-readable medium of claim 13 , wherein at least some of the multiple data objects include documents and configuration settings.