Systems and methods for managing single instancing data
Described in detail herein are systems and methods for managing single instancing data. Using a single instance database and other constructs (e.g. sparse files), data density on archival media (e.g. magnetic tape) is improved, and the number of files per storage operation is reduced. According to one aspect of a method for managing single instancing data, for each storage operation, a chunk folder is created on a storage device that stores single instancing data. The chunk folder contains three files: 1) a file that contains data objects that have been single instanced; 2) a file that contains data objects that have not been eligible for single instancing; and 3) a metadata file used to track the location of data objects within the other files. A second storage operation subsequent to a first storage operation contains references to data objects in the chunk folder created by the first storage operation instead of the data objects themselves.
1. A computer-implemented method for storing deduplicated data, wherein the method is performed by one or more computing systems in a data storage system, each computing system comprising one or more hardware processors and computer memory, the method comprising:
performing a first storage operation that generates:
a first logical data container comprising single-instanced data objects;
a second logical data container comprising data objects that are not single-instanced; and
an index that stores references to locations within the first logical data container of the single-instanced data objects and to locations within the second logical data container of the data objects that are not single-instanced;
receiving a first data object to be stored, wherein the first data object comprises a first metadata portion and a first payload data portion;
based on determining that the first payload data portion of the first data object is a duplicate of an existing single-instanced data object in the data storage system:
storing the first metadata portion of the first data object in the second logical data container, wherein the first payload data portion is not stored in the first logical data container, and
adding to the index a first reference to a location of the existing single-instanced data object,
wherein the existing single-instanced data object is stored in one of: the first logical data container and an other logical data container comprising single-instanced data objects,
wherein the other logical data container is distinct from the second logical data container comprising data objects that are not single-instanced; and
further adding to the index a second reference to a location within the second logical data container of the first metadata portion of the first data object.
2. The computer-implemented method of claim 1 further comprising:
receiving a request to restore the first payload data portion of the first data object;
based at least in part on parsing the index, determining the location of the existing single-instanced data object in one of: the first logical data container and the other logical data container comprising single-instanced data objects;
opening one of: the first logical data container and the other logical data container, where the existing single-instanced data object is located;
reading the existing single-instanced data object from the location thereof; and
streaming the existing single-instanced data object in response to the request to restore the first payload data portion of the first data object.
3. The computer-implemented method of claim 1 , wherein the first data object further comprises a second payload data portion, which is smaller than the first payload data portion and is also smaller than an administrable threshold; and further comprising:
storing the second payload data portion of the first data object in the second logical data container; and
adding to the index a third reference to a location within the second logical data container of the second payload data portion.
4. The computer-implemented method of claim 1 further comprising:
maintaining a single-instance database, separate from the index, wherein the single-instance database comprises a substantially unique identifier, within the data storage system, for each single-instanced data object in the data storage system, wherein the determining is based on checking whether a first identifier of the first payload data portion of the first data object exists in the single-instance database; and
based on the checking, further determining that the first identifier is identical to an existing substantially unique identifier of the existing single-instanced data object.
5. The computer-implemented method of claim 4 , wherein an entry for the existing substantially unique identifier in the single-instance database includes a reference to the location of the existing single-instanced data object in a certain logical data container and further includes an offset within the certain logical data container where the existing single-instanced data object is located.
6. The computer-implemented method of claim 4 further comprising: incrementing, in the single-instance database, a reference count of the existing substantially unique identifier.
7. The computer-implemented method of claim 6 , wherein after the incrementing, the first payload data portion of the first data object is tracked by the data storage system as a single-instanced data object.
8. The computer-implemented method of claim 1 further comprising:
based on determining that the first payload data portion of the first data object is not a duplicate of other single-instanced data objects in the data storage system:
storing the first payload data portion of the first data object in the first logical data container,
storing the first metadata portion of the first data object in the second logical data container, and
adding to the index a first reference to a location within the first logical data container of the first payload data portion.
9. The computer-implemented method of claim 8 further comprising:
maintaining a single-instance database, separate from the index, wherein the single-instance database comprises a substantially unique identifier, within the data storage system, for each single-instanced data object in the data storage system, wherein the determining that the first payload data portion of the first data object is not a duplicate of other single-instanced data objects in the data storage system is based on checking whether a first identifier of the first payload data portion of the first data object exists in the single-instance database; and
based on the checking, further determining that the first identifier is different from existing substantially unique identifiers in the single-instance database.
10. The computer-implemented method of claim 9 further comprising:
adding the first identifier of the first payload data object to the single-instance database, with a reference to the first logical data container, to reflect the storing of the first payload data portion of the first data object in the first logical data container.
11. A data storage system comprising:
one or more computing systems, each computing system comprising one or more hardware processors and computer memory;
wherein the one or more computing systems are configured to:
perform a first storage operation that generates:
a first logical data container comprising single-instanced data objects, wherein the first logical data container resides on a data storage device;
a second logical data container comprising data objects that are not single-instanced, wherein the second logical data container resides on a data storage device; and
an index that stores references to locations within the first logical data container of the single-instanced data objects and to locations within the second logical data container of the data objects that are not single-instanced, wherein the index resides on a data storage device;
receive a first data object to be stored, wherein the first data object comprises a first metadata portion and a first payload data portion;
based on determining that the first payload data portion of the first data object is a duplicate of an existing single-instanced data object in the data storage system:
store the first metadata portion of the first data object in the second logical data container, wherein the first payload data portion is not stored in the first logical data container, and
add to the index a first reference to a location of the existing single-instanced data object,
wherein the existing single-instanced data object is stored in one of: the first logical data container and an other logical data container comprising single-instanced data objects,
wherein the other logical data container is distinct from the second logical data container comprising data objects that are not single-instanced; and
further add to the index a second reference to a location within the second logical data container of the first metadata portion of the first data object.
12. The data storage system of claim 11 , wherein the one or more computing systems are further configured to:
receive a request to restore the first payload data portion of the first data object;
based at least in part on parsing the index, determine the location of the existing single-instanced data object in the one of: the first logical data container and the other logical data container comprising single-instanced data objects;
open the one of: the first logical data container and the other logical data container, where the existing single-instanced data object is located;
read the existing single-instanced data object from the location thereof; and
stream the existing single-instanced data object in response to the request to restore the first payload data portion of the first data object.
13. The data storage system of claim 11 , wherein the first data object further comprises a second payload data portion, which is smaller than the first payload data portion and is also smaller than an administrable threshold; and
wherein the one or more computing systems are further configured to:
store the second payload data portion of the first data object in the second logical data container; and
adding to the index a third reference to a location within the second logical data container of the second payload data portion.
14. The data storage system of claim 11 , wherein the one or more computing systems are further configured to:
maintain a single-instance database, separate from the index, wherein the single-instance database comprises a substantially unique identifier, within the data storage system, for each single-instanced data object in the data storage system, wherein the determining is based on checking whether a first identifier of the first payload data portion of the first data object exists in the single-instance database; and
based on the checking, further determine that the first identifier is identical to an existing substantially unique identifier of the existing single-instanced data object.
15. The data storage system of claim 14 , wherein an entry for the existing substantially unique identifier in the single-instance database includes a reference to the location of the existing single-instanced data object in a certain logical data container and further includes an offset within the certain logical data container where the existing single-instanced data object is located.
16. The data storage system of claim 14 , wherein the one or more computing systems are further configured to:
increment, in the single-instance database, a reference count of the existing substantially unique identifier.
17. The data storage system of claim 16 , wherein after incrementing the reference count, the first payload data portion of the first data object is tracked by the data storage system as a single-instanced data object.
18. The data storage system of claim 11 , wherein the one or more computing systems are further configured to:
based on determining that the first payload data portion of the first data object is not a duplicate of other single-instanced data objects in the data storage system:
store the first payload data portion of the first data object in the first logical data container,
store the first metadata portion of the first data object in the second logical data container, and
add to the index a first reference to a location within the first logical data container of the first payload data portion.
19. The data storage system of claim 18 , wherein the one or more computing systems are further configured to:
maintain a single-instance database, separate from the index, wherein the single-instance database comprises a substantially unique identifier, within the data storage system, for each single-instanced data object in the data storage system, wherein the determining that the first payload data portion of the first data object is not a duplicate of other single-instanced data objects in the data storage system is based on checking whether a first identifier of the first payload data portion of the first data object exists in the single-instance database; and
based on the checking, further determine that the first identifier is different from existing substantially unique identifiers in the single-instance database.
20. The data storage system of claim 19 , wherein the one or more computing systems are further configured to:
add the first identifier of the first payload data object to the single-instance database, with a reference to the first logical data container, to reflect the storing of the first payload data portion of the first data object in the first logical data container.