Utilizing data source identifiers to obtain deduplication efficiency within a clustered storage environment
Described is a system (and method) that intelligently distributes data within a clustered storage environment. To provide such a capability, the system may distribute backup files by considering a source of the data to be backed-up. In particular, the system may leverage the ability of front-end components such as a backup application to perform a granular data source identification of data. Such information may be propagated to back-end components such as a storage filesystem in the form of a data source identifier (e.g. placement tag). The data source identifiers may then be accessed by the clustered storage system to intelligently distribute backup files amongst a set of storage nodes forming a cluster. For example, backup files from the same data source may be stored on the same storage node to obtain the same deduplication efficiency as a single storage system.
1. A storage system comprising:
one or more processors; and
a non-transitory computer readable medium storing a plurality of instructions, which when executed, cause the one or more processors to:
receive, from a backup application, data to be stored by the storage system;
receive, from the backup application, a request to create a first backup file using the received data, wherein receiving the request to create the first backup file includes receiving a first data source identifier associated with the received data in a form of an argument to an operation that initiates the creation of a backup file to the storage system, the operation provided to the backup application by the storage system as part of an programming interface (API), the first data source identifier including an identity of a device from which the received data was obtained, the device being one of a plurality of devices that are components of a client system;
traverse a file system to retrieve data source identifiers associated with backup files already stored within one or more deduplication domains of the storage system, in response to the request to create the first backup file;
determine a first deduplication domain stores data originating from the same device as the received data if at least a second data source identifier associated with a second backup file already stored within the first deduplication domain matches the first data source identifier; and
write the first backup file to the first deduplication domain, in response to determining the first deduplication domain stores data originating from the same device as the received data.
2. The system of claim 1 , wherein writing the first backup file to the first deduplication domain is based on a distribution policy that prioritizes grouping backup files storing data originating from the same data source to a same deduplication domain over distributing backup files based on a resource utilization associated with the deduplication domains.
3. The system of claim 2 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
determine none of the deduplication domains store data originating from the same data source as the received data if none of the data source identifiers associated with the backup files already stored in the deduplication domains of the storage system match the first data source identifier; and
write the first backup file to any of the deduplication domains of the storage system, in response to determining no deduplication domain stores data originating from the same data source as the received data.
4. The system of claim 3 , wherein writing the first backup file to any of the deduplication domains of the storage system is based on the distribution policy that prioritizes distributing backup files to deduplication domains based on the resource utilization associated with the deduplication domains without further regard to the data source identifier associated with the received data.
5. The system of claim 1 , wherein traversing the file system to retrieve the data source identifiers includes accessing an index node of the file system and traversing the index node to retrieve the data source identifiers.
6. The system of claim 1 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
store the first data source identifier within the file system maintained by the system as an attribute associated with the first backup file.
7. The system of claim 1 , wherein the storage system comprises a clustered storage system, and each deduplication domain comprises a storage node, or a storage partition associated with one or more storages nodes.
8. A method comprising:
receiving, from a backup application, data to be stored by the storage system;
receiving, from the backup application, a request to create a first backup file using the received data, wherein receiving the request to create the first backup file includes receiving a first data source identifier associated with the received data, the first data source identifier including an identity of a device from which the received data was obtained, the device being one of a plurality of devices that are components of a client system;
traversing, by the storage system, a file system to retrieve data source identifiers associated with backup files already stored within one or more deduplication domains of the storage system, in response to the request to create the first backup file;
determining, by the storage system, a first deduplication domain stores data originating from the same data source as the received data if at least a second data source identifier associated with a second backup file already stored within the first deduplication domain matches the first data source identifier; and
writing, by the storage system, the first backup file to the first deduplication domain, in response to determining the first deduplication domain stores data originating from the same data source as the received data.
9. The method of claim 8 , wherein writing the first backup file to the first deduplication domain is based on a distribution policy that prioritizes grouping backup files storing data originating from the same data source to a same deduplication domain over distributing backup files based on a resource utilization associated with the deduplication domains.
10. The method of claim 9 , further comprising:
determining none of the deduplication domains store data originating from the same data source as the received data if none of the data source identifiers associated with the backup files already stored in the deduplication domains of the storage system match the first data source identifier; and
writing the first backup file to any of the deduplication domains of the storage system, in response to determining no deduplication domain stores data originating from the same data source as the received data.
11. The method of claim 10 , wherein writing the first backup file to any of the deduplication domains of the storage system is based on the distribution policy that prioritizes distributing backup files to deduplication domains based on the resource utilization associated with the deduplication domains without further regard to the data source identifier associated with the received data.
12. The method of claim 8 , wherein traversing the file system to retrieve the data source identifiers includes accessing an index node of the file system and traversing the index node to retrieve the data source identifiers.
13. A computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein to be executed by one or more processors, the program code including instructions to:
receive, from a backup application, data to be stored by the storage system;
receive, from the backup application, a request to create a first backup file using the received data, wherein receiving the request to create the first backup file includes receiving a first data source identifier associated with the received data, the first data source identifier including an identity of a device from which the received data was obtained, the device being one of a plurality of devices that are components of a client system;
traverse a file system to retrieve data source identifiers associated with backup files already stored within one or more deduplication domains of the storage system, in response to the request to create the first backup file;
determine a first deduplication domain stores data originating from the same data source as the received data if at least a second data source identifier associated with a second backup file already stored within the first deduplication domain matches the first data source identifier; and
write the first backup file to the first deduplication domain, in response to determining the first deduplication domain stores data originating from the same data source as the received data.
14. The computer program product of claim 13 , wherein writing the first backup file to the first deduplication domain is based on a distribution policy that prioritizes grouping backup files storing data originating from the same data source to a same deduplication domain over distributing backup files based on a resource utilization associated with the deduplication domains.
15. The computer program product of claim 14 , wherein the program code includes further instructions to:
determine none of the deduplication domains store data originating from the same data source as the received data if none of the data source identifiers associated with the backup files already stored in the deduplication domains of the storage system match the first data source identifier; and
write the first backup file to any of the deduplication domains of the storage system, in response to determining no deduplication domain stores data originating from the same data source as the received data.
16. The computer program product of claim 15 , wherein writing the first backup file to any of the deduplication domains of the storage system is based on the distribution policy that prioritizes distributing backup files to deduplication domains based on the resource utilization associated with the deduplication domains without further regard to the data source identifier associated with the received data.
17. The computer program product of claim 13 , wherein traversing the file system to retrieve the data source identifiers includes accessing an index node of the file system and traversing the index node to retrieve the data source identifiers.