Sharing services between deduplication systems
Disclosed herein are methods, systems, and processes to share data storage-related services between multiple deduplication systems. A determination is made as to whether a shared file that exists in a library of shared files corresponds to a local file. In response to a determination that the shared file exists in the library of shared files, the local file is virtually segmented into data segments, and local virtual mapping metadata is generated based, at least in part, on the data segments. The local virtual mapping metadata is then stored.
1. A computer-implemented method comprising:
determining, at a deduplication system, whether a shared file corresponds to a local file, wherein
the shared file exists in a library of shared files, and
the library of shared files is maintained by a library server;
in response to a determination that the shared file exists in the library of shared files,
virtually segmenting the local file into a plurality of data segments, wherein
virtually segmenting the local file comprises generating local virtual mapping metadata,
the local virtual mapping metadata is generated based, at least in part, on the plurality of data segments,
the local virtual mapping metadata comprises
fingerprint information corresponding to each of the plurality of data segments, and
at least one of a deduplication system identifier, a unique file identifier, an offset of each data segment, or a size of each data segment,
the local virtual mapping metadata identifies the deduplication system, and
the local virtual mapping metadata permits identification of one or more data segments of the plurality of data segments of the local file that can be excluded from a backup and deduplication operation;
storing the local virtual mapping metadata in the deduplication system; and
performing a deduplication operation at the deduplication system, wherein
the deduplication operation is performed, based at least in part, by using the local virtual mapping metadata stored in the deduplication system.
2. The computer-implemented method of claim 1 , wherein the shared file is stored by a library server,
the local file is selected to be backed up and deduplicated as part of a backup and deduplication operation, and
the local file is determined to be the shared file if a local file identifier generated for the local file matches a unique file identifier generated for the shared file by the library server.
3. The computer-implemented method of claim 2 , further comprising:
receiving a name of the shared file and a name of a file fingerprinting algorithm used to generate the unique file identifier from the library server; and
generating the local file identifier based on the file fingerprinting algorithm.
4. The computer-implemented method of claim 3 , wherein
the name of the shared file and the name of the file fingerprinting algorithm is received via an Application Programming Interface (API).
5. The computer-implemented method of claim 4 , further comprising:
receiving an indication from the library server via the API that the shared file is the local file; and
transmitting the local virtual mapping metadata to the library server.
6. The computer-implemented method of claim 2 , wherein
the shared file is one of a plurality of shared files stored by the library server, and
the determining, the virtually segmenting, and the storing are performed by a deduplication system.
7. The computer-implemented method of claim 2 , further comprising:
instructing the library server to virtually segment the shared file if the local file identifier generated for the local file matches the unique file identifier generated for the shared file by the library server, wherein
the virtual segmentation of the shared file by the library server generates a library virtual mapping metadata.
8. The computer-implemented method of claim 1 , wherein
the shared file comprises non-user-sensitive data, and
the shared file is shared by a plurality of deduplication systems.
9. A non-transitory computer readable storage medium comprising program instructions executable to perform a method comprising:
determining, at a deduplication system, whether a shared file corresponds to a local file, wherein
the shared file exists in a library of shared files, and
the library of shared files is maintained by a library server;
in response to a determination that the shared file exists in the library of shared files,
virtually segmenting the local file into a plurality of data segments, wherein
virtually segmenting the local file comprises generating local virtual mapping metadata,
the local virtual mapping metadata is generated based, at least in part, on the plurality of data segments,
the local virtual mapping metadata comprises
fingerprint information corresponding to each of the plurality of data segments, and
at least one of a deduplication system identifier, a unique file identifier, an offset of each data segment, or a size of each data segment,
the local virtual mapping metadata identifies the deduplication system, and
the local virtual mapping metadata permits identification of one or more data segments of the plurality of data segments of the local file that can be excluded from a backup and deduplication operation;
storing the local virtual mapping metadata in the deduplication system; and
performing a deduplication operation at the deduplication system, wherein
the deduplication operation is performed, based at least in part, by using the local virtual mapping metadata stored in the deduplication system.
10. The non-transitory computer readable storage medium of claim 9 , wherein
the shared file is stored by a library server,
the local file is selected to be backed up and deduplicated as part of a backup and deduplication operation,
the local file is determined to be the shared file if a local file identifier generated for the local file matches a unique file identifier generated for the shared file by the library server.
11. The non-transitory computer readable storage medium of claim 10 , wherein the method further comprises:
receiving a name of the shared file and a name of a file fingerprinting algorithm used to generate the unique file identifier from the library server; and
generating the local file identifier based on the file fingerprinting algorithm.
12. The non-transitory computer readable storage medium of claim 11 , wherein
the shared file is one of a plurality of shared files stored by the library server,
the shared file comprises non-user-sensitive data,
the shared file is shared by a plurality of deduplication systems,
the determining, the virtually segmenting, and the storing are performed by a deduplication system of the plurality of deduplication systems, and
the name of the shared file and the name of the file fingerprinting algorithm is received via an Application Programming Interface (API).
13. The non-transitory computer readable storage medium of claim 12 , wherein the method further comprises:
receiving an indication from the library server via the API that the shared file is the local file; and
transmitting the local virtual mapping metadata to the library server, or
instructing the library server to virtually segment the shared file if the local file identifier generated for the local file matches the unique file identifier generated for the shared file by the library server, wherein
the virtual segmentation of the shared file by the library server generates a library virtual mapping metadata.
14. A system comprising:
one or more processors; and
a memory coupled to the one or more processors, wherein the memory stores program instructions executable by the one or more processors to perform a method comprising:
determining, at a deduplication system, whether a shared file corresponds to a local file, wherein
the shared file exists in a library of shared files, and
the library of shared files is maintained by a library server;
in response to a determination that the shared file exists in the library of shared files, virtually segmenting the local file into a plurality of data segments, wherein virtually segmenting the local file comprises generating local virtual mapping metadata,
the local virtual mapping metadata is generated based, at least in part, on the plurality of data segments,
the local virtual mapping metadata comprises
fingerprint information corresponding to each of the plurality of data segments, and
at least one of a deduplication system identifier, a unique file identifier, an offset of each data segment, or a size of each data segment,
the local virtual mapping metadata identifies the deduplication system, and
the local virtual mapping metadata permits identification of one or more data segments of the plurality of data segments of the local file that can be excluded from a backup and deduplication operation;
storing the local virtual mapping metadata in the deduplication system; and
performing a deduplication operation at the deduplication system, wherein
the deduplication operation is performed, based at least in part, by using the local virtual mapping metadata stored in the deduplication system.
15. The system of claim 14 , wherein
the shared file is stored by a library server,
the local file is selected to be backed up and deduplicated as part of a backup and deduplication operation,
the local file is determined to be the shared file if a local file identifier generated for the local file matches a unique file identifier generated for the shared file by the library server.
16. The system of claim 15 , wherein the method further comprises:
receiving a name of the shared file and a name of a file fingerprinting algorithm used to generate the unique file identifier from the library server; and
generating the local file identifier based on the file fingerprinting algorithm.
17. The system of claim 16 , wherein
the shared file is one of a plurality of shared files stored by the library server,
the shared file comprises non-user-sensitive data,
the shared file is shared by a plurality of deduplication systems,
the determining, the virtually segmenting, and the storing are performed by a deduplication system of the plurality of deduplication systems, and
the name of the shared file and the name of the file fingerprinting algorithm is received via an Application Programming Interface (API).
18. The system of claim 17 , wherein the method further comprises:
receiving an indication from the library server via the API that the shared file is the local file; and
transmitting the local virtual mapping metadata to the library server, or
instructing the library server to virtually segment the shared file if the local file identifier generated for the local file matches the unique file identifier generated for the shared file by the library server, wherein
the virtual segmentation of the shared file by the library server generates a library virtual mapping metadata.