Source-agnostic service for performing deduplication for an object storage
Described is a system for a providing a service (or microservice) for performing deduplication for an object storage. The service (or microservice) may be source-agnostic in that it may receive data from multiple types of source systems by providing a uniform set of functions for deduplicating and writing the data to a destination object storage. The set of functions encapsulate a previously dispersed set of functionality provided by various components. Accordingly, the service provides a single scalable and stateless component for performing deduplication. For example, the service (e.g. deduplication service) may receive object related information and perform a filtering to accelerate network transfers. Accordingly, the service provides the ability to only transfer and write data that does not already exist on the object storage.
1. A system comprising:
one or more processors; and
a non-transitory computer readable medium storing a plurality of instructions, which when executed, cause the one or more processors to execute a service to perform deduplication when copying an object from a source site to a destination site, the service configured to:
receive a segment check request to determine whether a first set of segments of the object are already stored on an object storage of the destination site, the object being divided into a plurality of slices, each slice comprising a set of segments, the segment check request including a first slice recipe associated with a first slice of the plurality of slices, the first slice recipe including fingerprints for the first set of segments;
search, using the first slice recipe, a fingerprint index of the destination site to identify a second set of segments, amongst the first set of segments, that are not already stored on the object storage;
provide, in response to the segment check request, a list of the second set of segments;
receive a copy data request to write data to the object storage, the copy data request including the second set of segments and at least a second slice recipe including fingerprints for the second set of segments; and
write, in response to the copy data request, the second set of segments and the second slice recipe to the object storage.
2. The system of claim 1 , wherein the service is further configured to:
receive an object check request to determine whether the object is already stored on the object storage, the object check request including a first object recipe associated with the object; and
provide, in response to the object check request, an indication that the object is not already stored on the object storage in response to determining the first object recipe does not reference an existing object on the object storage.
3. The system of claim 2 , wherein determining the first object recipe does not reference an existing object on the object storage includes searching an object catalog of the destination site using an object identifier associated with the first object recipe.
4. The system of claim 2 , wherein determining the first object recipe does not reference an existing object on the object storage includes determining information included in the first object recipe does not match information included in object recipes stored on the object storage.
5. The system of claim 1 , wherein the service is further configured to:
receive a copy object recipe request to write an object recipe on the object storage, the copy object recipe request including the first object recipe updated to reference the second slice recipe; and
write, in response to the copy object recipe request, the updated first object recipe to the object storage.
6. The system of claim 1 , wherein the second set of segments of the copy data request are received as compression regions, and writing the second set of segments includes writing the one or more compression regions to the object storage.
7. The system of claim 1 , wherein the service is further configured to:
update the fingerprint index of the destination site to reference the second set of segments.
8. The system of claim 1 , wherein searching the fingerprint index includes identifying a similarity group associated with the first slice recipe, and searching the similarity group for the fingerprints for the first set of segments.
9. The system of claim 1 , wherein the service is further configured to:
receive a request to establish a data transfer connection between the source site and the service, the request including a set of data transfer parameters including at least an access key; and
establish the data transfer connection between the source site and the service, in response to authenticating the access key.
10. The system of claim 1 , wherein the service is further configured to:
receive a delete object recipe request to delete an object recipe on the object storage, the delete object recipe request including a second object recipe; and
delete, in response to the delete object recipe request, the second object recipe from the object storage.
11. A method of performing deduplication when copying an object from a source site to a destination site comprising:
receiving a segment check request to determine whether a first set of segments of the object are already stored on an object storage of the destination site, the object being divided into a plurality of slices, each slice comprising a set of segments, the segment check request including a first slice recipe associated with a first slice of the plurality of slices, the first slice recipe including fingerprints for the first set of segments;
searching, using the first slice recipe, a fingerprint index of the destination site to identify a second set of segments, amongst the first set of segments, that are not already stored on the object storage;
providing, in response to the segment check request, a list of the second set of segments;
receive a copy data request to write data to the object storage, the copy data request including the second set of segments and at least a second slice recipe including fingerprints for the second set of segments; and
writing, in response to the copy data request, the second set of segments and the second slice recipe to the object storage.
12. The method of claim 11 , further comprising:
receiving an object check request to determine whether the object is already stored on the object storage, the object check request including a first object recipe associated with the object; and
providing, in response to the object check request, an indication that the object is not already stored on the object storage in response to determining the first object recipe does not reference an existing object on the object storage.
13. The method of claim 11 , further comprising:
receiving a copy object recipe request to write an object recipe on the object storage, the copy object recipe request including the first object recipe updated to reference the second slice recipe; and
writing, in response to the copy object recipe request, the updated first object recipe to the object storage.
14. The method of claim 11 , wherein the second set of segments of the copy data request are received as compression regions, and writing the second set of segments includes writing the one or more compression regions to the object storage.
15. The method of claim 11 , further comprising:
receiving a request to establish a data transfer connection between the source site and the service, the request including a set of data transfer parameters including at least an access key; and
establishing the data transfer connection between the source site and the service, in response to authenticating the access key.
16. A computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein to be executed by one or more processors, the program code including instructions to execute a service to perform deduplication when copying an object from a source site to a destination site, the service configured to configured to:
receive a segment check request to determine whether a first set of segments of the object are already stored on an object storage of the destination site, the object being divided into a plurality of slices, each slice comprising a set of segments, the segment check request including a first slice recipe associated with a first slice of the plurality of slices, the first slice recipe including fingerprints for the first set of segments;
search, using the first slice recipe, a fingerprint index of the destination site to identify a second set of segments, amongst the first set of segments, that are not already stored on the object storage;
provide, in response to the segment check request, a list of the second set of segments;
receive a copy data request to write data to the object storage, the copy data request including the second set of segments and at least a second slice recipe including fingerprints for the second set of segments; and
write, in response to the copy data request, the second set of segments and the second slice recipe to the object storage.
17. The computer program product of claim 16 , wherein the service is further configured to:
receive an object check request to determine whether the object is already stored on the object storage, the object check request including a first object recipe associated with the object; and
provide, in response to the object check request, an indication that the object is not already stored on the object storage in response to determining the first object recipe does not reference an existing object on the object storage.
18. The computer program product of claim 16 , wherein the service is further configured to:
receive a copy object recipe request to write an object recipe on the object storage, the copy object recipe request including the first object recipe updated to reference the second slice recipe; and
write, in response to the copy object recipe request, the updated first object recipe to the object storage.
19. The computer program product of claim 16 , wherein the second set of segments of the copy data request are received as compression regions, and writing the second set of segments includes writing the one or more compression regions to the object storage.
20. The computer program product of claim 16 , wherein the service is further configured to:
receive a request to establish a data transfer connection between the source site and the service, the request including a set of data transfer parameters including at least an access key; and
establish the data transfer connection between the source site and the service, in response to authenticating the access key.