WAN optimized micro-service based deduplication
A system for managing an object storage includes frontend micro-services and backend micro-services. The frontend micro-services obtain a request to store data in an object storage; divide the data into slices; send a slice analysis request, based on a slice of the slices, to the backend micro-services; obtain, from the plurality of backend micro-services, a list of segments of the slice that are not stored in the object storage; and add a segment specified by the list of segments to a compression region. The backend micro-services identify segments of the slice specified by the slice analysis request that are not stored in the object storage and generate the list of segments of the slice based on the identified segments.
1. A system for managing an object storage, comprising:
a plurality of frontend micro-services programmed to:
obtain a request to store data in an object storage;
divide the data into slices;
send a slice analysis request, based on a slice of the slices, to a plurality of backend micro-services, wherein the slice analysis request comprises:
a sketch of the slice that does not uniquely identify the slice, and
a fingerprint of each segment of the slice that virtually uniquely identifies the respective segment;
obtain, from the plurality of backend micro-services, a list of segments of the slice that are not stored in the object storage;
add a segment specified by the list of segments to a compression region;
the plurality of backend micro-services programmed to:
identify segments of the slice specified by the slice analysis request that are not stored in the object storage using the sketch of the slice and the fingerprints; and
generate the list of segments of the slice based on the identified segments;
a load balancer programmed to:
obtain requests to store data in the object storage; and
distribute the requests to the plurality of frontend micro-services,
wherein the system dynamically adjusts a number of available frontend micro-services based on a number of pending requests in the load balancer.
2. The system of claim 1 , further comprising:
a request queue that passes slice analysis requests from the frontend micro-services to the backend micro-services,
wherein the system dynamically adjusts a number of available backend micro-services based on a number of pending slice analysis requests in the request queue.
3. The system of claim 1 , wherein the plurality of frontend micro-services is further programmed to:
store the compression region in the object storage,
wherein the object storage is a persistent storage.
4. The system of claim 1 , wherein the request to store data in the object storage is received from a computing device.
5. The system of claim 4 , wherein the plurality of backend micro-services executes on a cloud platform operably connected to the computing device.
6. The system of claim 5 , wherein the plurality of frontend micro-services does not execute on the cloud platform.
7. A method of managing an object storage, comprising:
obtaining, by a frontend micro-service, a request to store data in an object storage;
dividing, by the frontend micro-service, the data into slices;
sending, by the frontend micro-service, a slice analysis request to a plurality of backend micro-services, wherein the slice analysis request comprises:
a sketch of the slice that does not uniquely identify the slice, and
a fingerprint of each segment of the slice that virtually uniquely identifies the respective segment;
obtaining, by the frontend micro-service, a list of segments of the slice of the slices that are not stored in the object storage; and
adding, by the frontend micro-service, a segment specified by the list of segments to a compression region.
8. The method of claim 7 , wherein dividing the data into slices comprises:
dividing a portion of the data into a plurality of segments;
grouping a portion of the segments of the plurality of segments into the slice;
generating an object recipe that specifies, in part, the slice; and
storing the object recipe in an object recipe storage.
9. The method of claim 8 , wherein dividing the portion of the data into the plurality of segments comprises:
generating a rolling hash of the data;
obtaining a plurality of segment break points by comparing each hash of the rolling hash to a predetermined bit pattern; and
dividing the portion of the data based on the plurality of segment break points.
10. The method of claim 9 , wherein grouping the portion of the segments of the plurality of segments into the slice comprises:
determining a plurality of slice break points by comparing each hash of the rolling hash to a second predetermined bit pattern;
identifying the portion of the segments of the plurality of segments based on plurality of slice break points; and
grouping the segments of the identified portion of the segments of the plurality of segments into the slice,
wherein the second predetermined bit pattern has a length that is longer than a length of the predetermined bit pattern.
11. The method of claim 7 , further comprising:
storing, by the frontend micro-service, the compression region in the object storage,
wherein the object storage is a persistent storage.
12. The method of claim 7 , wherein the request to store data in the object storage is received from a computing device, wherein a plurality of frontend micro-services execute on the computing device, wherein the plurality of backend micro-services are not executing on the computing device.
13. The method of claim 8 , wherein the data is not sent to the plurality of backend micro-services.
14. A method of managing an object storage, comprising:
obtaining, by a backend micro-service, a slice analysis request, wherein the slice analysis request comprises:
a sketch of the slice of data that does not uniquely identify the slice, and
a fingerprint of each segment of a plurality of segments of the slice that virtually uniquely identifies the respective segment;
identifying, by the backend micro-service, segments of the plurality of segments that are not stored in the object storage using the sketch of the slice of the data and the fingerprint of each segment of the plurality of segments of the slice;
generating, by the backend micro-service, a list of segments that specifies each segment of the identified segments;
sending, by the backend micro-service, the list of segments to a client that sent the slice analysis request;
monitoring, by a system comprising a plurality of backend micro-services, a cardinality of pending slice analysis requests in a request queue that provides the slice analysis request to a plurality of backend micro-services including the backend micro-service; and
dynamically adjusting, by the system, a cardinality of the plurality of backend micro-services based on the monitoring of the cardinality of the pending slice analysis requests in the request queue.
15. The method of claim 14 , wherein identifying a first set of segments of the plurality of segments that are not stored in the object storage and a second set of segments of the plurality of segments that are stored in the object storage comprises:
matching the sketch of the slice to a similarity group of a plurality of similarity groups, wherein each similarity group of the plurality of similarity groups has a different sketch;
obtain meta-data, associated with the matched similarity group, that specifies a plurality of fingerprints stored in the object storage; and
identifying fingerprints, specified by the slice analysis request, that do not match any of the fingerprints of the plurality of fingerprint specified by the meta-data.
16. The method of claim 15 , further comprising:
adding each fingerprint of the first set of segments to the meta-data.
17. The method of claim 14 , wherein the slice analysis request is obtained from a computing device, wherein a plurality of frontend micro-services executing on the computing device receive the list of segments, wherein the plurality of backend micro-services are not executing on the computing device.