Micro-service based deduplication

1. A system for managing an object storage, comprising:

a physical device configured to execute first instructions adapted to cause a plurality of frontend micro-services to:

obtain a request to store data in an object storage;

divide the data into slices; and

a second physical device configured to execute second instructions adapted to cause a plurality of backend micro-services to:

generate a sketch of each slice;

match each slice to a similarity group of a plurality of similarity groups using the respective sketch of each slice;

obtain meta-data associated with each matched similarity group;

add at least a portion of a slice of the slices to a compression region using the meta-data,

wherein the sketch of each slice comprises a bit string that does not uniquely identify the slice.

2. The system of claim 1 , further comprising:

a request queue that passes the slices from the frontend micro-services to the backend micro-services,

wherein a number of available backend micro-services is dynamically adjusted based on a number of pending slices in the request queue.

3. The system of claim 1 , further comprising:

a load balancer to:

obtain requests to store data in the object storage;

distribute the requests to the plurality of frontend micro-services,

wherein the system dynamically adjusts a number of available frontend micro-services based on a number of pending requests in the load balancer.

4. The system of claim 1 , wherein dividing the data into slices comprises:

dividing a portion of the data into a plurality of segments;

grouping a portion of the segments of the plurality of segments into a slice;

generating an object recipe that specifies, in part, the slice; and

storing the object recipe in an object recipe storage.

5. The system of claim 4 , wherein dividing the portion of the data into the plurality of segments comprises:

generating a rolling hash of the data;

obtaining a plurality of segment break points by comparing each hash of the rolling hash to a predetermined bit pattern; and

dividing the data based on the plurality of segment break points.

6. The system of claim 5 , wherein grouping the portion of the segments of the plurality of segments into the slice comprises:

determining a plurality of slice break points by comparing each hash of the rolling hash to a second predetermined bit pattern;

identifying the portion of the segments of the plurality of segments based on plurality of slice break points; and

grouping the segments of the identified portion of the segments of the plurality of segments into the slice,

wherein a second predetermined bit pattern has a length that is longer than a length of the predetermined bit pattern.

7. The system of claim 1 , wherein storing at least the portion of the slice of the slices in the compression region using the meta-data comprises:

identifying a portion of the segments of the slice that each have a fingerprint that does not match any fingerprint specified by the meta-data;

adding a segment of the identified portion of the segments of the slice to the compression region;

updating the meta-data based on the identified portion of the segments;

generating a slice recipe that specifies:

the matched similarity group, and

each fingerprint of each segment of the slice;

storing the slice recipe in a slice recipe storage.

8. The system of claim 7 , wherein the fingerprint of each segment comprises a bit string that virtually uniquely identifies the segment.

9. The system of claim 8 , wherein the fingerprint of each segment is obtained by performing a secure hash of the segment.

10. The system of claim 1 , wherein the plurality of backend services is further to:

store the compression region in the object storage,

wherein the object storage is a persistent storage.

11. The system of claim 1 , wherein the request to store data in the object storage is received from a computing device.

12. The system of claim 11 , wherein the plurality of frontend micro-services executes on a cloud platform operably connected to the computing device.

13. The system of claim 12 , wherein the plurality of backend micro-services executes on the cloud platform.

14. A method of managing an object storage, comprising:

obtaining, by a frontend micro-service of a plurality of frontend micro-services, a request to store data in an object storage;

dividing, by the frontend micro-service, the data into slices;

generating, by a backend micro-service of a plurality of backend micro-services, a sketch of each slice;

matching, by the backend micro-service, each slice to a similarity group of a plurality of similarity groups using the respective sketch of each slice;

obtaining, by the backend micro-service, meta-data associated with each matched similarity group; and

adding, by the backend micro-service, at least a portion of a slice of the slices to a compression region using the meta-data,

wherein the sketch of each slice comprises a bit string that does not uniquely identify the slice.

15. The method of claim 14 , further comprising:

monitoring a number of pending slices in a request queue that passes the slices from the plurality of frontend micro-services to the plurality of backend micro-services; and

dynamically adjusting a number of backend micro-services of the plurality of backend micro-services based on the monitoring the number of pending slices in the request queue.

16. The method of claim 14 , further comprising:

distributing, by a load balancer, requests to store data in the object storage to the plurality of frontend micro-services;

monitoring, by the load balancer, a number of requests of the requests that are queued to be distributed; and

dynamically adjusting a number of available frontend micro-services based on the monitoring of the number of requests of the requests that are queued to be distributed.

17. A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method for managing an object storage, the method comprising:

obtaining, by a frontend micro-service of a plurality of frontend micro-services, a request to store data in an object storage;

dividing, by the frontend micro-service, the data into slices;

generating, by a backend micro-service of a plurality of backend micro-services, a sketch of each slice;

matching, by the backend micro-service, each slice to a similarity group of a plurality of similarity groups using the respective sketch of each slice;

obtaining, by the backend micro-service, meta-data associated with each matched similarity group; and

adding, by the backend micro-service, at least a portion of a slice of the slices to a compression region using the meta-data,

wherein the sketch of each slice comprises a bit string that does not uniquely identify the slice.

18. The non-transitory computer readable medium of claim 17 , wherein the method further comprises:

monitoring a number of pending slices in a request queue that passes the slices from the plurality of frontend micro-services to the plurality of backend micro-services; and

dynamically adjusting a number of backend micro-services of the plurality of backend micro-services based on the monitoring the number of pending slices in the request queue.

19. The non-transitory computer readable medium of claim 17 , further comprising:

distributing, by a load balancer, requests to store data in the object storage to the plurality of frontend micro-services;

monitoring, by the load balancer, a number of requests of the requests that are queued to be distributed; and

dynamically adjusting a number of frontend micro-services of the plurality of frontend micro-services based on the monitoring of the number of requests of the requests that are queued to be distributed.