Methods and systems for serverless data deduplication
Methods and systems for serverless data deduplication are disclosed. A blob of data is received at a cloud services platform, where the blob of data includes incremental data. The blob of data is used to create an object in a first object store included in the cloud services platform. A function as a service (FaaS) function is triggered when the object is created. The FaaS function deduplicates the object to generate a deduplicated object. The deduplicated object is stored in a second object store included in the cloud services platform.
1. A computer-implemented method for function as a service (FaaS) based data deduplication, the method comprising:
receiving a blob of data as a set of cloud-based objects in a cloud services platform, wherein the blob of data includes incremental data;
for each object in the set of cloud-based objects,
inserting the object into a first cloud-based object store included in the cloud services platform,
triggering, in the cloud services platform, a FaaS-based data deduplication function in response to the insertion of the object, to deduplicate the object in the cloud services platform,
deduplicating the object in the cloud services platform to generate a deduplicated object, and
storing the deduplicated object in a second cloud-based object store included in the cloud services platform;
wherein deduplicating the object in the cloud services platform comprises:
dividing the object into a plurality of data pieces,
for each of the data pieces,
creating a corresponding hash value for the data piece,
visiting a cloud-based database service of the cloud services platform to obtain the hash value, as a key, and an object identifier (ID) that uniquely identifies data corresponding to the hash value,
determining whether the hash value is in existence in the cloud-based database service,
in response to determining that the hash value is in existence in the cloud-based database service, replacing the corresponding data piece with a pointer to a data location of the data piece,
otherwise in response to determining that the hash value is not in existence in the cloud-based database service, preserving the data piece and adding the hash value to a key value pair with an object ID of the deduplicated object, wherein the preserved data piece is included in the deduplicated object.
2. The method of claim 1 , wherein the set of cloud-based objects comprises Amazon Web Services (AWS) S3 objects.
3. The method of claim 2 , wherein the data pieces are of variable length.
4. The method of claim 2 , wherein the first and second cloud-based object stores are first and second AWS S3 buckets, respectively.
5. The method of claim 1 , further comprising collecting by a backup agent the blob of data at a production site, wherein the backup agent is disposed on a data path between a virtual machine and a data store included in the production site.
6. The method of claim 1 , further comprising deleting the created object in the first cloud-based object store.
7. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations, the operations comprising:
receiving a blob of data as a set of cloud-based objects in a cloud services platform, wherein the blob of data includes incremental data;
for each object in the set of cloud-based objects,
inserting the object into a first cloud-based object store included in the cloud services platform,
triggering, in the cloud services platform, a function as a service (FaaS) based data deduplication function in response to the insertion of the object, to deduplicate the object in the cloud services platform,
deduplicating the object in the cloud services platform to generate a deduplicated object, and
storing the deduplicated object in a second cloud-based object store included in the cloud services platform;
wherein deduplicating the object in the cloud services platform comprises:
dividing the object into a plurality of data pieces,
for each of the data pieces,
creating a corresponding hash value for the data piece,
visiting a cloud-based database service of the cloud services platform to obtain the hash value, as a key, and an object identifier (ID) that uniquely identifies data corresponding to the hash value,
determining whether the hash value is in existence in the cloud-based database service,
in response to determining that the hash value is in existence in the cloud-based database service, replacing the corresponding data piece with a pointer to a data location of the data piece,
otherwise in response to determining that the hash value is not in existence in the cloud-based database service, preserving the data piece and adding the hash value to a key value pair with an object ID of the deduplicated object, wherein the preserved data piece is included in the deduplicated object.
8. The non-transitory machine-readable medium of claim 7 , wherein the set of cloud-based objects comprises Amazon Web Services (AWS) S3 objects.
9. The non-transitory machine-readable medium of claim 8 , wherein the data pieces are of variable length.
10. The non-transitory machine-readable medium of claim 8 , wherein the first and second cloud-based object stores are first and second AWS S3 buckets, respectively.
11. The non-transitory machine-readable medium of claim 7 , wherein the operations further comprise collecting by a backup agent the blob of data at a production site, wherein the backup agent is disposed on a data path between a virtual machine and a data store included in the production site.
12. The non-transitory machine-readable medium of claim 7 , wherein the operations further comprise deleting the created object in the first cloud-based object store.
13. A data processing system, comprising:
a processor; and
a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations, the operations including:
receiving a blob of data as a set of cloud-based objects in a cloud services platform, wherein the blob of data includes incremental data;
for each object in the set of cloud-based objects,
inserting the object into a first cloud-based object store included in the cloud services platform,
triggering, in the cloud services platform, a function as a service (FaaS) based data deduplication function in response to the insertion of the object, to deduplicate the object in the cloud services platform,
deduplicating the object in the cloud services platform to generate a deduplicated object, and
storing the deduplicated object in a second cloud-based object store included in the cloud services platform;
wherein deduplicating the object in the cloud services platform comprises:
dividing the object into a plurality of data pieces,
for each of the data pieces,
creating a corresponding hash value for the data piece,
visiting a cloud-based database service of the cloud services platform to obtain the hash value, as a key, and an object identifier (ID) that uniquely identifies data corresponding to the hash value,
determining whether the hash value is in existence in the cloud-based database service,
in response to determining that the hash value is in existence in the cloud-based database service, replacing the corresponding data piece with a pointer to a data location of the data piece,
otherwise in response to determining that the hash value is not in existence in the cloud-based database service, preserving the data piece and adding the hash value to a key value pair with an object ID of the deduplicated object, wherein the preserved data piece is included in the deduplicated object.
14. The data processing system of claim 13 , wherein the set of cloud-based objects comprises Amazon Web Services (AWS) S3 objects.
15. The data processing system of claim 14 , wherein the data pieces are of variable length.
16. The data processing system of claim 14 , wherein the first and second cloud-based object stores are first and second AWS S3 buckets, respectively.
17. The data processing system of claim 13 , wherein the operations further include collecting by a backup agent the blob of data at a production site, wherein the backup agent is disposed on a data path between a virtual machine and a data store included in the production site.
18. The data processing system of claim 13 , wherein the operations further include deleting the created object in the first cloud-based object store.
19. A computer-implemented method of restoring an object at a production site, the method comprising:
in response to a request to restore an object or data pieces associated with the object, triggering a function as a service (FaaS) based data restore function to read the object in a cloud services platform;
retrieving a key value pair including a hash value and an object identifier (ID) by visiting a cloud-based database service of the cloud services platform to search and obtain the key value pair based on a key value pointer;
using the hash value and the object ID to identify and fetch data associated with the read object, wherein the object ID identifies a deduplicated object that includes the data and the hash value identifies the data; and
rebuilding the read object based on the fetched data;
wherein the object is deduplicated in the cloud services platform by:
dividing the object into a plurality of data pieces,
for each of the data pieces,
creating a corresponding hash value for the data piece,
visiting the cloud-based database service of the cloud services platform to obtain the hash value, as a key, and an object ID that uniquely identifies data corresponding to the hash value,
determining whether the hash value is in existence in the cloud-based database service,
in response to determining that the hash value is in existence in the cloud-based database service, replacing the corresponding data piece with a pointer to a data location of the data piece,
otherwise in response to determining that the hash value is not in existence in the cloud-based database service, preserving the data piece and adding the hash value to a key value pair with an object ID of the deduplicated object, wherein the preserved data piece is included in the deduplicated object.