Data deduplication techniques for distributed computing environments
Methods, apparatus, and processor-readable storage media for deduplicating data in distributed computing environments are provided herein. An example computer-implemented method includes obtaining, by a node of a distributed computing system, a set of data from an edge device, and identifying a data type corresponding to the set of data from among a plurality of data types. The method includes using information corresponding to the identified data type to determine whether one or more portions of data in the set of data were previously uploaded to at least one server of the distributed computing system. The method also includes, in response to determining that the one or more portions of the data were previously uploaded, updating the set of data by removing the one or more portions of the data from the set of data, and uploading the updated set of data to the at least one server.
1 . A computer-implemented method comprising:
obtaining, by a node of a distributed computing system, a set of data from at least one processing device;
identifying a data type corresponding to the set of data from among a plurality of data types, wherein the plurality of data types comprises two or more of: at least one unstructured data type, at least one structured data type, and at least one semi-structured data type;
using a first set of information corresponding to the identified data type to determine whether one or more portions of data in the set of data were previously uploaded to at least one server of the distributed computing system;
in response to determining that the one or more portions of the data were previously uploaded, updating the set of data by removing the one or more portions of the data from the set of data; and
uploading the updated set of data to the at least one server;
wherein the method is performed by at least one processing device comprising a processor coupled to a memory.
2 . The computer-implemented method of claim 1 , wherein the node comprises at least one second set of information corresponding to a different one of the plurality of data types.
3 . The computer-implemented method of claim 2 , further comprising:
using the at least one second set of information to determine whether portions of at least one additional set of data were previously uploaded to the at least one server of the distributed computing system.
4 . The computer-implemented method of claim 3 , wherein:
the portions of the set of data that were previously uploaded are stored by the at least one server using a first storage technology associated with the identified data type; and
the portions of the at least one additional set of data that were previously uploaded are stored by the at least one server using a different, second storage technology associated with the data type corresponding to the at least one additional set of data.
5 . The computer-implemented method of claim 1 , further comprising:
obtaining information for updating the first set of information from the at least one server in response to uploading the updated set of data.
6 . The computer-implemented method of claim 1 , wherein:
the at least one server maintains a global set of information for a plurality of nodes of the distributed computing system, wherein the global set of information comprises hash values that uniquely identify portions of data uploaded by one or more of the plurality of nodes and information indicating the data type of the portions of data uploaded by the one or more nodes.
7 . The computer-implemented method of claim 6 , wherein:
the first set of information comprises hashing functions corresponding to the global set of information for portions of the set of data that were previously uploaded to the at least one server of the distributed computing system.
8 . The computer-implemented method of claim 1 , wherein:
the node is assigned to one of a plurality of clusters of nodes of the distributed computing system, wherein the plurality of clusters of nodes is determined based at least in part on a similarity coefficient.
9 . The computer-implemented method of claim 8 , wherein the first set of information is distributed across the nodes in the cluster of nodes that the node is assigned to.
10 . The computer-implemented method of claim 1 , wherein:
the first set of information is distributed across one of a plurality of clusters of nodes of the distributed computing system, wherein the plurality of clusters of nodes is determined based at least in part on a similarity coefficient.
11 . A non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes the at least one processing device:
to obtain, by a node of a distributed computing system, a set of data from at least one processing device;
to identify a data type corresponding to the set of data from among a plurality of data types, wherein the plurality of data types comprises two or more of: at least one unstructured data type, at least one structured data type, and at least one semi-structured data type;
to use a first set of information corresponding to the identified data type to determine whether one or more portions of data in the set of data were previously uploaded to at least one server of the distributed computing system;
in response to determining that the one or more portions of the data were previously uploaded, to update the set of data by removing the one or more portions of the data from the set of data; and
to upload the updated set of data to the at least one server.
12 . The non-transitory processor-readable storage medium of claim 11 , wherein the node comprises at least one second set of information corresponding to a different one of the plurality of data types.
13 . The non-transitory processor-readable storage medium of claim 12 , wherein the program code when executed by the at least one processing device causes the at least one processing device:
to use the at least one second set of information to determine whether portions of at least one additional set of data were previously uploaded to the at least one server of the distributed computing system.
14 . The non-transitory processor-readable storage medium of claim 13 , wherein:
the portions of the set of data that were previously uploaded are stored by the at least one server using a first storage technology associated with the identified data type; and
the portions of the at least one additional set of data that were previously uploaded are stored by the at least one server using a different, second storage technology associated with the data type corresponding to the at least one additional set of data.
15 . An apparatus comprising:
at least one processing device comprising a processor coupled to a memory;
the at least one processing device being configured:
to obtain, by a node of a distributed computing system, a set of data from at least one processing device;
to identify a data type corresponding to the set of data from among a plurality of data types, wherein the plurality of data types comprises two or more of: at least one unstructured data type, at least one structured data type, and at least one semi-structured data type;
to use a first set of information corresponding to the identified data type to determine whether one or more portions of data in the set of data were previously uploaded to at least one server of the distributed computing system;
in response to determining that the one or more portions of the data were previously uploaded, to update the set of data by removing the one or more portions of the data from the set of data; and
to upload the updated set of data to the at least one server.
16 . The apparatus of claim 15 , wherein the node comprises at least one second set of information corresponding to a different one of the plurality of data types.
17 . The apparatus of claim 16 , wherein the at least one processing device is further configured:
to use the at least one second set of information to determine whether portions of at least one additional set of data were previously uploaded to the at least one server of the distributed computing system.
18 . The apparatus of claim 15 , wherein the at least one processing device is further configured:
to obtain information for updating the first set of information from the at least one server in response to uploading the updated set of data.
19 . The apparatus of claim 15 , wherein:
the at least one server maintains a global set of information for a plurality of nodes of the distributed computing system, wherein the global set of information comprises hash values that uniquely identify portions of data uploaded by one or more of the plurality of nodes and information indicating the data type of the portions of data uploaded by the one or more nodes.
20 . The apparatus of claim 15 , wherein:
the node is assigned to one of a plurality of clusters of nodes of the distributed computing system, wherein the plurality of clusters of nodes is determined based at least in part on a similarity coefficient.