Versatile data reduction for internet of things
One example method includes collaborative deduplication. A deduplication engine implemented at a cloud level collaborates or coordinates with an extension engine of the deduplication at an edge node. This allows data ingested at a node to be collaboratively deduplicated prior to transfer to the cloud and after transfer to the cloud.
1. A method for collaboratively deduplicating data, the method comprising:
receiving data from an edge device at an extension engine operating on an edge node, wherein the edge node has aggregated the data to be receive during a transfer window based on a list and the extension engine identifies files;
checking, by the extension engine, the data using a local catalog to determine files in the data that have been previously transmitted to a deduplication engine operating in a datacenter, wherein the local catalog includes metadata configured to determine that first files in the data have been previously sent to the deduplication engine and that second files in the data have not been sent to the deduplication engine based on the local catalog;
collaborating over a time period, by the extension engine and the deduplication engine, to identify third files from the second files that have not been deduplicated by the deduplication engine based on a global catalog accessible to the deduplication engine, wherein the global catalog associates data with hashes of deduplicated files;
transmitting the third files to the deduplication engine;
deduplicating, by the deduplication engine, the third files; and
updating the local catalog such that the local catalog reflects that the third files have been deduplicated by the deduplication engine.
2. The method of claim 1 , further comprising generating a list of the second files and transmitting the list to the deduplication engine.
3. The method of claim 2 , further comprising determining the third files from the list and the global catalog.
4. The method of claim 3 , further comprising instructing the extension engine to transmit the third files to the deduplication engine.
5. The method of claim 1 , further comprising deduplicating the third files by chunking the files, comparing hashes of the chunks with hashes stored in the global catalog, and storing new chunks in storage of a cloud.
6. The method of claim 1 , wherein checking the data using the local catalog includes deduplicating based on chunks having a larger size than chunks used by the deduplication engine.
7. The method of claim 1 , wherein the deduplication engine receives a list from multiple extension mechanisms at multiple edge nodes and each extension mechanism identifies third files, further comprising deduplicating all of the third files.
8. The method of claim 7 , further comprising updating each of the extension engines based on the list.
9. A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
receiving data from an edge device at an extension engine operating on an edge node, wherein the edge node has aggregated the data to be received during a transfer window based on a list and the extension engine identifies files;
checking, by the extension engine, the data using a local catalog to determine files in the data that have been previously transmitted to a deduplication engine operating in a datacenter, wherein the local catalog includes metadata configured to determine that first files in the data that have been previously sent to the deduplication engine and that second files in the data have not been sent to the deduplication engine based on the local catalog;
collaborating over a time period, by the extension engine and the deduplication engine identify third files from the second files that have not been deduplicated by the deduplication engine based on a global catalog accessible to the deduplication engine, wherein the global catalog associates data with hashes of deduplicated files;
transmitting the third files to the deduplication engine;
deduplicating, by the deduplication engine, the third files; and
updating the local catalog such that the local catalog reflects that the third files have been deduplicated by the deduplication engine.
10. The non-transitory storage medium of claim 9 , further comprising generating a list of the second files and transmitting the list to the deduplication engine.
11. The non-transitory storage medium of claim 10 , further comprising determining the third files from the list and the global catalog.
12. The non-transitory storage medium of claim 11 , further comprising instructing the extension engine to transmit the third files to the deduplication engine.
13. The non-transitory storage medium of claim 9 , further comprising deduplicating the third files by chunking the files, comparing hashes of the chunks with hashes stored in the global catalog, and storing new chunks in storage of a cloud.
14. The non-transitory storage medium of claim 9 , further comprising providing the deduplication engine with pointers to the first files and the second files that are not transmitted to the deduplication engine.
15. The non-transitory storage medium of claim 9 , wherein the deduplication engine receives a list from multiple extension mechanisms at multiple edge nodes and each extension mechanism identifies third files, further comprising deduplicating all of the third files.
16. The non-transitory storage medium of claim 15 , further comprising updating each of the extension engines based on the list.