Inter-cloud shared content data management
Methods, systems, and computer program products for multi-cloud shared data management. A first storage facility on a first cloud is established and a second storage facility on a second cloud established. The first cloud and the second cloud comprise at least respective first deduplication metadata and second deduplication metadata. In response to receipt of a request to store a content object comprising at least one candidate deduplication chunk, an inter-cloud deduplication protocol is initiated. The inter-cloud deduplication protocol determines whether or not a candidate deduplication chunk of a data item is already stored at either the first cloud or the second cloud. In the event that the candidate deduplication chunk of the data item deemed to have been already stored then, rather than storing a further copy of the bit sequence corresponding to the candidate deduplication chunk, instead, merely referring to the location of the already stored candidate deduplication chunk.
1 . A non-transitory computer readable medium having stored thereon a sequence of instructions which, when stored in memory and executed by a processor cause the processor to perform acts comprising:
receiving a request to store a content object on a first storage facility on a first cloud of a data warehouse spanning the first storage facility on the first cloud and a second storage facility on a second cloud, wherein the first cloud comprises at least a first controller, the first storage facility, and first deduplication metadata, the request to store the content object is received by the first controller, the content object comprises at least one candidate deduplication chunk. and the second cloud comprises at least a second controller, the second storage facility, and second deduplication metadata;
responsive to receipt of the request to store the content object comprising the at least one candidate deduplication chunk:
transmitting, by the first controller on the first cloud, a first message to the second controller on the second cloud as part of an inter-cloud deduplication protocol, wherein the inter-cloud deduplication protocol exchanges one or more messages between the first controller on the first cloud and the second controller on the second cloud to determine whether a candidate deduplication chunk of a data item is already stored in the second storage facility on the second cloud;
receiving, at the first controller on the first cloud, a second message from the second controller on the second cloud as part of the inter-cloud deduplication protocol, wherein the second message indicates that the candidate deduplication chunk of the data item is already stored in the second storage facility on the second cloud; and
storing, by the first controller on the first cloud, a reference to a second location at the second public cloud as an already stored location of a bit sequence corresponding to the candidate deduplication chunk in response to the second message from the second controller on the second cloud.
2 . The non-transitory computer readable medium of claim 1 , wherein the acts further comprise:
responsive to receipt of a request at the first controller to store a second content object comprising at least one second candidate deduplication chunk:
transmitting, by the first controller on the first cloud, a third message to the second controller on the second cloud as part of the inter-cloud deduplication protocol to determine whether a second candidate deduplication chunk of a second data item is already stored in the second storage facility on the second public cloud; and
wherein a fourth message is sent to a third controller to determine whether the second candidate deduplication chunk of the second data item is already stored in a third storage facility on the third cloud, the data warehouse further spans the third storage facility on the third cloud, the third cloud comprises at least a third controller, the third storage facility, and third deduplication metadata.
3 . The non-transitory computer readable medium of claim 2 , wherein the fourth message is sent to the third controller on the third cloud by the second controller on the second cloud.
4 . The non-transitory computer readable medium of claim 2 , wherein the fourth message is sent to the third controller on the third cloud using a protocol different from the inter-cloud deduplication protocol.
5 . The non-transitory computer readable medium of claim 1 , further comprising instructions which, when stored in memory and executed by the processor cause the processor to perform further acts of generating a map entry in data item metadata to map a data unit to a physical storage location, wherein the data unit is a deduplicated data unit in deduplication metadata, the data item metadata and the deduplication metadata are different data structures, and the data item metadata and the deduplication metadata are distributed over a plurality of computing nodes.
6 . The non-transitory computer readable medium of claim 5 , further comprising instructions which, when stored in memory and executed by the processor cause the processor to perform further acts of performing a scan of the data item metadata for the data unit, wherein an in-use state is set in the deduplication metadata for the data unit when the map entry is detected for the data unit in the data item metadata, a not-in-use state and a deletion state are set for the data unit in the deduplication metadata when the map entry is not detected in the data item metadata for the data unit, and the data unit is deleted from the physical storage location based at least in part on a result of the scan when the data unit is determined to correspond to the not-in-use state and the deletion state.
7 . The non-transitory computer readable medium of claim 1 , wherein deduplication of the candidate deduplication chunk of a data item is managed without implementing a reference count value of the candidate deduplication chunk and the first storage facility on the first cloud and the second storage facility on the second cloud are blob storage facilities.
8 . The non-transitory computer readable medium of claim 1 , wherein the data warehouse was created using a graphical user interface having at least a first field for identifying the first cloud, a second field for identifying the second cloud, and a third field for identifying a plug-in for implementing the inter-cloud deduplication protocol.
9 . A method comprising:
receiving a request to store a content object on a first storage facility on a first cloud of a data warehouse spanning the first storage facility on the first cloud and a second storage facility on a second cloud, wherein the first cloud comprises at least a first controller, the first storage facility, and first deduplication metadata, the request to store the content object is received by the first controller, the content object comprises at least one candidate deduplication chunk, and the second cloud comprises at least a second controller, the second storage facility, and second deduplication metadata;
responsive to receipt of the request to store the content object comprising the at least one candidate deduplication chunk:
transmitting, by the first controller on the first cloud, a first message to the second controller on the second cloud as part of an inter-cloud deduplication protocol, wherein the inter-cloud deduplication protocol exchanges one or more messages between the first controller on the first cloud and the second controller on the second cloud to determine whether a candidate deduplication chunk of a data item is already stored in the second storage facility on the second cloud;
receiving, at the first controller on the first cloud, a second message from the second controller on the second cloud as part of the inter-cloud deduplication protocol, wherein the second message indicates that the candidate deduplication chunk of the data item is already stored in the second storage facility on the second cloud; and
storing, by the first controller on the first cloud, a reference to a second location at the second cloud as an already stored location of a bit sequence corresponding to the candidate deduplication chunk in response to the second message from the second controller on the second cloud.
10 . The method of claim 9 , further comprising:
responsive to receipt of a request at the first controller to store a second content object comprising at least one second candidate deduplication chunk:
transmitting, by the first controller on the first cloud, a third message to the second controller on the second cloud as part of the inter-cloud deduplication protocol to determine whether a second candidate deduplication chunk of a second data item is already stored in the second storage facility on the second cloud; and
wherein a fourth message is sent to a third controller on a third cloud to determine whether the second candidate deduplication chunk of the second data item is already stored in a third storage facility on the third cloud, the data warehouse further spans the third storage facility on the third cloud, the third cloud comprises at least a third controller, the third storage facility, and third deduplication metadata.
11 . The method of claim 10 , wherein the fourth message is sent to the third controller on the third cloud by the second controller on the second cloud.
12 . The method of claim 10 , wherein the fourth message is sent to the third controller on the third cloud using a protocol different from the inter-cloud deduplication protocol.
13 . The method of claim 9 , further comprising generating a map entry in data item metadata to map a data unit to a physical storage location, wherein the data unit is a deduplicated data unit in deduplication metadata, and wherein the data item metadata and the deduplication metadata are different data structures, and the data item metadata and the deduplication metadata are distributed over a plurality of computing nodes.
14 . The method of claim 13 , further comprising performing a scan of the data item metadata for the data unit, wherein an in-use state is set in the deduplication metadata for the data unit when the map entry is detected for the data unit in the data item metadata, a not-in-use state and a deletion state are set for the data unit in the deduplication metadata when the map entry is not detected in the data item metadata for the data unit, and the data unit is deleted from the physical storage location based at least in part on a result of the scan when the data unit is determined to correspond to the not-in-use state and the deletion state.
15 . The method of claim 9 , wherein deduplication of the candidate deduplication chunk of a data item is managed without implementing a reference count value of the candidate deduplication chunk and the first storage facility on the first cloud and the second storage facility on the second cloud are blob storage facilities.
16 . The method of claim 9 , wherein the data warehouse was created using a graphical user interface having at least a first field for identifying the first cloud, a second field for identifying the second cloud, and a third field for identifying a plug-in for implementing the inter-cloud deduplication protocol.
17 . A system comprising:
a storage medium having stored thereon a sequence of instructions; and
a processor that executes the sequence of instructions to cause the processor to perform acts comprising,
receiving a request to store a content object on a first storage facility on a first cloud of a data warehouse spanning the first storage facility on the first cloud and a second storage facility on a second cloud, wherein the first cloud comprises at least a first controller, the first storage facility, and first deduplication metadata, the request to store the content object is received by the first controller, the content object comprises at least one candidate deduplication chunk, and the second cloud comprises at least a second controller, the second storage facility, and second deduplication metadata;
responsive to receipt of the request to store the content object comprising the at least one candidate deduplication chunk:
transmitting, by the first controller on the first cloud, a first message to the second controller on the second cloud as part of initiating an inter-cloud deduplication protocol, wherein the inter-cloud deduplication protocol exchanges one or more messages between the first controller on the first public cloud and the second controller the second storage facility on the second publie-cloud to determine whether a candidate deduplication chunk of a data item is already stored in the second storage facility on the second cloud;
receiving, at the first controller on the first cloud, a second message from the second controller on the second cloud as part of the inter-cloud deduplication protocol, wherein the second message indicates that the candidate deduplication chunk of the data item is already stored in the second storage facility on the second cloud; and
storing, by the first controller on the first cloud, a reference to a second location at the second publie-cloud as an already stored location of a bit sequence corresponding to the candidate deduplication chunk in response to the second message from the second controller on the second cloud.
18 . The system of claim 17 , wherein the acts further comprise:
responsive to receipt of a request at the first controller to store a second content object comprising at least one second candidate deduplication chunk:
transmitting, by the first controller on the first cloud, a third message to the second controller on the second cloud as part of the inter-cloud deduplication protocol to determine whether a second candidate deduplication chunk of a second data item is already stored in the second storage facility on the second cloud; and
wherein a fourth message is sent to a third controller on a third cloud to determine whether the second candidate deduplication chunk of the second data item is already stored in a third storage facility on the third cloud, the data warehouse further spans the third storage facility on the third cloud, the third cloud comprises at least a third controller, the third storage facility, and third deduplication metadata.
19 . The system of claim 18 , wherein the fourth message is sent to the third controller on the third cloud by the second controller on the second cloud.
20 . The system of claim 18 , wherein the fourth message is sent to the third controller on the third cloud using a protocol different from the inter-cloud deduplication protocol.
21 . The system of claim 17 , further comprising instructions which, when stored in memory and executed by the processor cause the processor to perform further acts of generating a map entry in data item metadata to map a data unit to a physical storage location, wherein the data unit is a deduplicated data unit in deduplication metadata, and wherein the data item metadata and the deduplication metadata are different data structures, and the data item metadata and the deduplication metadata are distributed over a plurality of computing nodes.
22 . The system of claim 21 , further comprising instructions which, when stored in memory and executed by the processor cause the processor to perform further acts of performing a scan of the data item metadata for the data unit, wherein an in-use state is set in the deduplication metadata for the data unit when the map entry is detected for the data unit in the data item metadata, a not-in-use state and a deletion state are set for the data unit in the deduplication metadata when the map entry is not detected in the data item metadata for the data unit, and the data unit is deleted from the physical storage location based at least in part on a result of the scan when the data unit is determined to correspond to the not-in-use state and the deletion state.
23 . The system of claim 17 , wherein deduplication of the candidate deduplication chunk of a data item is managed without implementing a reference count value of the candidate deduplication chunk and the first storage facility on the first cloud and the second storage facility on the second cloud are blob storage facilities.
24 . The system of claim 17 , wherein the data warehouse was created using a graphical user interface having at least a first field for identifying the first cloud, a second field for identifying the second cloud, and a third field for identifying a plug-in for implementing the inter-cloud deduplication protocol.