DATA MOVEMENT AMONG DISTRIBUTED DATA CENTERS
Embodiments of the present disclosure provide a method and an apparatus for data movement among distributed data centers in a peer network. The method comprises: reducing an amount of data to be moved by pre-processing the data; generating a torrent file for the data; distributing the torrent file to a peer data center, and in response to receiving a data request from the peer data center, transmitting a segment of the data to the peer data center. Compared with the prior art, the embodiments of the present disclosure can realize reliable and fast data movement among distributed data centers over an unreliable and low-bandwidth network, can support an adaptive network topology among distributed data centers without complex pre-configuration or cumbersome runtime coordination, can support different types of data storage, and facilitates coordinated cloud computing and data aggregation among distributed data centers.
1 . A method for data movement among distributed data centers in a peer network, comprising:
reducing an amount of data to be moved by pre-processing the data;
generating a torrent file for the data;
distributing the torrent file to a peer data center; and
in response to receiving a data request from the peer data center, transmitting a segment of the data to the peer data center.
2 . The method according to claim 1 , wherein the pre-processing the data comprises:
de-duplicating and compressing the data.
3 . The method according to claim 1 , wherein the generating a torrent file for the data comprises:
segmenting the data into a plurality of segments of an equal length, the segment being included in the plurality of segments;
determining a hash value for the segment; and
writing an index and the hash value for the segment into the torrent file.
4 . The method according to claim 1 , wherein the torrent file includes configuration information related to the data, and the generating a torrent file for the data comprises:
generating the configuration information related to the data; and
writing the configuration information into the torrent file.
5 . The method according to claim 4 , wherein the generating the configuration information related to the data comprises:
determining one or more candidates for the configuration information;
determining a cost for each of the one or more candidates; and
selecting, from the one or more candidates, a candidate with the lowest cost as the configuration information related to the data.
6 . The method according to claim 1 , wherein the distributing the torrent file to a peer data center comprises:
distributing the torrent file to the peer data center via a torrent server with a reliable network protocol.
7 . The method according to claim 1 , wherein the data includes a file, and the transmitting a segment of the data to the peer data center comprises:
segmenting the file into a plurality of segments of an equal length, the segment being included in the plurality of segments; and
transmitting the segment of the file to the peer data center.
8 . The method according to claim 1 , wherein the data includes a data block, and the transmitting a segment of the data to the peer data center comprises:
transmitting the data block to the peer data center.
9 . The method according to claim 1 , wherein the data includes an object, and the transmitting a segment of the data to the peer data center comprises:
transmitting the object to the peer data center.
10 - 42 . (canceled)
43 . A system, comprising:
two or more of distributed data centers, wherein each distributed data center includes one or more data storage systems; and
computer-executable program logic in memory of one or more computers enabled to move data among the distributed data centers in a peer network, wherein the computer-executable program logic is configured for the execution of:
reducing an amount of data to be moved by pre-processing the data;
generating a torrent file for the data;
distributing the torrent file to a peer data center; and
in response to receiving a data request from the peer data center, transmitting a segment of the data to the peer data center.
44 . The system according to claim 43 , wherein pre-processing of the data comprises:
de-duplicating and compressing the data.
45 . The system according to claim 43 , wherein the generating a torrent file for the data comprises:
segmenting the data into a plurality of segments of an equal length, the segment being included in the plurality of segments;
determining a hash value for the segment; and
writing an index and the hash value for the segment into the torrent file.
46 . The system according to claim 43 , wherein the torrent file includes configuration information related to the data, and the generating a torrent file for the data comprises:
generating the configuration information related to the data; and
writing the configuration information into the torrent file.
47 . The system according to claim 46 , wherein the generating the configuration information related to the data comprises:
determining one or more candidates for the configuration information;
determining a cost for each of the one or more candidates; and
selecting, from the one or more candidates, a candidate with the lowest cost as the configuration information related to the data.
48 . A computer program product for moving data among distributed data centers in a peer network, wherein each of the distributed data centers includes one or more data storage systems, the computer program product comprising:
a non-transitory computer readable medium encoded with computer-executable code, the code configured to enable the execution of:
reducing an amount of data to be moved by pre-processing the data;
generating a torrent file for the data;
distributing the torrent file to a peer data center; and
in response to receiving a data request from the peer data center, transmitting a segment of the data to the peer data center.
49 . The computer program product according to claim 48 , wherein pre-processing of the data comprises:
de-duplicating and compressing the data.
50 . The computer program product according to claim 48 , wherein the generating a torrent file for the data comprises:
segmenting the data into a plurality of segments of an equal length, the segment being included in the plurality of segments;
determining a hash value for the segment; and
writing an index and the hash value for the segment into the torrent file.
51 . The computer program product according to claim 48 , wherein the torrent file includes configuration information related to the data, and the generating a torrent file for the data comprises:
generating the configuration information related to the data; and
writing the configuration information into the torrent file.
52 . The computer program product according to claim 51 , wherein the generating the configuration information related to the data comprises:
determining one or more candidates for the configuration information;
determining a cost for each of the one or more candidates; and
selecting, from the one or more candidates, a candidate with the lowest cost as the configuration information related to the data.