Method, device, and computer program product for data migration
An example data migration includes, based on historical access data for objects, determining access patterns for the objects, and, based on the access patterns, determining access frequencies for the objects in a full replication stage. Based on the access frequencies for the objects, full replication is carried out on the objects, and, in response to completion of the full replication of the objects, incremental replication is carried out on at least part of the objects to complete data migration. In this way, the access frequencies for the objects can be predicted more accurately, then the objects with low access frequencies are preferentially replicated in the full replication stage, and the workload of incremental replication is reduced, thereby improving the efficiency and quality of data migration, reducing the probability of migration failure, avoiding unnecessary data transmission, and effectively reducing the demands for network bandwidth and data storage.
1 . A method, comprising:
based on historical access data for objects, determining, by a computing system comprising at least one processor, an access pattern for an object of the objects;
based on the access pattern, predicting, by the computing system, an access frequency of the object during a duration of a future replication of the objects, wherein the replication comprises a baseline replication phase and an incremental replication phase performed after the baseline replication phase;
based on the access frequency, determining, by the computing system, a likelihood that the object will be accessed by a client system during the baseline replication phase;
based on the likelihood being determined to be above a likelihood threshold, determining, by the computing system, to replicate the object from source equipment to destination equipment during the incremental replication phase; during the incremental replication phase of the object:
determining, by the computing system, that a stub representative of the object exists at the destination equipment, resulting in a stub determination, wherein the stub was stored at the destination equipment during a previous replication of the object, based on the object not yet having been replicated to the destination equipment,
based on the stub determination, replicating, by the computing system, a first part of the object that was not replicated during the baseline replication phase, and
replicating, by the computing system, a second part of the object, wherein the second part comprises a changed part of the object that was changed after the first part was determined to be replicated; and
determining, by the computing system, which of the baseline replication phase or the incremental replication phase is to be used to migrate the object, comprising:
in response to the access frequency being predicted to be greater than a preset value, determining that the object is hot data to be migrated during the incremental replication phase, or
in response to the access frequency being predicted to be less than or equal to the preset value, determining that the object is cold data to be migrated during the baseline replication phase.
2 . The method of claim 1 , wherein the access pattern comprises at least one of a trend pattern, a cyclical pattern, or an irregular pattern.
3 . The method of claim 2 , wherein, the predicting of the access frequency comprises:
in response to the access pattern being determined to be the trend pattern, predicting the access frequency based on a trend function.
4 . The method of claim 3 , wherein the predicting of the access frequency comprises:
based on the trend function and the duration of the future replication of the objects, predicting the access frequency at time points within the duration of the future replication of the objects; and
based on the predicting of the access frequency at the time points, predicting an average access frequency for the object for the duration of the future replication of the objects, wherein the likelihood that the object will be accessed during the baseline replication phase is further based on the average access frequency.
5 . The method of claim 2 , wherein the predicting of the access frequency comprises:
in response to the access pattern being determined to be the cyclical pattern, predicting the access frequency based on a cyclical function.
6 . The method of claim 5 , wherein the predicting of the access frequency further comprises:
based on the predicting of the access frequency based on the cyclical function, determining a maximum value and a minimum value of access frequencies of the object in the historical access data, wherein the likelihood that the object will be accessed during the baseline replication phase is further based on an average of the maximum value and the minimum value.
7 . The method of claim 2 , wherein the predicting of the access frequency further comprises:
in response to the access pattern being determined to be the irregular pattern, determining an average of access frequencies of the object in the historical access data, wherein the likelihood that the object will be accessed during the baseline replication phase is further based on the average of the access frequencies.
8 . The method of claim 1 , wherein, in response to the object being determined to be the cold data, arranging the object to be replicated before a different object determined to be the hot data.
9 . The method of claim 1 , wherein the object comprises a folder, and wherein the predicting of the access frequency of the object comprises:
traversing the folder layer by layer to obtain folder historical access data for at least one file in the folder;
based on the folder historical access data, determining an access pattern of the file; and
in response to at least one of:
the access pattern being determined to be a recognized access pattern, or
a number of traversed layers of the folder being determined to have reached a threshold, stopping the traversing.
10 . A device, comprising:
at least one processor; and
a storage apparatus coupled to the at least one processor and having instructions stored thereon, wherein the instructions, when executed by the at least one processor, cause the device to perform actions comprising:
based on historical access data for objects, determining an access pattern for an object of the objects;
based on the access pattern, predicting an access frequency of the object during a duration of a replication of the objects, wherein the replication comprises a baseline replication phase and an incremental replication phase performed after the baseline replication phase;
based on the access frequency, determining a likelihood that the object will be accessed by a client system during the baseline replication phase;
based on the likelihood being determined to be above a likelihood threshold, determining to replicate the object from source equipment to destination equipment during the incremental replication phase;
during the incremental replication phase of the object:
determining that a stub representative of the object exists at the destination equipment, resulting in a stub determination, wherein the stub was stored at the destination equipment during a previous replication of the object, based on the object not yet having been replicated to the destination equipment,
based on the stub determination, replicating a first part of the object that was not replicated during the baseline replication phase, and
replicating a second part of the object, wherein the second part comprises a changed part of the object that was changed after the first part was determined to be replicated; and
determining whether the baseline replication phase or the incremental replication phase is to be used to migrate the object, comprising:
in response to the access frequency being predicted to be greater than a preset value, determining that the object is hot data to be migrated during the incremental replication phase, or
in response to the access frequency being predicted to be less than or equal to the preset value, determining that the object is cold data to be migrated during the baseline replication phase.
11 . The device of claim 10 , wherein the respective access patterns comprise at least one of a trend pattern, a cyclical pattern, or an irregular pattern.
12 . The device of claim 11 , wherein the actions further comprise, in response to an access pattern of the respective access patterns being determined to be the trend pattern, employing a trend function to predict an access frequency based on the access pattern.
13 . The device of claim 12 , wherein the actions further comprise:
based on the trend function, predicting respective access frequences of the at least two objects at time points within the baseline replication of the at least two objects; and
based on the predicting of the respective access frequences at the time points, predicting respective average access frequences of the at least two objects within the baseline replication of the at least two objects.
14 . The device of claim 11 , wherein the determining of the respective access frequencies comprises:
in response to the access pattern being determined to be the cyclical pattern, predicting the respective access frequencies based on a cyclical function.
15 . The device of claim 14 , wherein the actions further comprise:
based on the cyclical function, determining respective maximum and respective minimum values of the respective access frequencies in the historical access data; and
based on the respective maximum and respective minimum values, determining the respective access frequency.
16 . The device of claim 11 , wherein the actions further comprise:
in response to the access pattern being determined to be the irregular pattern, determining an average of access frequencies comprised in the access pattern in the historical access data.
17 . A non-transitory machine-readable medium comprising executable instructions that, when executed by at least one processor of a computing device, facilitate performance of operations, the operations comprising:
based on historical access data for a group of objects, determining respective access patterns for the group of objects;
based on the respective access patterns, determining respective access frequencies for the group of objects in a full replication stage;
based on the respective access frequencies, performing a full replication on the group of objects from source storage equipment to destination storage equipment; and
in response to completion of the full replication of the group of objects, based on a stub object representative of at least one object of the group of objects being stored at the destination storage equipment, performing an incremental replication on the at least one object to complete data migration, wherein the performing of the full replication comprises:
for an object in the group of objects, determining whether an access frequency corresponding to the object is greater than a preset value,
in response to the access frequency being determined to be greater than the preset value, determining that the object is hot data, or
in response to the access frequency being determined to be less than or equal to the preset value, determining that the object is cold data, and
based on the hot data or the cold data, performing the full replication on the group of objects.
18 . The non-transitory machine-readable medium of claim 17 , wherein, in response to the object being determined to be the cold data, scheduling the object to be replicated before a different object determined to be the hot data.
19 . The non-transitory machine-readable medium of claim 17 , wherein the respective access patterns comprise at least one of a trend pattern, a cyclical pattern, or an irregular pattern.
20 . The non-transitory machine-readable medium of claim 19 , wherein, in response to an access pattern of the respective access patterns being determined to be the trend pattern, employing a trend function to predict the access frequency of the object based on the access pattern.