Information processing device, data processing method, and recording medium
Distributed batch processing on an eventually consistent storage system is efficiently performed. A control node includes an execution control unit and a re-execution control unit. The execution control unit causes a processing node to execute predetermined processing that includes reading of data for a key stored in a distributed data store. The re-execution control unit determines, after causing the processing node to execute the predetermined processing, presence or absence of a possibility that inconsistency occurred on data for the key stored in the distributed data store at a time of execution of the predetermined processing, based on a representative value of the data. Then, the re-execution control unit causes, when it is determined that there is a possibility that the inconsistency occurred, the processing node to re-execute the predetermined processing at a time point when resolution of the inconsistency is verified.
1. An information processing device comprising:
a memory storing instructions; and
one or more processors configured to execute the instructions to:
execute predetermined processing that includes reading of data for each of a plurality of keys stored in a distributed storage system including a plurality of data store nodes;
determine, after executing the predetermined processing for each of the plurality of keys, presence or absence of a possibility that inconsistency between the data store nodes occurred on data for any key among the plurality of keys stored in the distributed storage system at a time of execution of the predetermined processing, based on a representative value of the data for all of the plurality of keys; and
when it is determined that there is a possibility that the inconsistency occurred, identify all of the keys among the plurality of keys having a possibility that the inconsistency occurred, re-execute the predetermined processing for all of the keys identified at a time point when resolution of the inconsistency for all of the keys identified is verified.
2. The information processing device according to claim 1 , wherein,
in the distributed storage system, data for each of the plurality of keys is duplicated and stored in first and second regions, and
using a representative value of data for each of the plurality of keys and a representative value concerning all data for the plurality of keys which are generated for each of the first region and the second region, presence or absence of a possibility that inconsistency occurred on data for any of the plurality of keys is determined and a key having a possibility of the inconsistency is identified.
3. The information processing device according to claim 2 , wherein
when a representative value concerning all data for the plurality of keys in the first region and a representative value concerning all data for the plurality of keys in the second region, prior to execution of the predetermined processing, are different, it is determined that there is a possibility that inconsistency occurred on data for any of the plurality of keys, and
out of the plurality of keys, a key for which data in the first region and data in the second region have different representative values is identified, as a key having a possibility of the inconsistency.
4. The information processing device according to claim 1 , wherein
the one or more processors are further configured to execute the instructions to execute predetermined pre-processing that includes writing of data into the distributed storage system, and
the one or more processors are configured to execute the predetermined processing after the predetermined pre-processing is executed.
5. A data processing method comprising:
executing predetermined processing that includes reading of data for each of a plurality of keys stored in a distributed storage system including a plurality of data store nodes;
determining, after executing the predetermined processing for each of the plurality of keys, presence or absence of a possibility that inconsistency between the data store nodes occurred on data for any key among the plurality of keys stored in the distributed storage system at a time of execution of the predetermined processing, based on a representative value of the data for all of the plurality of keys; and
when it is determined that there is a possibility that the inconsistency occurred, identify all of the keys among the plurality of keys having possibility that the inconsistency occurred, re-executing the predetermined processing for all of the keys identified at a time point when resolution of the inconsistency for all of the keys identified is verified.
6. The data processing method according to claim 5 , wherein,
in the distributed storage system, data for each of the plurality of keys is duplicated and stored in first and second regions, and
using a representative value of data for each of the plurality of keys and a representative value concerning all data for the plurality of keys which are generated for each of the first region and the second region, presence or absence of a possibility that inconsistency occurred on data for any of the plurality of keys is determined and a key having a possibility of the inconsistency is identified.
7. The data processing method according to claim 6 , wherein,
when a representative value concerning all data for the plurality of keys in the first region and a representative value concerning all data for the plurality of keys in the second region, prior to execution of the predetermined processing, are different, it is determined that there is a possibility that inconsistency occurred on data for any of the plurality of keys, and
out of the plurality of keys, a key for which data in the first region and data in the second region have different representative values is identified, as a key having a possibility of the inconsistency.
8. A non-transitory computer readable storage medium recording thereon a program causing a computer to perform a method comprising:
executing predetermined processing that includes reading of data for each of a plurality of keys stored in a distributed storage system including a plurality of data store nodes;
determining, after executing the predetermined processing for each of the plurality of keys, presence or absence of a possibility that inconsistency between the data store nodes occurred on data for any key among the plurality of keys stored in the distributed storage system at a time of execution of the predetermined processing, based on a representative value of the data for all of the plurality of keys; and
when it is determined that there is a possibility that the inconsistency occurred, identify all of the keys among the plurality of keys having a possibility that the inconsistency occurred, re-executing the predetermined processing for all of the keys identified at a time point when resolution of the inconsistency for all of the keys identified is verified.