Method, electronic device, and computer program product for collecting training data
Embodiments of the present disclosure relate to a method, an electronic device, and a computer program product for collecting training data. The method for collecting training data provided in embodiments of the present disclosure includes: scanning a plurality of data sources to acquire information relating to a plurality of training data to be collected, and creating a collection list based on the information, the collection list including at least a plurality of identifiers of the plurality of training data and a plurality of storage locations of the plurality of training data in the plurality of data sources. The method further includes: collecting the plurality of training data from the plurality of data sources based at least on the collection list.
1. A method for collecting training data, comprising:
scanning a plurality of data sources to acquire information relating to a plurality of training data to be collected;
creating a collection list based on the information, the collection list comprising at least a plurality of identifiers of the plurality of training data and a plurality of storage locations of the plurality of training data in the plurality of data sources, wherein the collection list enables tracking and resuming interrupted collection of the training data; creating a collection status table recording collection success or failure status for each identifier; and
collecting the plurality of training data from the plurality of data sources based at least on the collection list and the collection status table, wherein collecting comprises: re-collecting only training data having a failure status after a predetermined time period since a previous collection attempt.
2. The method according to claim 1 , wherein the collection list further comprises a plurality of names of a plurality of filters for filtering the plurality of training data.
3. The method according to claim 1 , wherein collecting the training data from the plurality of data sources comprises:
updating the collection status table based on the collection of the training data, the collection status table recording at least the plurality of identifiers and the plurality of collection statuses corresponding to the plurality of identifiers.
4. The method according to claim 3 , wherein collecting the plurality of training data from the plurality of data sources further comprises:
setting, in response to not successfully collecting first training data corresponding to a first identifier from a first data source of the plurality of data sources, a first collection status corresponding to the first identifier to failure in the collection status table; and
setting, in response to successfully collecting second training data corresponding to a second identifier from a second data source of the plurality of data sources, a second collection status corresponding to the second identifier to success in the collection status table.
5. The method according to claim 4 , further comprising:
determining whether a collection status of failure exists in the collection status table;
acquiring the first identifier corresponding to the first collection status in response to determining that the first collection status of failure exists in the collection status table;
re-collecting the first training data corresponding to the first identifier from the first data source based on the collection list; and
updating the first collection status to success in response to successfully collecting the first training data from the first data source.
6. The method according to claim 5 , wherein re-collecting the first training data corresponding to the first identifier from the first data source comprises:
re-collecting the first training data from the first data source after a predetermined time period since a previous collection.
7. The method according to claim 4 , further comprising:
determining whether a collection status of failure exists in the collection status table; and
completing collection of the plurality of training data in response to no collection status of failure existing in the collection status table.
8. An electronic device, comprising:
a processor; and
a memory coupled to the processor, the memory having instructions stored therein, wherein the instructions, when executed by the processor, cause the device to perform actions comprising:
scanning a plurality of data sources to acquire information relating to a plurality of training data to be collected;
creating a collection list based on the information, the collection list comprising at least a plurality of identifiers of the plurality of training data and a plurality of storage locations of the plurality of training data in the plurality of data sources, wherein the collection list enables tracking and resuming interrupted collection of the training data; creating a collection status table recording collection success or failure status for each identifier; and
collecting the plurality of training data from the plurality of data sources based at least on the collection list and the collection status table, wherein collecting comprises: re-collecting only training data having a failure status after a predetermined time period since a previous collection attempt.
9. The device according to claim 8 , wherein the collection list further comprises a plurality of names of a plurality of filters for filtering the plurality of training data.
10. The device according to claim 8 , wherein collecting the training data from the plurality of data sources comprises:
updating the collection status table based on the collection of the training data, the collection status table recording at least the plurality of identifiers and the plurality of collection statuses corresponding to the plurality of identifiers.
11. The device according to claim 10 , wherein collecting the plurality of training data from the plurality of data sources further comprises:
setting, in response to not successfully collecting first training data corresponding to a first identifier from a first data source of the plurality of data sources, a first collection status corresponding to the first identifier to failure in the collection status table; and
setting, in response to successfully collecting second training data corresponding to a second identifier from a second data source of the plurality of data sources, a second collection status corresponding to the second identifier to success in the collection status table.
12. The device according to claim 11 , wherein the actions further comprise:
determining whether a collection status of failure exists in the collection status table;
acquiring the first identifier corresponding to the first collection status in response to determining that the first collection status of failure exists in the collection status table;
re-collecting the first training data corresponding to the first identifier from the first data source based on the collection list; and
updating the first collection status to success in response to successfully collecting the first training data from the first data source.
13. The device according to claim 12 , wherein re-collecting the first training data corresponding to the first identifier from the first data source comprises:
re-collecting the first training data from the first data source after a predetermined time period since a previous collection.
14. The device according to claim 11 , wherein the actions further comprise:
determining whether a collection status of failure exists in the collection status table; and
completing collection of the plurality of training data in response to no collection status of failure existing in the collection status table.
15. A computer program tangibly stored on a computer-readable medium and comprising instructions, wherein the instructions, when executed, by a processor cause the program to:
scan a plurality of data sources to acquire information relating to a plurality of training data to be collected;
create a collection list based on the information, the collection list comprising at least a plurality of identifiers of the plurality of training data and a plurality of storage locations of the plurality of training data in the plurality of data sources, wherein the collection list enables tracking and resuming interrupted collection of the training data; creating a collection status table recording collection success or failure status for each identifier; and
collect the plurality of training data from the plurality of data sources based at least on the collection list and the collection status table, wherein collecting comprises: re-collecting only training data having a failure status after a predetermined time period since a previous collection attempt.
16. The computer-readable medium of claim 15 wherein the collection list further comprises a plurality of names of a plurality of filters for filtering the plurality of training data.
17. The computer-readable medium of claim 15 wherein the machine-executable instructions configured to collect the training data from the plurality of data sources are further configured to:
update the collection status table based on the collection of the training data, the collection status table recording at least the plurality of identifiers and the plurality of collection statuses corresponding to the plurality of identifiers.
18. The computer-readable medium of claim 17 wherein the machine-executable instructions configured to collect the training data from the plurality of data sources are further configured to:
set, in response to not successfully collecting first training data corresponding to a first identifier from a first data source of the plurality of data sources, a first collection status corresponding to the first identifier to failure in the collection status table; and
set, in response to successfully collecting second training data corresponding to a second identifier from a second data source of the plurality of data sources, a second collection status corresponding to the second identifier to success in the collection status table.
19. The computer-readable medium of claim 18 further configured to:
determine whether a collection status of failure exists in the collection status table;
acquire the first identifier corresponding to the first collection status in response to determining that the first collection status of failure exists in the collection status table;
re-collect the first training data corresponding to the first identifier from the first data source based on the collection list; and
update the first collection status to success in response to successfully collecting the first training data from the first data source.
20. The computer-readable medium of claim 19 , wherein the machine-executable instructions configured to re-collect the first training data corresponding to the first identifier from the first data source are further configured to:
re-collect the first training data from the first data source after a predetermined time period since a previous collection.