DATA COLLECTION APPARATUS, DATA COLLECTION METHOD, AND PROGRAM
A data collection device according to an embodiment includes: an acquisition unit that acquires data when the data is stored in a shared storage area available to one or more users; a determination unit that determines whether or not a format of the data acquired by the acquisition unit is a format in which text included in the data can be extracted by a predetermined library; an extraction unit that extracts the text included in the data by a text extraction method according to a determination result determined by the determination unit; and a storage unit that stores the text extracted by the extraction unit in a database as learning data for a machine learning model that implements a natural language processing task.
1 . A data collection device comprising a processor configured to execute operations comprising:
acquiring data when the data is stored in a shared storage area available to one or more users;
determining whether or not a format of the acquired data is a format in which text included in the data is extractable by a predetermined library;
extracting the text included in the data by a text extraction method according to a result of the determining; and
storing a storage unit configured to store the extracted text in a database as learning data for a machine learning model that implements a natural language processing task.
2 . The data collection device according to claim 1 , wherein the extracting further comprises:
when the result of the determining indicates that the format of the data is the format in which the text included in the data is extractable by a predetermined library, extracting the text included in the data by the library; and
when the result of the determining indicates that the format of the data is not the format in which the text included in the data is extractable by a predetermined library, extracting the text included in the data by optical character recognition.
3 . The data collection device according to claim 1 , wherein the storing further comprises processing the text into an input format of the machine learning model, and storing the processed text in the database as the learning data.
4 . The data collection device according to claim 1 , wherein the storing further comprises dividing the text into predetermined units, and storing each piece of the divided text in the database as the learning data.
5 . The data collection device according to claim 1 , wherein the shared storage area comprises includes at least one of a shared folder of a storage existing in a local area network or a shared folder of an external storage available via the Internet.
6 . The data collection device according to claim 1 , wherein the database comprises a data store having a search function for the text.
7 . A computer implemented method for collecting data, comprising:
acquiring data when the data is stored in a shared storage area available to one or more users;
determining whether or not a format of the data acquired in the acquisition step is a format in which text included in the data is extractable by a predetermined library;
extracting the text included in the data by a text extraction method according to a result of the determining; and
storing the text extracted in the extraction step in a database as learning data for a machine learning model that implements a natural language processing task.
8 . A computer-readable non-transitory recording medium storing a computer-executable program instructions that when executed by a processor cause a computer to execute operations comprising:
acquiring data when the data is stored in a shared storage area available to one or more users;
determining whether or not a format of the acquired data is a format in which text included in the data is extractable by a predetermined library;
extracting the text included in the data by a text extraction method according to a result of the determining; and
storing the extracted text in a database as learning data for a machine learning model that implements a natural language processing task.
9 . The computer implemented method according to claim 7 , further comprising:
when the result of the determining indicates that the format of the data is the format in which the text included in the data is extractable by a predetermined library, extracting the text included in the data by the library; and
when the result of the determining indicates that the format of the data is not the format in which the text included in the data is extractable by a predetermined library, extracting the text included in the data by optical character recognition.
10 . The computer implemented method according to claim 7 , wherein the storing further comprises processing the text into an input format of the machine learning model, and storing the processed text in the database as the learning data.
11 . The computer implemented method according to claim 7 , wherein the storing further comprises dividing the text into predetermined units, and storing each piece of the divided text in the database as the learning data.
12 . The computer implemented method according to claim 7 , wherein the shared storage area comprises at least one of a shared folder of a storage existing in a local area network or a shared folder of an external storage available via the Internet.
13 . The computer implemented method according to claim 7 , wherein the database comprises a data store having a search function for the text.
14 . The computer-readable non-transitory recording medium according to claim 8 , the computer-executable program instructions when executed further causing the computer to execute operations comprising:
when the result of the determining indicates that the format of the data is the format in which the text included in the data is extractable by a predetermined library, extracting the text included in the data by the library; and
when the result of the determining indicates that the format of the data is not the format in which the text included in the data is extractable by a predetermined library, extracting the text included in the data by optical character recognition.
15 . The computer-readable non-transitory recording medium according to claim 8 , wherein the storing further comprises processing the text into an input format of the machine learning model, and storing the processed text in the database as the learning data.
16 . The computer-readable non-transitory recording medium according to claim 8 , wherein the storing further comprises dividing the text into predetermined units, and storing each piece of the divided text in the database as the learning data.
17 . The computer-readable non-transitory recording medium according to claim 8 , wherein the shared storage area comprises at least one of a shared folder of a storage existing in a local area network or a shared folder of an external storage available via the Internet.
18 . The computer-readable non-transitory recording medium according to claim 8 , wherein the database comprises a data store having a search function for the text.