Accelerated document categorization using machine learning
A computer-implemented method is provided. The method may comprise: obtaining at least one document to be classified; classifying, using a machine learning model including an artificial neural network (ANN) and an attention mechanism, the at least one document into at least two classes; determining, for each of the at least one document, a confidence value of the classifying, based on one or more outputs of one or more nodes comprised in the ANN; assigning, to each of the at least one document, based at least in part on the confidence value, one of at least two categories that are associated with different degrees of credibility of the classifying; and providing for display one or more of the at least one document with: the assigned category and attention information that indicates significance of one or more parts of each document provided for display in the classifying of said document.
1 . A computer-implemented method comprising:
obtaining at least one document to be classified, the at least one document containing textual data;
classifying, using a machine learning model, the at least one document into at least two classes, the machine learning model including an artificial neural network, ANN, and an attention mechanism and the machine learning model being trained using a training dataset comprising:
a plurality of documents containing textual data, and
information indicating which one of the at least two classes each one of the plurality of documents belongs to;
determining, for each of the at least one document, a confidence value of the classifying, based on one or more outputs of one or more nodes comprised in the ANN, the one or more outputs being obtained from the classifying,
wherein the confidence value is determined based on:
one or more activation levels of the one or more nodes in a final layer of the ANN,
a maximum predictive probability of one of the at least two classes, and
a predictive uncertainty output by the machine learning model;
assigning, to each of the at least one document, based at least in part on the confidence value, one of at least two categories that are associated with different degrees of credibility of the classifying;
filtering the at least one document to exclude a document of the at least one document from further processing based on a respective confidence value of the document satisfying an upper confidence threshold or a lower confidence threshold; and
providing, for output and after the filtering, one or more of the at least one document with:
the one of the at least two categories assigned to each of the one or more of the at least one document, and
attention information that indicates significance of one or more parts of each document provided for display in the classifying of the document using the machine learning model,
wherein each document of the one or more of the at least one document provided for output has a confidence value that does not satisfy the upper confidence threshold and does not satisfy the lower confidence threshold,
wherein the one or more parts of each document are provided with at least one word-level indicator to cause display of an indication of a significance of at least one word based on one or more word-level vectors included in the attention information, and
wherein the one or more parts of each document are provided with at least one sentence-level indicator to cause display of an indication of a significance of at least one sentence based on one or more sentence-level vectors included in the attention information.
2 . The method according to claim 1 , wherein, in the classifying, the machine learning model performs a binary classification with a first class of relevant documents and a second class of irrelevant documents.
3 . The method according to claim 1 , wherein the one or more of the at least one document provided for display is or are assigned to a specified one of the at least two categories.
4 . The method according to claim 3 , wherein the specified one of the at least two categories is associated with the lowest degree of credibility of the classifying.
5 . The method according to claim 1 , further comprising:
collecting the plurality of documents to be included in the training dataset, each one of the plurality of documents having been assigned to one of the at least two classes;
pre-processing the textual data contained in the plurality of documents;
generating, after the pre-processing, the training dataset comprising the plurality of documents and information indicating which one of the at least two classes each one of the plurality of documents belongs to; and
training the machine learning model using the training dataset.
6 . The method according to claim 1 , wherein the machine learning model is trained further using a pretrained word embedding.
7 . The method according to claim 1 , wherein at least a part of the ANN included in the machine learning model forms a language model trained to determine a probability of a target word occurring, given one or more words surrounding the target word; and
wherein the machine learning model is trained by:
fine-tuning the language model using textual data relating to a specified domain in which the plurality of documents included in the training dataset are comprised; and
training the machine learning model to classify the plurality of documents comprised in the training dataset into the at least two classes.
8 . The method according to claim 1 , wherein the ANN included in the machine learning model comprises:
a recurrent neural network with an attention layer;
a hierarchical attention network; or
a transformer network.
9 . The method according to claim 1 , wherein the confidence value is determined further based on:
a measure indicating whether the at least one document input to the ANN is an outlier with respect to the training dataset.
10 . The method according to claim 1 , wherein, in the providing for output of the one or more of the at least one document, the attention information is provided for display so as to display the one or more parts of each document in manners different from each other based on the significance of the respective part or parts indicated by the attention information.
11 . The method according to claim 1 , wherein the one or more parts of each document provided for display in the classifying of the document using the machine learning model include one or more of the following:
one or more words in the textual data contained in the document, the one or more words including the at least one word; or
one or more sentences in the textual data contained in the document, the one or more sentences including the at least one sentence.
12 . The method according to claim 1 , wherein the at least one document to be classified and the plurality of documents comprised in the training dataset are patent documents.
13 . A computer-implemented method comprising:
obtaining at least one document to be classified, the at least one document containing textual data;
classifying, using a machine learning model, the at least one document into at least two classes, the machine learning model including an artificial neural network, ANN, and an attention mechanism and the machine learning model being trained using a training dataset comprising:
a plurality of documents containing textual data, and
information indicating which one of the at least two classes each one of the plurality of documents belongs to;
determining, for each of the at least one document, a confidence value of the classifying, based on one or more outputs of one or more nodes comprised in the ANN, the one or more outputs being obtained from the classifying of the at least one document using the machine learning model,
wherein the confidence value is determined based on:
one or more activation levels of the one or more nodes in a final layer of the ANN,
a maximum predictive probability of one of the at least two classes, and
a predictive uncertainty output by the machine learning model;
assigning, to each of the at least one document, based at least in part on the confidence value, one of at least two categories that are associated with different degrees of credibility of the classifying;
filtering the at least one document to exclude a document of the at least one document from further processing based on a respective confidence value of the document satisfying an upper confidence threshold or a lower confidence threshold; and
selecting one or more of the at least one document for further processing based on the filtering and one of the at least two categories assigned to each of the at least one document,
wherein each document of the one or more of the at least one document selected for further processing has a confidence value that does not satisfy the upper confidence threshold and does not satisfy the lower confidence threshold
providing, for display, the one or more documents of the at least one document with attention information that indicates significance, in the classifying, of one or more parts of each of the one or more documents of the at least one document provided for display,
wherein the one or more parts of each document are provided with at least one word-level indicator to cause display of an indication of a significance of at least one word based on one or more word-level vectors included in the attention information, and
wherein the one or more parts of each document are provided with at least one sentence-level indicator to cause display of an indication of a significance of at least one sentence based on one or more sentence-level vectors included in the attention information.
14 . A computer-implemented method comprising:
obtaining at least one document to be classified, the at least one document containing textual data;
classifying, using a machine learning model, the at least one document into at least two classes, the machine learning model including an artificial neural network, ANN, and an attention mechanism and the machine learning model being trained using a training dataset comprising:
a plurality of documents containing textual data, and
information indicating which one of the at least two classes each one of the plurality of documents belongs to;
identifying, for each of the at least one document, a confidence value associated with the classifying based on:
one or more activation levels of one or more nodes in a final layer of the ANN,
a maximum predictive probability of one of the at least two classes, and
a predictive uncertainty output by the machine learning model;
filtering the at least one document to exclude a document of the at least one document from further processing based on a respective confidence value of the document satisfying an upper confidence threshold or a lower confidence threshold; and
providing for display, after the filtering and based on corresponding confidence values for one or more documents of the at least one document not satisfying the upper confidence threshold and not satisfying the lower confidence threshold, the one or more documents of the at least one document with attention information that indicates significance, in the classifying, of one or more parts of each of the one or more documents of the at least one document provided for display,
wherein the one or more parts of each document are provided with at least one word-level indicator to cause display of an indication of a significance of at least one word based on one or more word-level vectors included in the attention information, and
wherein the one or more parts of each document are provided with at least one sentence-level indicator to cause display of an indication of a significance of at least one sentence based on one or more sentence-level vectors included in the attention information.
15 . A system comprising:
at least one storage medium configured to store a machine learning model including an artificial neural network, ANN, and an attention mechanism, wherein the machine learning model is trained using a training dataset comprising:
a plurality of documents containing textual data, and
information indicating which one of at least two classes each one of the plurality of documents belongs to; and
at least one processor in communication with the storage medium, the processor being configured to:
obtain at least one document to be classified, the at least one document containing textual data;
classify, using the machine learning model, the at least one document into the at least two classes;
determine, for each of the at least one document, a confidence value of the classifying, based on one or more outputs of one or more nodes comprised in the ANN, the one or more outputs being obtained from the classifying of the at least one document using the machine learning model,
wherein the confidence value is determined based on:
one or more activation levels of the one or more nodes in a final layer of the ANN,
a maximum predictive probability of one of the at least two classes, and
a predictive uncertainty output by the machine learning model;
assign, to each of the at least one document, based at least in part on the confidence value, one of at least two categories that are associated with different degrees of credibility of the classifying;
filter the at least one document to exclude a document of the at least one document from further processing based on a respective confidence value of the document satisfying an upper confidence threshold or a lower confidence threshold; and
provide, for output and after the filtering, one or more of the at least one document with:
the one of the at least two categories assigned to each of the one or more of the at least one document, and
attention information that indicates significance of one or more parts of each document provided for display in the classifying of the document using the machine learning model,
wherein each document of the one or more of the at least one document provided for output has a confidence value that does not satisfy the upper confidence threshold and does not satisfy the lower confidence threshold,
wherein the one or more parts of each document are provided with at least one word-level indicator to cause display of an indication of a significance of at least one word based on one or more word-level vectors included in the attention information, and
wherein the one or more parts of each document are provided with at least one sentence-level indicator to cause display of an indication of a significance of at least one sentence based on one or more sentence-level vectors included in the attention information.
16 . The system according to claim 15 , wherein the one or more of the at least one document provided for output is or are assigned to a specified one of the at least two categories, the specified one of the at least two categories being associated with the lowest degree of credibility of the classifying; and
wherein, in the providing for output of the one or more of the at least one document, the attention information is provided for display so as to display the one or more parts of each document in manners different from each other based on the significance of the respective part or parts indicated by the attention information.
17 . The system according to claim 15 , wherein the at least one processor is further configured to:
collect the plurality of documents to be included in the training dataset, each one of the plurality of documents having been assigned to one of the at least two classes;
pre-process the textual data contained in the plurality of documents;
generate, after the pre-processing, the training dataset comprising the plurality of documents and information indicating which one of the at least two classes each one of the plurality of documents belongs to; and
train the machine learning model using the training dataset.
18 . A non-transitory computer-readable medium comprising instructions that, when executed, configure one or more hardware processors to perform a method comprising:
obtaining at least one document to be classified, the at least one document containing textual data;
classifying, using a machine learning model, the at least one document into at least two classes, the machine learning model including an artificial neural network, ANN, and an attention mechanism and the machine learning model being trained using a training dataset comprising:
a plurality of documents containing textual data, and
information indicating which one of the at least two classes each one of the plurality of documents belongs to;
determining, for each of the at least one document, a confidence value of the classifying, based on one or more outputs of one or more nodes comprised in the ANN, the one or more outputs being obtained from the classifying of the at least one document using the machine learning model,
wherein the confidence value is determined based on:
one or more activation levels of the one or more nodes in a final layer of the ANN,
a maximum predictive probability of one of the at least two classes, and
a predictive uncertainty output by the machine learning model;
assigning, to each of the at least one document, based at least in part on the confidence value, one of at least two categories that are associated with different degrees of credibility of the classifying;
filtering the at least one document to exclude a document of the at least one document from further processing based on a respective confidence value of the document satisfying an upper confidence threshold or a lower confidence threshold; and
providing, for output and after the filtering, one or more of the at least one document with:
the one of the at least two categories assigned to each of the one or more of the at least one document, and
attention information that indicates significance of one or more parts of each document provided for display in the classifying of the document using the machine learning model,
wherein each document of the one or more of the at least one document provided for output has a confidence value that does not satisfy the upper confidence threshold and does not satisfy the lower confidence threshold,
wherein the one or more parts of each document are provided with at least one word-level indicator to cause display of an indication of a significance of at least one word based on one or more word-level vectors included in the attention information, and
wherein the one or more parts of each document are provided with at least one sentence-level indicator to cause display of an indication of a significance of at least one sentence based on one or more sentence-level vectors included in the attention information.
19 . The non-transitory computer-readable medium according to claim 18 , wherein the one or more of the at least one document provided for output is or are assigned to a specified one of the at least two categories, the specified one of the at least two categories being associated with the lowest degree of credibility of the classifying.
20 . The non-transitory computer-readable medium according to claim 18 , wherein, in the providing for display of the one or more of the at least one document, the attention information is provided for display so as to display the one or more parts of each document in manners different from each other based on the significance of the respective part or parts indicated by the attention information.