Systems and methods for cloud content-based document clustering and classification integration
A computer-implemented method, includes accessing, by a processor, a set of asset documents. The method also includes performing, by the processor, feature extraction on text of each document of the set of asset documents using a feature extraction module to generate a set of features, where each feature of the set of features represents a document of the set of asset documents. The method also includes generating, by the processor, a set of lower-dimensional features from the set of features using a singular value decomposition module. The method also includes generating, by the processor, a set of clusters from the set of lower-dimensional features using a clustering module. The method also includes training, by the processor, a machine-learning model of a classification microservice using the set of clusters generated from the clustering module.
1. A computer-implemented method, comprising:
accessing, by a processor, a set of asset documents;
receiving a feature extraction tool selection from a set of feature extraction tools stored within a feature extraction module, wherein the set of feature extraction tools stored within the feature extraction module comprises a bidirectional encoder representation of transformers (BERT) based feature extraction tool or a bag-of-words feature extraction tool;
performing, by the processor, feature extraction on text of each document of the set of asset documents using the feature extraction tool to generate a set of features, wherein each feature of the set of features represents a document of the set of asset documents;
providing, by the processor, the set of features to a singular value decomposition module;
generating, by the processor, a set of lower-dimensional features from the set of features using the singular value decomposition module;
generating, by the processor, a set of clusters representing document classes from the set of lower-dimensional features using a clustering module, wherein the clustering module is configured to perform a density based spatial clustering operation by assigning at least one document of the set of asset documents as a border sample document within the cluster of the set of clusters;
performing, by the processor, a grid search to adjust adjustable parameters of the feature extraction module, the singular value decomposition module, the clustering module, or a combination thereof; and
training, by the processor, a machine-learning model of a classification microservice using the set of clusters generated from the clustering module, wherein the classification microservice is configured to assign a document class of the set of document classes to additional documents added to the set of asset documents and further train the machine-learning model using results of clustering the additional documents.
2. The computer-implemented method of claim 1 , wherein performing the feature extraction, generating the set of lower-dimensional features, generating the set of clusters, and training the machine-learning model are each performed by cloud-based microservices coordinated by a scalable Kubernetes system.
3. The computer-implemented method of claim 1 , wherein the density based spatial clustering operation includes assigning one or more document of the set of asset documents as a core sample document within a cluster of the set of clusters or as an outlier document.
4. The computer-implemented method of claim 1 , further comprising:
generating, by the processor, a machine-encoded text representation of each document of the set of asset documents using an optical character recognition module, wherein the feature extraction is performed on the machine-encoded text representation of each document of the set of asset documents.
5. A computing system, comprising:
one or more processors; and
one or more memory devices including instructions that are executable by the one or more processors for causing the one or more processors to:
access a set of asset documents;
receive a feature extraction tool selection from a set of feature extraction tools stored within a feature extraction module, wherein the set of feature extraction tools stored within the feature extraction module comprises a bidirectional encoder representation of transformers (BERT) based feature extraction tool or a bag-of-words feature extraction tool;
perform feature extraction on text of each document of the set of asset documents using the feature extraction tool selection to generate a set of features, wherein each feature of the set of features represents a document of the set of asset documents;
provide, by the processor, the set of features to a singular value decomposition module;
generate a set of lower-dimensional features from the set of features using the singular value decomposition module;
generate a set of clusters representing document classes from the set of lower-dimensional features using a clustering module, wherein the clustering module is configured to perform a density based spatial clustering operation by assigning at least one document of the set of asset documents as a border sample document within the cluster of the set of clusters;
perform a grid search to adjust adjustable parameters of the feature extraction module, the singular value decomposition module, the clustering module, or a combination thereof; and
train a machine-learning model of a classification microservice using the set of clusters generated from the clustering module, wherein the classification microservice is configured to assign a document class of the set of document classes to additional documents added to the set of asset documents and further train the machine-learning model using results of clustering the additional documents.
6. The computing system of claim 5 , wherein the instructions are further executable by the one or more processors for causing the one or more processors to:
generate a machine-encoded text representation of each document of the set of asset documents using an optical character recognition module, wherein the feature extraction is performed on the machine-encoded text representation of each document of the set of asset documents.
7. The computing system of claim 5 , wherein the density based spatial clustering operation includes assigning one or more document of the set of asset documents as a core sample document within a cluster of the set of clusters or as an outlier document.
8. A non-transitory computer-readable medium comprising computer-executable instructions to cause a computer to:
access a set of asset documents;
receive a feature extraction tool selection from a set of feature extraction tools stored within a feature extraction module, wherein the set of feature extraction tools stored within the feature extraction module comprises a bidirectional encoder representation of transformers (BERT) based feature extraction tool or a bag-of-words feature extraction tool;
perform feature extraction on text of each document of the set of asset documents using the feature extraction tool selection to generate a set of features, wherein each feature of the set of features represents a document of the set of asset documents;
provide the set of features to a singular value decomposition module;
generate a set of lower-dimensional features from the set of features using the singular value decomposition module;
generate a set of clusters representing document classes from the set of lower-dimensional features using a clustering module, wherein the clustering module is configured to perform a density based spatial clustering operation by assigning at least one document of the set of asset documents as a border sample document within the cluster of the set of clusters;
perform a grid search to adjust adjustable parameters of the feature extraction module, the singular value decomposition module, the clustering module, or a combination thereof; and
train a machine-learning model of a classification microservice using the set of clusters generated from the clustering module, wherein the classification microservice is configured to assign a document class of the set of document classes to additional documents added to the set of asset documents and further train the machine-learning model using results of clustering the additional documents.
9. The non-transitory computer-readable medium of claim 8 , comprising further computer-executable instructions to cause the computer to:
generate a machine-encoded text representation of each document of the set of asset documents using an optical character recognition module, wherein the feature extraction is performed on the machine-encoded text representation of each document of the set of asset documents.
10. The method of claim 1 , further comprising:
generating, for each cluster in the set of clusters, a silhouette value, wherein the silhouette value is associated with a density of documents in each cluster;
determining an average silhouette value for the set of clusters; and
adjusting one or more adjustable parameters of the feature extraction module, the singular value decomposition module, the clustering module, or a combination thereof to maximize the average silhouette value.
11. The method of claim 10 , wherein the silhouette value is generated using a distance between lower-dimensional features of documents within a cluster of the set of clusters.
12. The computing system of claim 5 , wherein the instructions are further executable by the one or more processors for causing the one or more processors to:
generate, for each cluster in the set of clusters, a silhouette value, wherein the silhouette value is associated with a density of documents in each cluster;
determine an average silhouette value for the set of clusters; and
adjust one or more adjustable parameters of the feature extraction module, the singular value decomposition module, the clustering module, or a combination thereof to maximize the average silhouette value.
13. The computing system of claim 12 , wherein the silhouette value is generated using a distance between lower-dimensional features of documents within a cluster of the set of clusters.
14. The non-transitory computer-readable medium of claim 8 , comprising further computer-executable instructions to cause the computer to:
generate, for each cluster in the set of clusters, a silhouette value, wherein the silhouette value is associated with a density of documents in each cluster;
determine an average silhouette value for the set of clusters; and
adjust one or more adjustable parameters of the feature extraction module, the singular value decomposition module, the clustering module, or a combination thereof to maximize the average silhouette value.
15. The non-transitory computer-readable medium of claim 14 , wherein the silhouette value is generated using a distance between lower-dimensional features of documents within a cluster of the set of clusters.