Method and system of creating and summarizing unstructured natural language sentence clusters for efficient tagging

1. A computerized method for reducing domain noise, creating and summarizing human-written sentences into clusters for efficient tagging in natural language processing comprising:

receiving a typed, handwritten or printed text;

implementing an optical character recognition (OCR) process on human written text to generate a digital version of the human written text;

splitting the digital version of the typed, handwritten or printed text into an array of sentences, using a sentence splitter to generate a split sentence version;

determining a domain of the human written text;

based on the domain, implementing a domain noise reduction process on the split sentences version;

hierarchically clustering the split sentences version after the domain noise reduction process; and

summarizing the clustered sentences and reducing the amount of data to be tagged.

2. The computerized method of claim 1 , wherein the human written text comprises a set of insurance claim notes.

3. The computerized method of claim 1 , wherein the domain noise reduction process further comprises:

splitting the typed, handwritten or printed text into an array of sentences using a language model sentence splitter; and

hierarchically using a domain model to handle language errors in the text and further split the sentences into split sentences.

4. The computerized method of claim 3 , wherein the domain noise reduction process further comprises:

using a sentence embedding to convert the split sentences to a vector; and

based on a sentence embedding, using a model to cluster the split sentences into a specified number of clusters.

5. The computerized method of claim 4 , wherein the domain noise reduction process further comprises:

for each cluster, determining: a number of sentences in the cluster, a mean and standard deviation of the sentences from a cluster center in the cluster.

6. The computerized method of claim 5 , wherein the domain noise reduction process further comprises:

applying a second layer machine learning classifier to determine a set of coherent clusters with domain noise; and

discarding the domain noise sentences from the original data to create domain noise reduced text.

7. The computerized method of claim 3 , wherein the model comprises a K-Means model.

8. The computerized method of claim 6 , wherein the step of hierarchically clustering the domain noise reduced sentences version further comprises:

implementing a sentence embedding of the split sentences version and repeating a clustering process.

9. The computerized method of claim 8 , wherein the step of hierarchically clustering the split sentences version further comprises:

within each cluster, applying a different sentence embedding for each sub-cluster of the split sentences in each cluster.

10. The computerized method of claim 9 , wherein based on a set of cluster metrics, each sub-cluster is classified into a specified category.

11. The computerized method of claim 10 , wherein the set of cluster metrics comprises:

number of sentences in each sub-cluster, mean and standard deviation from cluster center.

12. The computerized method of claim 11 , wherein the specified category comprises at least one: a coherent category wherein all the sentences in the cluster are semantically close to each other; a mostly coherent category wherein most of the sentences in the cluster are semantically very close to each other, but there are a few outliers; a ring category, wherein the cluster sentences form a ring around the cluster center; a discordant category wherein, the clusters have sentences that are spread out all over the place from the cluster center.

13. The computerized method of claim 12 , wherein the step of hierarchically clustering the split sentences version further comprises:

using a text summarization process to summarize each cluster into a lesser number of sentences depending on the sub-cluster category.

14. The computerized method of claim 13 , wherein the summarized cluster sentences are collated to reduce the domain space for efficient tagging without losing valuable information.