Enhanced named entity recognition (NER) using custombuilt regular expression (REGEX) matcher and heuristic entity ruler

1 . A processor implemented method for Named Entity Recognition (NER), the method comprising:

receiving, via one or more hardware processors, a text document comprising unstructured text;

converting the text document into another format for further processing, wherein the conversion of text document into another format includes removing unreadable characters that do not get properly encoded in a text format, making clean document to send in natural language processing (NLP) pipeline;

preprocessing, via the one or more hardware processors, the unstructured text by sequentially applying cleaning, tokenization divides larger chunks of text into smaller tokens for faster processing and converts string representation of data to numerical representation of data into a token and vectors by a tok2vec to speedup the processing time of NLP processes, vectorization, Parts of Speech (PoS) tagging, and parsing based on the PoS tagging and sentence semantics on the unstructured text to generate a parsed data for mapping dependency of various parts of sentences to each other;

applying, via the one or more hardware processors:

(i) a pretrained transformer-based NER model, on the parsed data to categorize the parsed data by identifying one or more NER model-NEs in the parsed data in accordance with a plurality of NE classes recognized by the pre-trained transformer-based NER model; and

(ii) a custom-built REGEX matcher on the parsed data, wherein the custom-built REGEX matcher is based on a set of customized REGEX rules associated with a REGEX knowledge base with an ability to learn and unlearn to capture NEs of given text completely and curated as per domain requirement, identify updated NEs for a domain of interest, find alpha-numerical text and their variations, and handle complex requirements of patterns, wherein the updated NEs comprising (a) the one or more NER model-NEs, and (b) an expression associated with the one or more NER model-NEs, wherein the set of customized REGEX rules enable defining conventional REGEX patterns, semantic level rules and business rules that are expressed in patterns, wherein the one or more NER model-NEs among the plurality of NER model-NEs are part of the expression, combining the output of NER and regex and give single NE;

applying the NER in the NLP pipeline to identify suitable category or NE classes for the NE token; and

applying, via the one or more hardware processors, a heuristic entity ruler on the updated NEs enabling class resolution among the plurality of NE classes and a heuristic entity ruler defined NE classes to extract a final NE list from the unstructured data and associated frequency of occurrence of each NE among the final NE list, wherein the final NE list with a classification report is customizable to requirements of end user for diverse applications, wherein the heuristic entity ruler is based on a set of heuristic rules associated with an entity ruler knowledge base, wherein the knowledge base is continuously increased and allows the end user to add or update with business rules and a feedback from the end user is added for next iterations of a NLP pipeline, wherein the class resolution states, all NEs identified by the pretrained transformer-based NER model and the custom-built REGEX matcher to be overwritten by the heuristic entity ruler if the heuristic entity ruler finds a new NE class, wherein the heuristic entity ruler defined NE classes are updated dynamically by training the heuristic entity ruler for deletion or addition of one or more NE classes without need for retraining the pretrained transformer-based NER model,

wherein with the heuristic entity ruler, new NEs are added by the end user to provide only name and list in plain English words and the heuristic entity ruler converts components of the pipeline, the name and the list in plain English into a training format,

wherein with the heuristic entity ruler for learning and unlearning requires less training time in the NLP pipeline only affects the selected NEs and remaining NEs are retained and provide high re-usability and any changes made do not require starting from scratch,

wherein the heuristic entity ruler retains highest priority tagged NEs class and ensures the identified NEs by the earlier NLP components are not overwritten in subsequent components for consideration of the final suitable tag, where the NEs defined in the NLP pipeline as the last component that identified NEs by this component are of highest priority and overwrite the pre-existing tag, provide the sub-level relevant NEs and providing accurate information.

2 . The method of claim 1 , wherein the custom-built REGEX matcher enables class resolution between the plurality of NER-NEs and updated NEs, wherein the custom-built REGEX enables the user to customize or add new heuristic rules for entity ruler or custom regex as the knowledgebase to train the model with automatic relearning and unlearning, provide the extracted NEs or masking in a structured format.

3 . A system for Named Entity Recognition (NER), the system comprising:

a memory storing instructions;

one or more Input/Output (I/O) interfaces; and

one or more hardware processors coupled to the memory via the one or more I/O interfaces, wherein the one or more hardware processors are configured by the instructions to:

receive a text document comprising unstructured text;

convert the text document into another format for further processing, wherein the conversion of text document into another format includes removing unreadable characters that do not get properly encoded in a text format, making clean document to send in natural language processing (NLP) pipeline;

preprocess the unstructured text by sequentially applying cleaning, tokenization divides larger chunks of text into smaller tokens for faster processing and converts string representation of data to numerical representation of data into a token and vectors by a tok2vec to speedup the processing time of NLP processes, vectorization, Parts of Speech (PoS) tagging, and parsing based on the PoS tagging and sentence semantics on the unstructured text to generate a parsed data for mapping dependency of various parts of sentences to each other;

apply:

(i) a pretrained transformer-based NER model, on the parsed data to categorize the parsed data by identifying one or more NER model-NEs in the parsed data in accordance with a plurality of NE classes recognized by the pre-trained transformer-based NER model; and

(ii) a custom-built REGEX matcher on the parsed data, wherein the custom-built REGEX matcher is based on a set of customized REGEX rules associated with a REGEX knowledge base with an ability to learn and unlearn to capture NEs of given text completely and curated as per domain requirement, identify updated NEs for a domain of interest, find alpha-numerical text and their variations, and handle complex requirements of patterns, wherein the updated NEs comprising (a) the one or more NER model-NEs, and (b) an expression associated with the one or more NER model-NEs, wherein the set of customized REGEX rules enable defining conventional REGEX patterns, semantic level rules and business rules that are expressed in patterns, wherein the one or more NER model-NEs among the plurality of NER model-NEs are part of the expression, combining the output of NER and regex and give single NE;

apply the NER in the NLP pipeline to identify suitable category or NE classes for the NE token; and

apply a heuristic entity ruler on the updated NEs enabling class resolution among the plurality of NE classes and a heuristic entity ruler defined NE classes to extract a final NE list from the unstructured data and associated frequency of occurrence of each NE among the final NE list, wherein the final NE list with a classification report is customizable to requirements of end user for diverse applications, wherein the heuristic entity ruler is based on a set of heuristic rules associated with an entity ruler knowledge base, wherein the knowledge base is continuously increased and allows the end user to add or update with business rules and a feedback from the end user is added for next iterations of a NLP pipeline, wherein the class resolution states, all NEs identified by the pretrained transformer-based NER model and the custom-built REGEX matcher to be overwritten by the heuristic entity ruler if the heuristic entity ruler finds a new NE class, wherein the heuristic entity ruler defined NE classes are updated dynamically by training the heuristic entity ruler for deletion or addition of one or more NE classes without need for retraining the pretrained transformer-based NER model, wherein with the heuristic entity ruler, new NEs are added by the end user to provide only name and list in plain English words and the heuristic entity ruler converts components of the pipeline, the name and the list in plain English into a training format, wherein with the heuristic entity ruler for learning and unlearning requires less training time in the NLP pipeline only affects the selected NEs and remaining NEs are retained and provide high re-usability and any changes made do not require starting from scratch, wherein the heuristic entity ruler retains highest priority tagged NEs class and ensures the identified NEs by the earlier NLP components are not overwritten in subsequent components for consideration of the final suitable tag, where the NEs defined in the NLP pipeline as the last component that identified NEs by this component are of highest priority and overwrite the pre-existing tag, provide the sub-level relevant NEs and providing accurate information.

4 . The system of claim 3 , wherein the custom-built REGEX matcher enables class resolution between the plurality of NER-NEs and updated NEs, wherein the custom-built REGEX enables the user to customize or add new heuristic rules for entity ruler or custom regex as the knowledgebase to train the model with automatic relearning and unlearning, provide the extracted NEs or masking in a structured format.

5 . One or more non-transitory machine-readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors cause:

receiving, a text document comprising unstructured text;

converting the text document into another format for further processing, wherein the conversion of text document into the another format includes removing unreadable characters that do not get properly encoded in a text format, making clean document to send in natural language processing (NLP) pipeline;

preprocessing, the unstructured text by sequentially applying cleaning, tokenization divides larger chunks of text into smaller tokens for faster processing and converts string representation of data to numerical representation of data into a token and vectors by a tok2vec to speedup the processing time of NLP processes, vectorization, Parts of Speech (PoS) tagging, and parsing based on the PoS tagging and sentence semantics on the unstructured text to generate a parsed data for mapping dependency of various parts of sentences to each other;

applying:

i) a pretrained transformer-based NER model, on the parsed data to categorize the parsed data by identifying one or more NER model-NEs in the parsed data in accordance with a plurality of NE classes recognized by the pre-trained transformer-based NER model; and

ii) a custom-built REGEX matcher on the parsed data, wherein the custom-built REGEX matcher is based on a set of customized REGEX rules associated with a REGEX knowledge base with an ability to learn and unlearn to capture NEs of given text completely and curated as per domain requirement, identify updated NEs for a domain of interest, find alpha-numerical text and their variations, and handle complex requirements of patterns, wherein the updated NEs comprising (a) the one or more NER model-NEs, and (b) an expression associated with the one or more NER model-NEs, wherein the set of customized REGEX rules enable defining conventional REGEX patterns, semantic level rules and business rules that are expressed in patterns, wherein the one or more NER model-NEs among the plurality of NER model-NEs are part of the expression, combining the output of NER and regex and give single NE;

applying the NER in the NLP pipeline to identify suitable category or NE classes for the NE token; and

applying, a heuristic entity ruler on the updated NEs enabling class resolution among the plurality of NE classes and a heuristic entity ruler defined NE classes to extract a final NE list from the unstructured data and associated frequency of occurrence of each NE among the final NE list, wherein the final NE list with a classification report is customizable to requirements of end user for diverse applications, wherein the heuristic entity ruler is based on a set of heuristic rules associated with an entity ruler knowledge base, wherein the knowledge base is continuously increased and allows the end user to add or update with business rules and feedback from the end user is added for next iterations of a NLP pipeline, wherein the class resolution states, all NEs identified by the pretrained transformer-based NER model and the custom-built REGEX matcher to be overwritten by the heuristic entity ruler if the heuristic entity ruler finds a new NE class, wherein the heuristic entity ruler defined NE classes are updated dynamically by training the heuristic entity ruler for deletion or addition of one or more NE classes without need for retraining the pretrained transformer-based NER model, wherein with the heuristic entity ruler, new NEs are added by the end user to provide only name and list in plain English words and the heuristic entity ruler converts components of the pipeline, the name and the list in plain English into a training format, wherein with the heuristic entity ruler for learning and unlearning requires less training time in the NLP pipeline only affects the selected NEs and remaining NEs are retained and provide high re-usability and any changes made do not require starting from scratch, wherein the heuristic entity ruler retains highest priority tagged NEs class and ensures the identified NEs by the earlier NLP components are not overwritten in subsequent components for consideration of the final suitable tag, where the NEs defined in the NLP pipeline as the last component that identified NEs by this component are of highest priority and overwrite the pre-existing tag, provide the sub-level relevant NEs and providing accurate information.

6 . The one or more non-transitory machine-readable information storage mediums of claim 5 , wherein the custom-built REGEX matcher enables class resolution between the plurality of NER-NEs and updated NEs, wherein the custom-built REGEX enables the user to customize or add new heuristic rules for entity ruler or custom regex as the knowledgebase to train the model with automatic relearning and unlearning, provide the extracted NEs or masking in a structured format.