METHOD AND SYSTEM FOR TOKEN BASED CLASSIFICATION FOR REDUCING OVERLAP IN FIELD EXTRACTION DURING PARSING OF A TEXT
A system and a method for token-based classification for reducing overlap in field extraction during parsing of a text is disclosed. The method includes extracting text from a resource. The method further includes splitting one or more sentences into a predetermined number of plurality of tokens. The method furthermore includes generating a plurality of lists using a machine learning model for identifying one or more fields in the text. The plurality of lists comprises at least a list of tokens, a list of tags and a list of confidence scores of tokens. The method furthermore includes post processing the plurality of lists for extracting one or more fields for parsing the text.
1 . A processor-implemented method of token-based classification for reducing overlap in field extraction during parsing of a text, the method comprising:
extracting the text from a resource;
splitting one or more sentences in the text into a predetermined number of plurality of tokens;
generating a plurality of lists using a machine learning model, for identifying one or more fields in the text, wherein the plurality of lists comprises at least a list of tokens, a list of tags and a list of confidence score of tokens; and
post-processing the plurality of lists for extracting one or more fields for parsing the text.
2 . The processor-implemented method of claim 1 , wherein extracting the text from the resource comprises:
receiving a PDF document and identifying one or more bounding boxes in text from the PDF document;
converting the one or more bounding boxes into a plurality of images; and
parsing the text from each section of the plurality of images.
3 . The processor-implemented method of claim 1 , wherein generating the plurality of lists comprises:
classifying the one or more sentences with a plurality of labels;
splitting the classified sentences into one or more tokens; and
passing the one or more tokens into a classifier for generating the plurality of lists.
4 . A processor-implemented method of training a machine learning model for token-based classification for reducing overlap in field extraction during parsing of text, the method comprising:
extracting the text from a resource;
generating a training set for the artificial intelligence model based on the extracted text and importing the training set into the artificial intelligence model; and
training and evaluating the artificial intelligence model using the training set for generating a plurality of lists for identifying one or more fields in the text, wherein the plurality of lists comprises at least a list of tokens, a list of tags and a list of confidence score of tokens.
5 . The processor-implemented method of claim 4 , wherein the machine learning model is a Cased-Sci-Bert model.
6 . A system for token-based classification for reducing overlap in field extraction during parsing of a text, the system comprising a processor configured to execute non-transitory machine-readable instructions that when executed perform:
extracting the text from a resource;
splitting one or more sentences in the text into a predetermined number of plurality of tokens;
generating a plurality of lists using a machine learning model, for identifying one or more fields in the text, wherein the plurality of lists comprises at least a list of tokens, a list of tags and a list of confidence score of tokens; and
post processing the plurality of lists for extracting one or more fields for parsing the text.
7 . The system of claim 6 , wherein extracting the text from the resource comprises:
receiving a PDF document and identifying one or more bounding boxes in text from the PDF document;
converting the one or more bounding boxes into a plurality of images; and
parsing the text from each section of the plurality of images.
8 . The system of claim 6 , wherein generating the plurality of lists comprises:
classifying the one or more sentences with a plurality of labels;
splitting the classified sentences into one or more tokens; and
passing the one or more tokens into a classifier for generating the plurality of lists.