Method and system for document classification and text information extraction
Variation in received documents types and templates used for each document type poses challenge in developing a generic background noise removal approach for automatic text information extraction technique. Embodiments herein provide a method and a system for document classification and text information extraction. Time efficient and accurate text detection engine-based Region of Interest (ROI) technique is provided to accurately identify text region followed by a multi-layered neural network based architecture for enhanced classification accuracy to identify the type of document. A multistage image pre-processing approach is provided for efficient, effective, and accurate background noise removal from the classified document, which includes unsupervised clustering, identification, segmentation, masking, contour approximation, selective subtraction, and dynamic thresholding. Further, text information extraction is applied on the pre-processed image to determine co-relation between text information blocks and associate values with the labels for attributes to be extracted to extract final text.
1. A method for classification of a document and text information extraction of the document, the method comprising:
receiving, via one or more hardware processors, a scanned image of the document;
detecting, via the one or more hardware processors, a Region of Interest (ROI) in the scanned image by marking a ROI bounding box on the scanned image using a text detection engine based-ROI technique, wherein the text detection engine based-ROI technique locates one or more corner coordinates of the ROI bounding box defining the ROI;
classifying, via the one or more hardware processors, the ROI into a document type among a plurality of document types using a trainable Deep Learning based multi-layered Neural Network (NN) classification model, wherein training of the trainable Deep Learning based multi-layered NN classification model comprises:
generating sample images as training data; and
performing pre-processing of the training data, augmentation of the training data, and data annotation of the training data to train the trainable Deep Learning based multi-layered NN classification model, wherein
the trainable Deep Learning based multi-layered NN classification model includes an architecture that is configurable based on requirements, and
a complexity of the architecture of the trainable Deep Learning based multi-layered NN classification model is varied based on the training data;
applying, via the one or more hardware processors, multistage pre-processing on the classified ROI to remove a background noise, wherein the multistage pre-processing comprises:
reading the ROI into a Red Green Blue (RGB) color space and flattening the ROI;
performing unsupervised clustering by applying K-means clustering on a plurality of pixels of the ROI in the RGB color space to generate a plurality of color clusters;
obtaining a plurality of centroids from each of the plurality of color clusters, wherein each centroid of the plurality of centroids represents a unique mean color associated with each of the plurality of clusters;
converting the plurality of centroids from the RGB color space to a Hue Saturation Value (HSV) space;
generating a plurality of color masks corresponding to the plurality of clusters, wherein each color mask of the plurality of color masks is generated based on a) the unique mean color associated with a centroid among the plurality of centroids and b) a range of the HSV color space defined around the centroid, wherein each of the plurality of color masks is generated such that each color mask sets:
a plurality first specific of pixels of the plurality of pixels of the centroid having HSV values within the HSV range to a first value, and
a plurality of second specific pixels of the plurality of pixels to a second value;
applying each of the plurality of color masks to the ROI to obtain a plurality of binary ROI images, wherein each of the plurality of binary ROI image comprises one or more contours indicating spatial locations of one or more pixels among the plurality of pixels in the ROI that belong to the unique mean color of the centroid;
identifying in each of the plurality of binary ROI images, one or more contours of interest from the plurality of contours, wherein the identified one or more contours of interest are a) closed contours and b) the identified one or more contours of interest has a size above a predefined contour size;
performing subtraction of the one or more identified contours of interest from the ROI in accordance to spatial positions of the plurality of pixels of the identified one or more contours of interest, wherein the subtraction eliminates the background noise while retaining information of interest to generate a first level pre-processed image; and
applying thresholding on the first level pre-processed image to obtain a second level pre-processed image using a threshold value derived dynamically from a histogram of the first level pre-processed image, wherein the threshold value is derived dynamically by:
calculating the histogram of the first level pre-processed image;
determining average pixel intensity of the first level pre-processed image and a left most value and a right most peak value in the histogram; and
calculating the threshold value by averaging the left most and the right most peak values;
applying, via the one or more hardware processors, a text detection technique on the second level pre-processed image to mark a plurality of bounding boxes around text information in the second level pre-processed image, wherein
each of the plurality of bounding boxes are identified by one or more spatial positions defined by the one or more corner coordinates and a corresponding height and a corresponding width,
one or more bounding boxes of the plurality of bounding boxes are clubbed based on a spatial proximity criteria,
the spatial proximity criteria analyzes a proximity among the one or more bounding boxes which has spatially relatable text to club the one or more bounding boxes, and
each of the one or more bounding boxes are clubbed to form relatable information into a single bounding;
extracting, via the one or more hardware processors, text information from each of the plurality of bounding boxes by applying an Optical Character Recognition (OCR); and
determining, via the one or more hardware processors, contextual relationship among the extracted text information and refining the extracted text information based on configuration rules for the document type.
2. The method of claim 1 , wherein the configuration rules comprise a precompiled knowledge base that is referred to identify undesired text from the extracted text information, wherein the undesired text is discarded from the extracted text information, and wherein the precompiled knowledge base is composition of KEYWORDS and VALUE_CATEGORY.
3. The method of claim 1 , further comprises correcting, via the one or more hardware processors, name values from the extracted text information using an encoder-decoder model that utilizes Long short-term memory (LSTM) based Recurrent neural networks (RNN) architecture.
4. The method of claim 1 , further comprises:
generating confidence scores for each of the classified document type, the detected text boxes, the extracted text information, and an output of a name validator;
assigning predefined weights to each of the generated confidence scores; and
aggregating the weighted confidence scores to compute a cumulative confidence score for the extracted text information.
5. A system for classification of a document and text information extraction of the document, the system comprising:
a memory storing instructions;
one or more Input/Output (I/O) interfaces; and
one or more hardware processors coupled to the memory via the one or more I/O interfaces, wherein the one or more hardware processors are configured by the instructions to:
receive a scanned image of the document;
detect a Region of Interest (ROI) in the scanned image by marking a ROI bounding box on the scanned image using a text detection engine based-ROI technique, wherein the text detection engine based-ROI technique locates one or more corner coordinates of the ROI bounding box defining the ROI;
classify the ROI into a document type among a plurality of document types using a trainable Deep Learning based multi-layered Neural Network (NN) classification model, wherein training of the trainable Deep Learning based multi-layered NN classification model comprises:
generating sample images as training data; and
performing pre-processing of the training data, augmentation of the training data, and data annotation of the training data to train the trainable Deep Learning based multi-layered NN classification model, wherein
the trainable Deep Learning based multi-layered NN classification model includes an architecture that is configurable based on requirements, and
a complexity of the architecture of the trainable Deep Learning based multi-layered NN classification model is varied based on the training data;
apply multistage pre-processing on the classified ROI to remove a background noise, wherein the multistage pre-processing comprises:
reading the ROI into a Red Green Blue (RGB) color space and flattening the ROI;
performing unsupervised clustering by applying K-means clustering on a plurality of pixels of the ROI in the RGB color space to generate a plurality of color clusters;
obtaining a plurality of centroids from each of the plurality of color clusters, wherein each centroid of the plurality of centroids represents a unique mean color associated with each of the plurality of clusters;
converting the plurality of centroids from the RGB color space to a Hue Saturation Value (HSV) space;
generating a plurality of color masks corresponding to the plurality of clusters, wherein each color mask of the plurality of color masks is generated based on a) the unique mean color associated with a centroid among the plurality of centroids and b) a range of the HSV color space defined around the centroid, wherein each of the plurality of color masks is generated such that each color mask sets:
a plurality first specific of pixels of the plurality of pixels of the centroid having HSV values within the HSV range to a first value, and
a plurality of second specific pixels of the plurality of pixels to a second value;
applying each of the plurality of color masks to the ROI to obtain a plurality of binary ROI images, wherein each of the plurality of binary ROI image comprises one or more contours indicating spatial locations of one or more pixels among the plurality of pixels in the ROI that belong to the unique mean color of the centroid;
identifying in each of the plurality of binary ROI images, one or more contours of interest from the plurality of contours, wherein the identified one or more contours of interest are a) closed contours and b) the identified one or more contours of interest has a size above a predefined contour size;
performing subtraction of the one or more identified contours of interest from the ROI in accordance to spatial positions of the plurality of pixels of the one or more identified contours of interest, wherein the subtraction eliminates the background noise while retaining information of interest to generate a first level pre-processed image; and
applying thresholding on the first level pre-processed image to obtain a second level pre-processed image using a threshold value derived dynamically from a histogram of the first level pre-processed image, wherein the threshold value is derived dynamically by:
calculating the histogram of the first level pre processed image;
determining average pixel intensity of the first level pre-processed image and a left most peak value and a right most peak value in the histogram; and
calculating the threshold value by averaging the left most peak value and the right most peak value;
apply a text detection technique on the second level pre-processed image to mark a plurality of bounding boxes around text information in the second level pre-processed image, wherein
each of the plurality of bounding boxes are identified by one or more spatial positions defined by the one or more corner coordinates and a corresponding height and a corresponding width,
one or more bounding boxes of the plurality of bounding boxes are clubbed based on a spatial proximity criteria,
the spatial proximity criteria analyzes a proximity among the one or more bounding boxes which has spatially relatable text to club the one or more bounding boxes, and
each of the one or more bounding boxes are clubbed to form relatable information into a single bounding;
extract text information from each of the plurality of bounding boxes by applying an Optical Character Recognition (OCR); and
determine contextual relationship among the extracted text information and refining the extracted text information based on configuration rules for the document type.
6. The system of claim 5 , wherein the configuration rules comprise a precompiled knowledge base that is referred to identify undesired text from the extracted text information, wherein the undesired text is discarded from the extracted text information, and wherein the precompiled knowledge base is composition of KEYWORDS and VALUE_CATEGORY.
7. The system of claim 5 , wherein the one or more hardware processors are further configured to correct Name values from the extracted data using an encoder-decoder model that utilizes Long short-term memory (LSTM) based Recurrent neural networks (RNN) architecture.
8. The system of claim 5 , wherein the one or more hardware processors are configured to:
generate confidence scores for each of the classified document type, detected text boxes, the extracted text information, and an output of a name validator;
assign predefined weights to each of the generated confidence scores; and
aggregating the weighted confidences score to compute a cumulative confidence score for the extracted text information.
9. One or more non-transitory machine-readable information storage mediums comprising one or more instructions, which when executed by one or more hardware processors causes:
receiving a scanned image of a document;
detecting a Region of Interest (ROI) in the scanned image by marking a ROI bounding box on the scanned image using a text detection engine based-ROI technique, wherein the text detection engine based-ROI technique locates one or more corner coordinates of the ROI bounding box defining the ROI;
classifying the ROI into a document type among a plurality of document types using a trainable Deep Learning based multi-layered Neural Network (NN) classification model, wherein training of the trainable Deep Learning based multi-layered NN classification model comprises:
generating sample images as training data; and
performing pre-processing of the training data, augmentation of the training data, and data annotation of the training data to train the trainable Deep Learning based multi-layered NN classification model, wherein
the trainable Deep Learning based multi-layered NN classification model includes an architecture that is configurable based on requirements, and
a complexity of the architecture of the trainable Deep Learning based multi-layered NN classification model is varied based on the training data;
applying multistage pre-processing on the classified ROI to remove a background noise, wherein the multistage pre-processing comprises:
reading the ROI into a Red Green Blue (RGB) color space and flattening the ROI;
performing unsupervised clustering by applying K-means clustering on a plurality of pixels of the ROI in the RGB color space to generate a plurality of color clusters;
obtaining a plurality of centroids from each of the plurality of color clusters, wherein each centroid of the plurality of centroids represents a unique mean color associated with each of the plurality of clusters;
converting the plurality of centroids from the RGB color space to a Hue Saturation Value (HSV) space;
generating a plurality of color masks corresponding to the plurality of clusters, wherein each color mask of the plurality of color masks is generated based on a) the unique mean color associated with a centroid among the plurality of centroids and b) a range of the HSV color space defined around the centroid, wherein each of the plurality of color masks is generated such that each color mask sets:
a plurality first specific of pixels of the plurality of pixels of the centroid having HSV values within the HSV range to a first value, and
a plurality of second specific pixels of the plurality of pixels to a second value;
applying each of the plurality of color masks to the ROI to obtain a plurality of binary ROI images, wherein each of the plurality of binary ROI image comprises one or more contours indicating spatial locations of one or more pixels among the plurality of pixels in the ROI that belong to the unique mean color of the centroid;
identifying in each of the plurality of binary ROI images, one or more contours of interest from the plurality of contours, wherein the identified one or more contours of interest are a) closed contours and b) the identified one or more contours of interest has a size above a predefined contour size;
performing subtraction of the one or more identified contours of interest from the ROI in accordance to spatial positions of the plurality of pixels of the identified one or more contours of interest, wherein the subtraction eliminates the background noise while retaining information of interest to generate a first level pre-processed image; and
applying thresholding on the first level pre-processed image to obtain a second level pre-processed image using a threshold value derived dynamically from a histogram of the first level pre-processed image, wherein the threshold value is derived dynamically by:
calculating the histogram of the first level pre processed image;
determining average pixel intensity of the first level pre-processed image and a left most value and a right most peak value in the histogram; and
calculating the threshold value by averaging the left most and the right most peak values;
applying a text detection technique on the second level pre-processed image to mark a plurality of bounding boxes around text information in the second level pre-processed image, wherein
each of the plurality of bounding boxes are identified by one or more spatial positions defined by the or more corner coordinates and a corresponding height and a corresponding width,
one or more bounding boxes of the plurality of bounding boxes are clubbed based on a spatial proximity criteria,
the spatial proximity criteria analyzes a proximity among the one or more bounding boxes which has spatially relatable text to club the one or more bounding boxes, and
each of the one or more bounding boxes are clubbed to form relatable information into a single bounding;
extracting text information from each of the plurality of bounding boxes by applying an Optical Character Recognition (OCR); and
determining contextual relationship among the extracted text information and refining the extracted text information based on configuration rules for the document type.