Granted Patent B2

US 12,265,516 · App. 18/080,627 · Granted Apr 1, 2025

Inventors: Stephen M. Thompson (Bonsall, CA); Iurii Vymenets (St. Petersburg, RU); Donghan Lee (Anaheim, CA); Markus Georg Lust (Freiburg, DE)

Assignee: TUNGSTEN AUTOMATION CORPORATION

G06F16/2282G06V10/70G06V30/412G06V30/413G06V30/414

Abstract

Claims (112)

1. A computer-implemented method for detecting and classifying columns of tables and/or tabular data arrangements within image data, the method comprising:

detecting one or more tables and/or one or more tabular data arrangements within the image data;

extracting the one or more tables and/or the one or more tabular data arrangements from the image data; and

classifying either:

a plurality of columns of the one or more extracted tables;

a plurality of columns of the one or more extracted tabular data arrangements; or

both the columns of the one or more extracted tables and the columns of the one or more extracted tabular data arrangements; and

wherein the classifying comprises:

performing a pairwise comparison of one or more feature vectors corresponding to columns of tables and/or tabular data arrangements in a training dataset to one or more feature vectors corresponding to columns of tables and/or tabular data arrangements in a test dataset; and

generating a pairwise similarity score for each pair of compared feature vectors; and

wherein each pairwise similarity score is a function of a number of common OCR elements between the corresponding pair of compared feature vectors.

2. The computer-implemented method as recited in claim 1 , wherein classifying the columns of the one or more extracted tables and/or the columns of the one or more extracted tabular data arrangements comprises:

training at least one machine learning model, using a training dataset representing a plurality of different types of tables and/or tabular data arrangements, wherein the classifying is based at least in part on OCR elements within the tables and/or the tabular data arrangements represented within the training dataset; and

outputting at least one trained column classification model based on the training, wherein each trained column classification model is configured to classify columns of a particular type of table or a particular type of tabular data arrangement represented within the training dataset; and

classifying, using the at least one trained column classification model, a plurality of columns of tables and/or tabular data arrangements represented by a test dataset.

3. The computer-implemented method as recited in claim 2 , wherein the training comprises building a plurality of feature vectors based on the training dataset;

wherein each feature vector corresponds to a unique column of one of the one or more tables and/or one of the one or more tabular data arrangements represented by the training dataset; and

wherein at least some of the feature vectors define an identity of one or more textual elements present in the corresponding column.

4. The computer-implemented method as recited in claim 2 , wherein the trained column classification model(s) each independently comprise:

a plurality of the feature vectors; and

an identification of some or all of the following:

one or more important single values present within the training dataset, wherein the important single values are optical character recognition (OCR) values;

one or more important column descriptor values present within the training dataset, wherein the important column descriptor values are OCR values;

one or more important value pairs present within the training dataset, wherein the important value pairs are OCR values;

one or more column descriptor important value pairs present within the training dataset, wherein the important value pairs are OCR values;

one or more half-important value pairs present within the training dataset, wherein the one or more half-important value pairs are OCR values;

one or more important value triplets present within the training dataset, wherein the one or more important value triplets are OCR values; and/or

sequence information regarding an order and/or a position of adjacent columns of the tables and/or the tabular data arrangements represented by the training dataset, wherein the sequence information comprises labels of one or more columns positioned to the left of a particular class of column in the training dataset, and/or labels of one or more columns positioned to the right of the particular class of column in the training dataset.

5. The computer-implemented method as recited in claim 1 , wherein the classifying further comprises building, based on the test dataset, the one or more feature vectors; and

wherein at least some of the feature vectors define an identity of one or more textual elements present in the corresponding column.

6. The computer-implemented method as recited in claim 1 , wherein classifying the columns of the one or more extracted tables and/or the columns of the one or more extracted tabular data arrangements further comprises:

training at least one machine learning model, using a training dataset representing a plurality of different types of tables and/or tabular data arrangements; and

outputting at least one trained column classification model based on the training, wherein each trained column classification model is configured to classify columns of a particular type of table or a particular type of tabular data arrangement represented within the training dataset;

classifying, using the at least one trained column classification model, a plurality of columns of tables and/or tabular data arrangements represented by a test dataset;

building, based on the test dataset, a plurality of feature vectors, wherein each feature vector: corresponds to a subregion of a unique column of one of the one or more tables and/or one of the one or more tabular data arrangements represented by the test dataset;

and

adjusting at least some of the similarity scores using at least one weight, wherein the at least one weight is determined based on whether the corresponding subregion(s) comprise:

one or more important single values present within the training set;

one or more important column descriptor values present within the training set;

one or more important value pairs present within the training dataset;

one or more column descriptor important value pairs present within the training set;

one or more half-important value pairs present within the training set; and/or

one or more important value triplets present within the training set.

7. The computer-implemented method as recited in claim 1 , wherein at least some of the plurality of columns of the one or more extracted tables and/or at least some of the plurality of columns of the one or more extracted tabular data arrangements are classified as either “columns of interest” or “columns not of interest”.

8. The computer-implemented method as recited in claim 1 , wherein the classifying comprises determining:

a relevancy of some or all of the columns of the one or more extracted tables; and/or

a relevancy of some or all of the columns of the one or more extracted tabular data arrangements.

9. The computer-implemented method as recited in claim 1 , wherein the classifying comprises identifying column descriptors that fully qualify at least some of the plurality of columns of the one or more extracted tables and/or at least some of the plurality of columns of the one or more extracted tabular data arrangements.

10. The computer-implemented method as recited in claim 1 , wherein the classifying comprises identifying one or more synonym sets within some or all of the columns of the one or more extracted tables and/or some or all of the columns of the one or more extracted tabular data arrangements.

11. The computer-implemented method as recited in claim 1 , wherein the classifying is based on unsupervised learning.

12. A computer program product for detecting and classifying tables and/or tabular data arrangements within image data, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, and wherein the program instructions configured to cause a computer to perform a method comprising:

detecting one or more tables and/or one or more tabular data arrangements within the image data;

extracting the one or more tables and/or the one or more tabular data arrangements from the image data; and

classifying either:

columns of the one or more extracted tables;

columns of the one or more extracted tabular data arrangements; or

both the columns of the one or more extracted tables and the columns of the one or more extracted tabular data arrangements; and

wherein the classifying comprises:

performing a pairwise comparison of one or more feature vectors corresponding to columns of tables and/or tabular data arrangements in a training dataset to one or more feature vectors corresponding to columns of tables and/or tabular data arrangements in a test dataset; and

generating a pairwise similarity score for each pair of compared feature vectors; and

wherein each pairwise similarity score is a function of a number of common OCR elements between the corresponding pair of compared feature vectors.

13. The computer program product as recited in claim 12 , wherein classifying the columns of the one or more extracted tables and/or the columns of the one or more extracted tabular data arrangements comprises:

training at least one machine learning model, using a training dataset representing a plurality of different types of tables and/or tabular data arrangements, wherein the classifying is based at least in part on OCR elements within the tables and/or the tabular data arrangements represented within the training dataset; and

outputting at least one trained column classification model based on the training, wherein each trained column classification model is configured to classify columns of a particular type of table or a particular type of tabular data arrangement represented within the training dataset; and

classifying, using the at least one trained column classification model, a plurality of columns of tables and/or tabular data arrangements represented by a test dataset.

14. The computer program product as recited in claim 13 , wherein the training comprises building a plurality of feature vectors based on the training dataset;

wherein each feature vector corresponds to a unique column of one of the one or more tables and/or one of the one or more tabular data arrangements represented by the training dataset; and

wherein at least some of the feature vectors define an identity of one or more textual elements present in the corresponding column.

15. The computer program product as recited in claim 13 , wherein the trained column classification model(s) each independently comprise:

a plurality of the feature vectors; and

an identification of some or all of the following:

one or more important single values present within the training dataset, wherein the important single values are optical character recognition (OCR) values;

one or more important column descriptor values present within the training dataset, wherein the important column descriptor values are OCR values;

one or more important value pairs present within the training dataset, wherein the important value pairs are OCR values;

one or more column descriptor important value pairs present within the training dataset, wherein the important value pairs are OCR values;

one or more half-important value pairs present within the training dataset, wherein the one or more half-important value pairs are OCR values;

one or more important value triplets present within the training dataset, wherein the one or more important value triplets are OCR values; and/or

sequence information regarding an order and/or a position of adjacent columns of the tables and/or the tabular data arrangements represented by the training dataset, wherein the sequence information comprises labels of one or more columns positioned to the left of a particular class of column in the training dataset, and/or labels of one or more columns positioned to the right of the particular class of column in the training dataset.

16. The computer program product as recited in claim 12 , wherein the classifying further comprises building, based on the test dataset, the one or more feature vectors; and

wherein at least some of the feature vectors define an identity of one or more textual elements present in the corresponding column.

17. The computer program product as recited in claim 12 , wherein classifying the columns of the one or more extracted tables and/or the columns of the one or more extracted tabular data arrangements further comprises:

training at least one machine learning model, using a training dataset representing a plurality of different types of tables and/or tabular data arrangements; and

outputting at least one trained column classification model based on the training, wherein each trained column classification model is configured to classify columns of a particular type of table or a particular type of tabular data arrangement represented within the training dataset;

classifying, using the at least one trained column classification model, a plurality of columns of tables and/or tabular data arrangements represented by a test dataset;

building, based on the test dataset, a plurality of feature vectors, wherein each feature vector: corresponds to a subregion of a unique column of one of the one or more tables and/or one of the one or more tabular data arrangements represented by the test dataset;

and

adjusting at least some of the similarity scores using at least one weight, wherein the at least one weight is determined based on whether the corresponding subregion(s) comprise:

one or more important single values present within the training set;

one or more important column descriptor values present within the training set;

one or more important value pairs present within the training dataset;

one or more column descriptor important value pairs present within the training set;

one or more half-important value pairs present within the training set; and/or one or more important value triplets present within the training set.

18. A computer-implemented method for detecting and classifying columns of tables and/or tabular data arrangements within image data, the method comprising:

detecting one or more tables and/or one or more tabular data arrangements within the image data;

extracting the one or more tables and/or the one or more tabular data arrangements from the image data; and

classifying either:

a plurality of columns of the one or more extracted tables;

a plurality of columns of the one or more extracted tabular data arrangements; or

both the columns of the one or more extracted tables and the columns of the one or more extracted tabular data arrangements; and

wherein the classifying comprises:

training at least one machine learning model, using a training dataset representing a plurality of different types of tables and/or tabular data arrangements; and

outputting at least one trained column classification model based on the training, wherein each trained column classification model is configured to classify columns of a particular type of table or a particular type of tabular data arrangement represented within the training dataset;

classifying, using the at least one trained column classification model, a plurality of columns of tables and/or tabular data arrangements represented by a test dataset;

building, based on the test dataset, a plurality of feature vectors, wherein each feature vector: corresponds to a subregion of a unique column of one of the one or more tables and/or one of the one or more tabular data arrangements represented by the test dataset

comparing each of the feature vectors of the trained classification model to a plurality of feature vectors built based on the training dataset;

generating a pairwise similarity score based on each comparison; and

adjusting at least some of the similarity scores using at least one weight, wherein the at least one weight is determined based on whether the corresponding subregion(s) comprise:

one or more important single values present within the training set;

one or more important column descriptor values present within the training set;

one or more important value pairs present within the training dataset;

one or more column descriptor important value pairs present within the training set;

one or more half-important value pairs present within the training set; and/or one or more important value triplets present within the training set.

Assignments (2)

Continuity (3)

References Cited (72)