Granted Patent B2

US 12,443,614 · App. 18/114,075 · Granted Oct 14, 2025

Inventors: Robert Raymond Lindner (Fitchburg, WI); Carlos Vera-Ciro (Madison, WI)

Assignee: VEDA Data Solutions, Inc.

G06F16/248G06F16/2237

Abstract

Claims (94)

1. A method for determining which labels correspond to columns of a data file comprising columns and rows, the method comprising:

receiving the data file containing a plurality of fields organized as a table with a plurality of columns and a plurality of rows, the data file having inconsistent labeling for the columns;

training a model to detect a label based on data within the columns, wherein the model is trained using:

a number of Monte Carlo training sets having sample data files, and

rules for common types of demographic information;

for respective columns in the plurality of columns, selecting, from a plurality of consistent labels, a label corresponding to the respective columns of the data file using the model to detect the label based on data within the column;

for respective first and second columns in the plurality of columns, determining a column score indicating a likelihood that the first column has the label corresponding to the first column given that the second column has the label corresponding to the second column;

determining, based on the column score, a placement score indicating a likelihood that labels from the plurality of consistent labels corresponding to the respective columns are correct, wherein for all of the labels the determining the placement score further comprises,

determining a first frequency of a first label amongst the labels,

determining a second frequency of a second label amongst the labels,

determining a frequency score indicating a likelihood that the first label occurs at the first frequency given that the second label occurs at the second frequency, and

determining the placement score based on the frequency score;

adjusting the labels corresponding to each of the respective first and second columns based on the placement score;

repeating, until the placement score converges, steps of:

for the respective first and second columns in the plurality of columns, determining the column score indicating the likelihood that the first column has the label corresponding to the first column given that the second column has the label corresponding to the second column;

determining, based on the column score, the placement score indicating the likelihood that the labels corresponding to the respective columns are correct, wherein for all of the labels, the determining the placement score further comprises:

determining the first frequency of the first label amongst the labels,

determining the second frequency of the second label amongst the labels,

determining the frequency score indicating the likelihood that the first label occurs at the first frequency given that the second label occurs at the second frequency, and

determining the placement score based on the frequency score; and

adjusting the labels correspond to each of the respective first and second columns based on the placement score; and

generating, by a column detector on a computing device, a reformatted data file based on the adjusted labels.

2. The method of claim 1 , wherein the determining the frequency score comprises looking up a number of times the frequency score has shown up in a historical matrix.

3. The method of claim 1 , further comprising repeating steps of:

determining the first frequency of the first label among the labels,

determining the second frequency of the second label among the labels, and

determining the frequency score indicating the likelihood that the first label occurs at the first frequency given that the second label occurs at the second frequency,

wherein the steps are repeated for respective columns in the plurality of columns to determine a plurality of frequency scores, and wherein the determining the placement score further comprises determining the placement score based on the plurality of frequency scores.

4. The method of claim 1 , further comprising for respective columns in the plurality of columns, determining an evidence score indicating a likelihood that the label is correct, wherein the determining the placement score further comprises determining the placement score based on the evidence scores.

5. The method of claim 4 , wherein the data file lists health care providers.

6. The method of claim 1 , wherein the determining the placement score comprises looking up a number of times the placement score has shown up in a historical matrix.

7. The method of claim 1 , wherein the data file stores demographic information.

8. A non-transitory computer-readable device having instructions stored thereon that, when executed by at least one computing device, cause the at least one computing device to perform operations for determining which labels correspond to columns of a data file comprising columns and rows, the operations comprising:

receiving the data file containing a plurality of fields organized as a table with a plurality of columns and a plurality of rows, the data file having inconsistent labeling for the columns;

training a model to detect a label based on data within the columns, wherein the model is trained using:

a number of Monte Carlo training sets having sample data files, and

rules for common types of demographic information;

for respective columns in the plurality of columns, selecting, from a plurality of consistent labels, a label corresponding to the respective columns of the data file using the model to detect the label based on data within the column;

for respective first and second columns in the plurality of columns, determining a column score indicating a likelihood that the first column has the label corresponding to the first column given that the second column has the label corresponding to the second column;

determining, based on the column score, a placement score indicating a likelihood that labels from the plurality of consistent labels corresponding to the respective columns are correct, wherein for all of the labels, the operations further comprise:

determining a first frequency of a first label amongst the labels,

determining a second frequency of a second label amongst the labels,

determining a frequency score indicating a likelihood that the first label occurs at the first frequency given that the second label occurs at the second frequency, and

determining the placement score based on the frequency score;

adjusting the labels corresponding to each of the respective columns based on the placement score;

repeating, until the placement score converges, steps of:

for the respective first and second columns in the plurality of columns, determining the column score indicating the likelihood that the first column has the label corresponding to the first column given that the second column has the label corresponding to the second column;

determining, based on the column score, the placement score indicating the likelihood that the labels corresponding to the respective columns are correct, wherein for all of the labels, the operations further comprise:

determining the first frequency of the first label amongst the labels,

determining the second frequency of the second label amongst the labels,

determining the frequency score indicating the likelihood that the first label occurs at the first frequency given that the second label occurs at the second frequency, and

determining the placement score based on the frequency score; and

adjusting the labels correspond to each of the respective first and second columns based on the placement score; and

generating, by a column detector on the non-transitory computer-readable device, a reformatted data file based on the adjusted labels.

9. The non-transitory computer-readable device of claim 8 , wherein the determining the frequency score comprises looking up a number of times the frequency score has shown up in a historical matrix.

10. The non-transitory computer-readable device of claim 8 , wherein the operations further comprise repeating steps of:

determining the first frequency of the first label among the labels,

determining the second frequency of the second label among the labels, and

determining the frequency score indicating the likelihood that the first label occurs at the first frequency given that the second label occurs at the second frequency,

wherein the steps are repeated for respective columns in the plurality of columns to determine a plurality of frequency scores, and wherein the determining the placement score further comprises determining the placement score based on the plurality of frequency scores.

11. The non-transitory computer-readable device of claim 8 , wherein the operations further comprise for respective columns in the plurality of columns, determining an evidence score indicating a likelihood that the label is correct, wherein the determining the placement score further comprises determining the placement score based on the evidence scores.

12. The non-transitory computer-readable device of claim 8 , wherein the determining the placement score comprises looking up the number of times the placement score has shown up in a historical matrix.

13. The non-transitory computer-readable device of claim 8 , wherein the data file stores demographic information.

14. The non-transitory computer-readable device of claim 13 , wherein the data file lists health care providers.

15. The non-transitory computer-readable device of claim 8 , wherein the operations further comprise repeating, until an optima is found, the steps of:

for the respective first and second columns in the plurality of columns, determining the column score indicating the likelihood that the first column has the label corresponding to the first column given that the second column has the label corresponding to the second column,

determining, based on the column score, the placement score indicating the likelihood that the labels corresponding to the respective columns are correct, and

adjusting the labels corresponding to each of the respective first and second columns.

16. The non-transitory computer-readable device of claim 8 , wherein the adjusting which of the labels correspond to each of the respective columns to obtain the reformatted data file comprises applying a nonlinear optimization technique.

17. The non-transitory computer-readable device of claim 16 , wherein the nonlinear optimization technique includes at least one of hill climbing, stochastic hill climbing, simulated, local beams search, or genetic algorithms.

18. A system for determining which labels correspond to columns of a data file comprising columns and rows, comprising:

a processor; and

a memory with instructions stored thereon that when executed by the processor, cause the processor to:

receive the data file containing a plurality of fields organized as a table with a plurality of columns and a plurality of rows, the data file having inconsistent labeling for the columns;

train a model to detect a label based on data within the columns, wherein the model is trained using:

a number of Monte Carlo training sets having sample data files, and

rules for common types of demographic information;

for respective columns in the plurality of columns, select, from a plurality of consistent labels, a label corresponding to the respective columns of the data file using the model to detect the label based on data within the column;

for respective first and second columns in the plurality of columns, determine a column score indicating a likelihood that the first column has the label corresponding to the first column given that the second column has the label corresponding to the second column;

determine, based on the column score, a placement score indicating a likelihood that labels from the plurality of consistent labels corresponding to the respective columns are correct, wherein for all of the labels, the instructions further cause the processor to:

determine a first frequency of a first label amongst the labels,

determine a second frequency of a second label amongst the labels,

determine a frequency score indicating a likelihood that the first label occurs at the first frequency given that the second label occurs at the second frequency, and

determine the placement score based on the frequency score;

adjust the labels corresponding to each of the respective columns based on the placement score;

repeat, until the placement score converges, steps of:

for the respective first and second columns in the plurality of columns, determine the column score indicating the likelihood that the first column has the label corresponding to the first column given that the second column has the label corresponding to the second column;

determine, based on the column score, the placement score indicating the likelihood that the labels corresponding to the respective columns are correct, wherein for all of the labels, the instructions further cause the processor to:

determine the first frequency of the first label amongst the labels,

determine the second frequency of the second label amongst the labels,

determine the frequency score indicating the likelihood that the first label occurs at the first frequency given that the second label occurs at the second frequency, and

determine the placement score based on the frequency score; and

adjust the labels correspond to each of the respective first and second columns based on the placement score; and

generate, by a column detector on the system, a reformatted data file based on the adjusted labels.

Assignments (3)

Continuity (2)

References Cited (11)