Granted Patent B2

US 12,423,937 · App. 18/166,004 · Granted Sep 23, 2025

Inventors: Yu Zui You (Ningbo, CN); Zhan Peng Huo (Beijing, CN); Jun Zhu (Shanghai, CN); Kuo-Liang Chou (New Taipei, TW); Xuan Feng (Beijing, CN); Jun Hao (Dalian, CN)

Assignee: International Business Machines Corporation

G06V10/20G06V10/774

Abstract

Claims (73)

1. A method for performing automated pre-processing of data for a machine-learning based prediction system, the method comprising:

receiving a plurality of raw data sets;

receiving a plurality of processed data sets, wherein each of the plurality of processed data sets corresponds to one of the plurality of raw data sets;

generating a plurality of pre-processing templates based on the plurality of raw data sets and the processed data set, wherein generating the plurality of pre-processing templates comprises: detecting and extracting attributes from the plurality of processed data sets;

comparing the plurality of processed data sets to the corresponding plurality of raw data sets;

and learning data processing methods used by data scientists;

receiving an input data set;

generating, for each of the plurality of pre-processing templates, a matching score for the input data set, wherein the matching score is calculated by executing a computer-implemented matching algorithm that, for each column in the input data set:

automatically extracting and comparing the column attribute and header to corresponding columns in the pre-processing template using a similarity metric;

applying user-assigned weights associated with each template column to the similarity results to generate a weighted column score; and

combining the weighted column scores for all columns in the input data set using a predetermined aggregation function to produce the matching score for the template;

selecting one of the plurality of pre-processing templates based on the matching score;

pre-processing the input data set using the selected pre-processing template; and

providing the pre-processed input data set to the machine learning based prediction system.

2. The method of claim 1 , wherein each of the plurality of raw data sets includes a plurality of columns of data and wherein generating the plurality of pre-processing templates includes detecting an attribute for each of the plurality of columns of data.

3. The method of claim 1 , wherein each of the plurality of processed data sets includes a plurality of columns of data and wherein generating the plurality of pre-processing templates includes detecting a user assigned attribute for at least one of the plurality of columns of data.

4. The method of claim 1 , wherein generating the matching score for one of the plurality of pre-processing templates comprises:

identifying an attribute for each column of the input data set;

identifying a corresponding column of the pre-processing template based on the attribute;

generate a column matching score for each column of the input data set to the corresponding column, wherein the column matching score is based on analyzing column attributes and headers; and

creating the matching score by combining the column matching score for each column of the input data set.

5. The method of claim 4 , wherein combining the column matching score for each column of the input data set includes calculating a weighted combination.

6. The method of claim 5 , wherein each column of the pre-processing template is assigned a weight that corresponds to the attribute type of the column.

7. The method of claim 1 , wherein selecting one of the plurality of pre-processing templates includes selecting the one of the plurality of pre-processing templates having a highest matching score.

8. The method of claim 1 , further comprising providing the input data set to a user for manual pre-processing based on a determination that none of the matching scores are above a threshold minimum score.

9. A computing system having a memory having computer readable instructions and one or more processors for executing the computer readable instructions, the computer readable instructions controlling the one or more processors to perform operations comprising:

receiving a plurality of raw data sets;

receiving a plurality of processed data sets, wherein each of the plurality of processed data sets corresponds to one of the plurality of raw data sets;

generating a plurality of pre-processing templates based on the plurality of raw data sets and the processed data set, wherein generating the plurality of pre-processing templates comprises: detecting and extracting attributes from the plurality of processed data sets;

comparing the plurality of processed data sets to the corresponding plurality of raw data sets;

and learning data processing methods used by data scientists;

receiving an input data set;

generating, for each of the plurality of pre-processing templates, a matching score for the input data set,

wherein the matching score is calculated by executing a computer-implemented matching algorithm that, for each column in the input data set:

automatically extracting and comparing the column attribute and header to corresponding columns in the pre-processing template using a similarity metric;

applying user-assigned weights associated with each template column to the similarity results to generate a weighted column score; and

combining the weighted column scores for all columns in the input data set using a predetermined aggregation function to produce the matching score for the template;

selecting one of the plurality of pre-processing templates based on the matching score;

pre-processing the input data set using the selected pre-processing template; and

providing the pre-processed input data set to the machine learning based prediction system.

10. The computing system of claim 9 , wherein each of the plurality of raw data sets includes a plurality of columns of data and wherein generating the plurality of pre-processing templates includes detecting an attribute for each of the plurality of columns of data.

11. The computing system of claim 9 , wherein each of the plurality of processed data sets includes a plurality of columns of data and wherein generating the plurality of pre-processing templates includes detecting a user assigned attribute for at least one of the plurality of columns of data.

12. The computing system of claim 9 , wherein generating the matching score for one of the plurality of pre-processing templates comprises:

identifying an attribute for each column of the input data set;

identifying a corresponding column of the pre-processing template based on the attribute;

generate a column matching score for each column of the input data set to the corresponding column, wherein the column matching score is based on analyzing column attributes and headers; and

creating the matching score by combining the column matching score for each column of the input data set.

13. The computing system of claim 12 , wherein combining the column matching score for each column of the input data set includes calculating a weighted combination.

14. The computing system of claim 13 , wherein each column of the pre-processing template is assigned a weight that corresponds to the attribute type of the column.

15. The computing system of claim 9 , wherein selecting one of the plurality of pre-processing templates includes selecting the one of the plurality of pre-processing templates having a highest matching score.

16. The computing system of claim 9 , wherein the operations further comprise providing the input data set to a user for manual pre-processing based on a determination that none of the matching scores are above a threshold minimum score.

17. A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to perform operations comprising:

receiving a plurality of raw data sets;

receiving a plurality of processed data sets, wherein each of the plurality of processed data sets corresponds to one of the plurality of raw data sets;

generating a plurality of pre-processing templates based on the plurality of raw data sets and the processed data set, wherein generating the plurality of pre-processing templates comprises: detecting and extracting attributes from the plurality of processed data sets;

comparing the plurality of processed data sets to the corresponding plurality of raw data sets;

and learning data processing methods used by data scientists;

receiving an input data set;

generating, for each of the plurality of pre-processing templates, a matching score for the input data set,

wherein the matching score is calculated by executing a computer-implemented matching algorithm that, for each column in the input data set:

automatically extracting and comparing the column attribute and header to corresponding columns in the pre-processing template using a similarity metric;

applying user-assigned weights associated with each template column to the similarity results to generate a weighted column score; and

combining the weighted column scores for all columns in the input data set using a predetermined aggregation function to produce the matching score for the template;

selecting one of the plurality of pre-processing templates based on the matching score;

pre-processing the input data set using the selected pre-processing template; and

providing the pre-processed input data set to the machine learning based prediction system.

18. The computer program product of claim 17 , wherein each of the plurality of raw data sets includes a plurality of columns of data and wherein generating the plurality of pre-processing templates includes detecting an attribute for each of the plurality of columns of data.

19. The computer program product of claim 17 , wherein each of the plurality of processed data sets includes a plurality of columns of data and wherein generating the plurality of pre-processing templates includes detecting a user assigned attribute for at least one of the plurality of columns of data.

20. The computer program product of claim 17 , wherein generating the matching score for one of the plurality of pre-processing templates comprises:

identifying an attribute for each column of the input data set;

identifying a corresponding column of the pre-processing template based on the attribute;

generate a column matching score for each column of the input data set to the corresponding column, wherein the column matching score is based on analyzing column attributes and headers; and

creating the matching score by combining the column matching score for each column of the input data set.

Assignments (1)

Continuity (1)

References Cited (17)