Methods and apparatus for data preprocessing
Methods, apparatus, and/or computer program products for data preprocessing are provided. The invention utilizes selective modular processes that automate data cleaning tasks and provide an initial analysis of datasets using, among other things, Natural Language Processing and Machine Learning methods. The apparatus and methods preprocess text data, clean database data with minimal human intervention, and provide automatic analytical results that include basic data cleaning, spell check, clustering, outlier detection, and natural language processing. Each modular process can be selectively switched on or off based on user input, preprogrammed instructions, or AI inputs. Additionally, any data format may be input as the apparatus and methods are data agnostic and, therefore, are not designed specifically for one type of data or theme of data. The inventive apparatus and methods dramatically reduce preparation time on the front end of data projects.
1 . A method of data preprocessing comprising:
providing a plurality of data agnostic preprocessing modules within a data preprocessing processor each of the plurality of data agnostic preprocessing modules having respective data processing functionalities, where each module is configured to be able to be selectively applied for preprocessing of input text data files and allow input and processing of a plurality of types of text data;
receiving, with the data preprocessing processor via a graphical user interface (GUI), an aggregate selection data for selecting which of the plurality of data preprocessing modules are to be applied for data preprocessing, the aggregate selection data comprising a selection of at least one or more of the plurality of data agnostic preprocessing modules to be applied for the preprocessing of input text data files;
loading at least one input data file with the data preprocessing processor, the at least one input text data file comprising one or more data entries;
executing the one or more selected data agnostic preprocessing modules with the data preprocessing processor based on the received aggregate selection data on the at least one input text data file for preprocessing of the at least one input text data file, wherein at least one of the one or more data agnostic preprocessing modules are further configured to utilize computer-Implemented Natural Language Processing and Machine Learning to perform a data preprocessing function on the at least one input data file; and
outputting with the data preprocessing processor a primary output data file comprising preprocessed data resulting from the execution of the one or more selected data agnostic preprocessing modules on the at least one input text data file.
2 . The method of claim 1 , further comprising importing a plurality of libraries before executing the one or more selected data agnostic preprocessing modules.
3 . The method of claim 1 , wherein the plurality of data agnostic preprocessing modules include remove duplicate lines, drop empty columns, true unique ID, cleaned dates, multi-punch breakout, mandatory fields, minimum percent filled, data type mismatch, unique code count, data dictionary, spell check, clustering, outlier detection, basic statistics, calculation tool, time period summary, one to one analysis, character prefill, semantic trend, and best predictors modules.
4 . The method of claim 3 , wherein the remove duplicate lines module deletes lines that are identical across all fields, leaving one instance of a record;
wherein the true unique ID module returns a CSV file when the module has been executed;
wherein the drop empty columns module removes columns that are completely empty;
wherein the cleaned dates module conforms all dates in a designated column to one format;
wherein the multi-punch breakout module takes a designated column name and a delimiting character and breaks the column apart by the character;
wherein the mandatory fields module returns a file with the percentage of missing information and the total of rows missing;
wherein the minimum percent filled module checks if all columns are filled to a defined percentage;
wherein the data type mismatch module writes a file indicating the percent make up of each column in terms of data types;
wherein the unique code count module returns a count of the unique values from a selected column;
wherein the data dictionary module normalizes text fields within the data by replacing all variations and acronyms of a word or set of words with a single word or set of words;
wherein the spell check module identifies and fixes basic typographical errors within text cells;
wherein the clustering module analyzes the dataset via a k-means clustering module and a machine learning model that links similar records in clusters;
wherein the outlier detection module identifies records with values that are outliers for selected columns and returns a printout of the values with occurrences sorted least to greatest, potential outliers at different stages of the module, and a chart of occurrences of those values;
wherein the basic statistics module calculates the 25th percentile, mean, and 75th percentile value, median, mode, and standard deviation of all numeric columns;
wherein the calculation tool module calculates basic operations of columns with multiplication, division, addition, and subtraction;
wherein the time period summary module plots values of a designated column in either a month over month or year over year graph, given that the dataset has dates for every record;
wherein the one to one analysis module checks that values of one selected column corresponds to only one value of another selected column;
wherein the character prefill module prepends a selected character to all values in a column a varied number of times, normalizing the length of the values to a user selected length;
wherein the semantic trend module identifies main points of individual records for discerning general trends of the dataset; and
wherein the best predictors module selects fields that have a higher correlation to a designated field.
5 . The method of claim 1 , wherein the at least one of the one or more data agnostic preprocessing modules configured to utilize Machine Learning includes a clustering module that analyzes the at least one input text data file using a k-means clustering module and a machine learning model that links similar records into clusters, returns at least one image where data has been fitted into clusters, and assigns the clusters to a closest centroid.
6 . A data preprocessing apparatus comprising:
a non-transitory computer-readable medium with instructions thereon; and
one or more processors, when executing the instructions, configured to:
provide a plurality of data agnostic preprocessing modules with a data preprocessing processor implemented by the one or more processors, each of the plurality of data agnostic preprocessing modules having respective data processing functionalities, where each module is configured to be able to be selectively applied for preprocessing of input text data files and allow input and processing of a plurality of types of data;
execute a graphical user interface with the data preprocessing processor to receive an aggregate selection data for selecting which of the plurality of data preprocessing modules are to be applied for data preprocessing, the aggregate selection data comprising a selection of at least one or more of the plurality of data agnostic preprocessing modules selected to be applied for the preprocessing of input text data files;
load at least one input text data file with the data preprocessing processor, the at least one input text data file comprising one or more data entries;
execute the one or more selected data agnostic preprocessing modules with the data preprocessing processor based on the received aggregate selection data on the at least one input text data file for preprocessing of the at least one input text data file, wherein at least one of the one or more data agnostic preprocessing modules are further configured to utilize computer-implemented Natural Language Processing and Machine Learning to perform a data preprocessing function on the at least one input text data file; and
output with the data preprocessing processor a primary output data file comprising preprocessed data resulting from the execution of the one or more selected data agnostic preprocessing modules on the at least one input text data file.
7 . The apparatus of claim 6 , further comprising importing a plurality of libraries before executing the modules.
8 . The apparatus of claim 6 , wherein the plurality of data agnostic preprocessing modules include remove duplicate lines, drop empty columns, true unique ID, cleaned dates, multi-punch breakout, mandatory fields, minimum percent filled, data type mismatch, unique code count, data dictionary, spell check, clustering, outlier detection, basic statistics, calculation tool, time period summary, one to one analysis, character prefill, semantic trend, and best predictors modules.
9 . The apparatus of claim 8 , wherein the remove duplicate lines module deletes lines that are identical across all fields, leaving one instance of a record;
wherein the true unique ID module returns a CSV file when the module has been executed;
wherein the drop empty columns module removes columns that are completely empty;
wherein the cleaned dates module conforms all dates in a designated column to one format;
wherein the multi-punch breakout module takes a designated column name and a delimiting character and breaks the column apart by the character;
wherein the mandatory fields module returns a file with the percentage of missing information and the total of rows missing;
wherein the minimum percent filled module checks if all columns are filled to a defined percentage;
wherein the data type mismatch module writes a file indicating the percent make up of each column in terms of data types;
wherein the unique code count module returns a count of the unique values from a selected column;
wherein the data dictionary module normalizes text fields within the data by replacing all variations and acronyms of a word or set of words with a single word or set of words;
wherein the spell check module identifies and fixes basic typographical errors within text cells;
wherein the clustering module analyzes the dataset via a k-means clustering module and a machine learning model that links similar records in clusters;
wherein the outlier detection module identifies records with values that are outliers for selected columns and returns a printout of the values with occurrences sorted least to greatest, potential outliers at different stages of the module, and a chart of occurrences of those values;
wherein the basic statistics module calculates the 25th percentile, mean, and 75th percentile value, median, mode, and standard deviation of all numeric columns;
wherein the calculation tool module calculates basic operations of columns with multiplication, division, addition, and subtraction;
wherein the time period summary module plots values of a designated column in either a month over month or year over year graph, given that the dataset has dates for every record;
wherein the one to one analysis module checks that values of one selected column corresponds to only one value of another selected column;
wherein the character prefill module prepends a selected character to all values in a column a varied number of times, normalizing the length of the values to a user selected length;
wherein the semantic trend module identifies main points of individual records for discerning general trends of the dataset; and
wherein the best predictors module selects fields that have a higher correlation to a designated field.
10 . The data preprocessing apparatus of claim 6 , wherein the at least one of the one or more data agnostic preprocessing modules configured to utilize Machine Learning includes a clustering module that analyzes the at least one input text data file using a k-means clustering module and a machine learning model that links similar records into clusters, returns at least one image where data has been fitted into clusters, and assigns the clusters to a closest centroid.
11 . A computer program product comprising a non-transitory computer-readable storage medium containing computer program code, the computer program code when executed by one or more processors causes the one or more processors to perform operations, the computer program code comprising instructions to:
provide a plurality of data agnostic preprocessing modules with a data preprocessing processor implemented by the one or more processors, each of the plurality of data agnostic preprocessing modules having respective data processing functionalities, where each module is configured to be able to be selectively applied for preprocessing of input text data files and allow input and processing of a plurality of types of data;
execute a graphical user interface with the data preprocessing processor to receive an aggregate selection data for selecting which of the plurality of data preprocessing modules are to be applied for data preprocessing, the aggregate selection data comprising a selection of at least one or more of the plurality of data agnostic preprocessing modules selected to be applied for the preprocessing of input text data files;
load at least one input text data file comprising one or more data entries;
execute the one or more selected data agnostic preprocessing modules with the data preprocessing processor based on the received aggregate selection data on the at least one input text data file for preprocessing of the at least one input text data file, wherein at least one of the one or more data agnostic preprocessing modules are further configured to utilize computer-implemented Natural Language Processing and Machine Learning to perform a data preprocessing function on the at least one input text data file; and
output with the data preprocessing processor a primary output data file comprising preprocessed data resulting from the execution of the one or more selected data agnostic preprocessing modules on the at least one input text data file.
12 . The computer program product of claim 11 , further comprising instructions to import a plurality of libraries before executing the modules.
13 . The computer program product of claim 11 , wherein the modules comprise remove duplicate lines, drop empty columns, true unique ID, cleaned dates, multi-punch breakout, mandatory fields, minimum percent filled, data type mismatch, unique code count, data dictionary, spell check, clustering, outlier detection, basic statistics, calculation tool, time period summary, one to one analysis, character prefill, semantic trend, and best predictors modules.
14 . The computer program product of claim 13 , wherein the remove duplicate lines module deletes lines that are identical across all fields, leaving one instance of a record;
wherein the true unique ID module returns a CSV file when the module has been executed;
wherein the drop empty columns module removes columns that are completely empty,
wherein the cleaned dates module conforms all dates in a designated column to one format;
wherein the multi-punch breakout module takes a designated column name and a delimiting character and breaks the column apart by the character;
wherein the mandatory fields module returns a file with the percentage of missing information and the total of rows missing;
wherein the minimum percent filled module checks if all columns are filled to a defined percentage;
wherein the data type mismatch module writes a file indicating the percent make up of each column in terms of data types;
wherein the unique code count module returns a count of the unique values from a selected column;
wherein the data dictionary module normalizes text fields within the data by replacing all variations and acronyms of a word or set of words with a single word or set of words;
wherein the spell check module identifies and fixes basic typographical errors within text cells;
wherein the clustering module analyzes the dataset via a k-means clustering module and a machine learning model that links similar records in clusters;
wherein the outlier detection module identifies records with values that are outliers for selected columns and returns a printout of the values with occurrences sorted least to greatest, potential outliers at different stages of the module, and a chart of occurrences of those values;
wherein the basic statistics module calculates the 25th percentile, mean, and 75th percentile value, median, mode, and standard deviation of all numeric columns;
wherein the calculation tool module calculates basic operations of columns with multiplication, division, addition, and subtraction;
wherein the time period summary module plots values of a designated column in either a month over month or year over year graph, given that the dataset has dates for every record;
wherein the one to one analysis module checks that values of one selected column corresponds to only one value of another selected column;
wherein the character prefill module prepends a selected character to all values in a column a varied number of times, normalizing the length of the values to a user selected length;
wherein the semantic trend module identifies main points of individual records for discerning general trends of the dataset; and
wherein the best predictors module selects fields that have a higher correlation to a designated field.
15 . The computer program product of claim 11 , wherein the at least one of the one or more data agnostic preprocessing modules configured to utilize Machine Learning includes a clustering module that analyzes the at least one input text data file using a k-means clustering module and a machine learning model that links similar records into clusters, returns at least one image where data has been fitted into the clusters, and assigns the clusters to a closest centroid.