Unit prediction for custom feature engineering
A method may include obtaining a dataset that may include at least a first column and a second column, with a first title and a first value and a second title and a second value, respectively. The operations may further include determining a total similarity value between the first and second columns based on a metadata similarity value, a semantic similarity value, and/or a unit of measurement similarity value; adding the first and second columns to a cluster if the total similarity value is less than a threshold value; generating a new column to add to the cluster using a feature engineering function, where the new column may include a new title and a new value, the new value determined using the feature engineering function by acting on the first value and/or the second value. Further, the operations may include adding the new column to the dataset.
1. A method, comprising:
obtaining a dataset having one or more columns, each of the one or more columns includes a title and at least one value;
obtaining a task description corresponding to a machine learning model to be trained using the dataset;
extracting, for each of the one or more columns, the title and a sample value lacking a corresponding unit of measurement from the at least one value;
synthesizing a question to elicit, from a language model, an answer indicating a predicted unit of measurement predicting the corresponding unit of measurement associated with the sample value from each of the one or more columns, the question synthesized based on:
the title for each of the one or more columns;
the sample value for each of the one or more columns; and
the task description corresponding to the machine learning model to be trained using the dataset;
sending the question to the language model to obtain an answer; and
generating from the answer to the question, the predicted unit of measurement for the at least one value in each of the one or more columns;
adding the predicted unit of measurement to the at least one value in each of the one or more columns in the dataset; and
training the machine learning model using the dataset including the added predicted unit of measurement.
2. The method of claim 1 , further comprising, prior to synthesizing the question for each of the one or more columns, expanding the title from each of the one or more columns.
3. The method of claim 1 , wherein a number of sample values are extracted for each of the one or more columns and a number of questions are synthesized, the number of questions synthesized is determined based on a predefined minimum threshold value.
4. The method of claim 1 , further comprising, prior to sending the question to a language model, training the language model by sending at least one question-answer pair to the language model, including:
an example question drafted to elicit an example answer that includes an example unit of measurement; and
the example answer including the example unit of measurement.
5. The method of claim 1 , wherein at least three questions are synthesized; and
wherein the predicted unit of measurement is generated by selecting a majority answer from at least three answers received from the language model.
6. The method of claim 1 , further comprising:
determining a similarity value between the one or more columns in the dataset based on the predicted unit of measurement;
generating a cluster including the one or more columns in the dataset if the similarity value is greater than a threshold value;
generating a new column including a new title and at least one new value to add to the cluster using a feature engineering function;
adding the new column including the new title and the at least one new value to the dataset; and
training a machine learning model using the dataset including the new column.
7. One or more non-transitory computer-readable storage media configured to store instructions that, in response to being executed, cause a system to perform operations, the operations comprising:
obtaining a dataset having one or more columns, each of the one or more columns includes a title and at least one value;
obtaining a task description corresponding to a machine learning model to be trained using the dataset;
extracting, for each of the one or more columns, the title and a sample value lacking a corresponding unit of measurement from the at least one value;
synthesizing a question to elicit, from a language model, an answer indicating a predicted unit of measurement predicting the corresponding unit of measurement associated with the sample value from each of the one or more columns, the question synthesized based on:
the title for each of the one or more columns;
the sample value for each of the one or more columns; and
the task description corresponding to the machine learning model to be trained using the dataset;
sending the question to the language model to obtain an answer; and
generating from the answer to the question, the predicted unit of measurement for the at least one value in each of the one or more columns
adding the predicted unit of measurement to the at least one value in each of the one or more columns in the dataset; and
training a machine learning model using the dataset including the added predicted unit of measurement.
8. The one or more non-transitory computer-readable storage media of claim 7 , further comprising, prior to synthesizing the question for each of the one or more columns, expanding the title from each of the one or more columns.
9. The one or more non-transitory computer-readable storage media of claim 7 , wherein a number of sample values are extracted for each of the one or more columns and a number of questions are synthesized, the number of questions synthesized is determined based on a predefined minimum threshold value.
10. The one or more non-transitory computer-readable storage media of claim 7 , further comprising, prior to sending the question to a language model, training the language model by sending at least one question answer pair to the language model, the question answer pair including an example question drafted to elicit an example answer that includes an example unit of measurement and the example answer including the example unit of measurement.
11. The one or more non-transitory computer-readable storage media of claim 7 , wherein at least three questions are synthesized; and
wherein the predicted unit of measurement is generated by selecting a majority answer from at least three answers received from the language model.
12. The one or more non-transitory computer-readable storage media of claim 7 , further comprising:
determining a similarity value between the one or more columns in the dataset based on the predicted unit of measurement;
generating a cluster including the one or more columns in the dataset if the similarity value is greater than a threshold value;
generating a new column including a new title and at least one new value to add to the cluster using a feature engineering function;
adding the new column including the new title and the at least one new value to the dataset; and
training a machine learning model using the dataset including the new column.
13. A system comprising:
one or more processors; and
one or more non-transitory computer-readable storage media configured to store instructions that, in response to being executed, cause the system to perform operations, the operations comprising:
obtaining a dataset having one or more columns, each of the one or more columns includes a title and at least one value;
obtaining a task description corresponding to a machine learning model to be trained using the dataset;
extracting, for each of the one or more columns, the title and a sample value lacking a corresponding unit of measurement from the at least one value;
synthesizing a question to elicit, from a language model, an answer indicating a predicted unit of measurement predicting the corresponding unit of measurement associated with the sample value from each of the one or more columns, the question synthesized based on:
the title for each of the one or more columns;
the sample value for each of the one or more columns; and
a task description corresponding to a machine learning model to be trained using the dataset;
sending the question to the language model to obtain an answer; and
generating from the answer to the question, the predicted unit of measurement for the at least one value in each of the one or more columns
adding the predicted unit of measurement to the at least one value in each of the one or more columns in the dataset; and
training the machine learning model corresponding to the task description using the dataset including the added predicted unit of measurement.
14. The system of claim 13 , wherein a number of sample values are extracted for each of the one or more columns and a number of questions are synthesized, the number of questions synthesized is determined based on a predefined minimum threshold value.
15. The system of claim 13 , further comprising, prior to sending the question to a language model, training the language model by sending at least one question answer pair to the language model, the question answer pair including an example question drafted to elicit an example answer that includes an example unit of measurement and the example answer including the example unit of measurement.
16. The system of claim 13 , wherein at least three questions are synthesized; and
wherein the predicted unit of measurement is generated by selecting a majority answer from at least three answers received from the language model.
17. The system of claim 13 , further comprising:
determining a similarity value between the one or more columns in the dataset based on the predicted unit of measurement;
generating a cluster including the one or more columns in the dataset if the similarity value is greater than a threshold value;
generating a new column including a new title and at least one new value to add to the cluster using a feature engineering function;
adding the new column including the new title and the at least one new value to the dataset; and
training a machine learning model using the dataset including the new column.