Method for converting nominal to ordinal or continuous variables using time-series distances
A method and system for converting non-ordered categorical data stored within a column in a data set into an ordered or continuous data stored in a new column within the data set. Each distinct categorical value in the nominal data column is represented by a corresponding distinct numerical value in the new column. The new representative numerical values are derived by constructing separate time series for each distinct value in the nominal data column and by calculating the similarities between the shapes of the time series. The proximity of the time series is captured in a numeric distance score. Each distinct distance score corresponds to a distinct value in the nominal data column and is a valid representation of that value in machine learning, deep learning, and statistical analysis.
1. A method for time-series based encoding of categorical data as continuous data by a system comprising a server for data processing and analysis and an apparatus, wherein the apparatus comprises a non-transitory memory for storing program code executable by a processor, an Application Programming Interface (API) for external access to one or more resources and functions and a presentation framework for visualizing data and interacting with the data to generate tasks and actions on the server, the method comprising:
ingesting data for machine modeling;
defining one or more input fields for the machine modeling;
selecting an input field from one or more input fields, containing categorical data for encoding;
generating one or more time series values for one or more attributes in the selected input field containing the categorical data;
generating values of a comparison metric between the one or more time series values;
constructing one or more encoded input fields, containing the values of the comparison metric;
computing the machine modeling, using the one or more encoded input fields instead of the input fields containing the categorical data; and
decoding one or more model results for the one or more encoded input fields,
wherein the method encodes one or more categorical variables into one or more continuous variables based on a time series query,
wherein the time series values are generated using a query language as a result of query execution by the program code stored on the non-transitory memory,
wherein the method further comprises configuration parameters for time series generation query and wherein the configuration parameters for the time series based encoding are applied to one or more groups of the categorical input fields.
2. The method of claim 1 , wherein the categorical input fields include nominal, ordinal, or numeric data. types.
3. The method of claim 1 , wherein one or more categorical fields of the categorical input fields can be selected for time series based encoding.
4. The method of claim 1 , wherein different types of time series based encoding parameters are applied to different categorical input fields.
5. The method of claim 1 , wherein the comparison metric measures the similarity of the shapes of the time series.
6. The method of claim 1 , wherein the comparison metric measures area under the time series.
7. The method of claim 1 , wherein different calculations for the comparison metric can be defined and applied by a user.
8. The method of claim 1 , further comprises visualizing the time series.
9. The method of claim 1 , wherein the machine modeling generates results for the one or more encoded input fields.
10. The method of claim 1 wherein, the metric for the one or more encoded input fields is decoded by matching to the nearest encoded input field.
11. The method of claim 10 wherein, the matching is performed by an algorithm which can be defined by a user.
12. The method of claim 11 wherein, the decoded input fields are mapped to the categorical input fields.
13. A system for time-series based encoding of categorical data as continuous data comprising:
a server for data processing, analysis and storing files at a file storage; and
an apparatus comprising:
a non-transitory memory for storing program code executable by a processor, the program code further comprises of metadata for input definitions. governance of user interactions and query execution;
an Application Programming Interface (API) for external access to one or more resources and functions; and
a presentation framework for visualizing data and interacting with the data to generate tasks and actions on the server;
wherein the server further comprises:
ingesting data for machine modeling;
defining one or more input fields for the machine modeling;
selecting an input field from one or more input fields, containing categorical data for encoding;
generating one or more time series values for one or more attributes in the selected input field containing the categorical data;
generating values of a comparison metric between the one or more time series values;
constructing one or more encoded input fields, containing the values of the comparison metric;
computing the machine modeling, using the one or more encoded input fields instead of the input fields containing the categorical data; and
decoding one or more model results for the one or more encoded input fields,
wherein the server encodes one or more categorical variables into one or more continuous variables based on a time series query,
wherein time series values are generated using a query language as a result of query execution by the program code stored on the non-transitory memory,
wherein the server further comprises configuration parameters for time series generation query and wherein the configuration parameters for the time series based encoding are applied to one or more groups of categorical input fields.
14. The system of claim 13 , wherein the server encoding categorical variables into continuous variables based on the time series query is performed by an algorithm which can be defined by a user.
15. A program product for time-series based encoding of categorical data as continuous data comprising a non-transitory computer readable storage medium storing program code executable by a processor to perform:
ingesting data for machine modeling;
defining one or more input fields for the machine modeling;
selecting an input field from one or more input fields, containing categorical data for encoding;
generating one or more time series values for one or more attributes in the selected input field containing the categorical data;
generating values of a comparison metric between the one or more time series values;
constructing one or more encoded input fields, containing the values of the comparison metric;
computing the machine modeling, using the one or more encoded input fields instead of the input fields containing the categorical data; and
decoding one or more model results for the one or more encoded input fields,
wherein the program product encodes one or more categorical variables into one or more continuous variables based on a time series query,
wherein time series values are generated using a query language as a result of query execution by the program code stored on the non-transitory memory,
wherein the program product comprises configuration parameters for time series generation query and wherein the configuration parameters for the time series based encoding are applied to one or more groups of the categorical input fields.