Systems and/or methods for machine-learning based data correction and completion in sparse datasets

1. A data imputation computer system, comprising:

an interface;

at least one processor and a memory, the memory storing computer code executable by the at least one processor to perform functionality comprising:

receiving over the interface a sparse dataset, the dataset having data records organized in one or more classes, each class having a plurality of properties associated therewith;

dividing the data records in the dataset into a plurality of groups, each group including data records having properties with similar values;

for each group, identifying one or more properties of the records in the respective group that is/are to be ignored based on both record distances with respect to the records in the group, and distances among values for each of the properties of the records in the respective group;

further dividing the data records in the groups into sub-groups without regard to the one or more properties that is/are to be ignored, wherein in at least a first group includes a set of data records that is smaller and more cohesive than the data records of the groups from which the set of data records of the first sub group were generated;

for each sub-group:

identifying values of the data records therein that are empty but are to be filled in;

based on the data records in the respective sub-group, determining predicted values to be applied to the values identified as being empty but needing to be filled in; and

applying the predicted values to be applied to the values identified as being empty but needing to be filled in; and

providing as output the dataset corrected and/or completed using the applied predicted values.

2. The system of claim 1 , wherein the groups are clusters generated via a first clustering algorithm and wherein the sub-groups are sub-clusters generated from the clusters using a second clustering algorithm.

3. The system of claim 2 , wherein the first and second clustering algorithms are different from one another.

4. The system of claim 2 , wherein the first clustering algorithm is a k-means clustering algorithm.

5. The system of claim 2 , wherein the second clustering algorithm is a sequential k-Nearest Neighbor clustering algorithm.

6. The system of claim 2 , wherein the record distances with respect to the records in the group are distances to the respective cluster's centroid.

7. The system of claim 1 , wherein the memory stores further computer code executable by the at least one processor to perform further functionality comprising receiving an identification of one of the one or more classes, wherein the division of the data records in the dataset into the plurality of groups divides the data records based on the identified class(es).

8. The system of claim 1 , wherein the identifying further comprises calculating a property score for each property in each group, the property score for a given property in a given group being the number of non-empty values for the given property divided by the total number of data records in the given group.

9. The system of claim 8 , wherein the identifying further comprises:

weighting the distances among the values for each of the properties of the records in the respective group with the record distances with respect to the records in the group;

calculating final property reduction scores by dividing the weighted distances by the corresponding property scores; and

determining which property(ies) is/are to be ignored based on a comparison between the calculated final property reduction scores and a predetermined threshold.

10. The system of claim 1 , wherein the predicted value for a given property in a given sub-group is the most common non-null value for the given property among the data records in the given sub-group.

11. A data imputation method for use with a computing system including at least one processor and a memory, the method comprising:

using the at least one processor,

receiving over an interface a sparse dataset, the dataset having data records organized in one or more classes, each class having a plurality of properties associated therewith;

dividing the data records in the dataset into a plurality of groups, each group including data records having properties with similar values;

for each group, identifying one or more properties of the records in the respective group that is/are to be ignored based on both record distances with respect to the records in the group, and distances among values for each of the properties of the records in the respective group;

further dividing the data records in the groups into sub-groups without regard to the one or more properties that is/are to be ignored, at least a first subgroup includes a set of data records that is smaller and more cohesive than the data records of the groups from which the set of data records of the first sub group were generated;

for each sub-group:

identifying values of the data records therein that are empty but are to be filled in;

based on the data records in the respective sub-group, determining predicted values to be applied to the values identified as being empty but needing to be filled in; and

applying the predicted values to be applied to the values identified as being empty but needing to be filled in; and

providing as output the dataset corrected and/or completed using the applied predicted values.

12. The method of claim 11 , wherein the group are clusters generated via a first clustering algorithm and wherein the sub-groups are sub-clusters generated from the clusters using a second clustering algorithm.

13. The method of claim 12 , wherein the first clustering algorithm is a k-means clustering algorithm and the second clustering algorithm is a sequential k-Nearest Neighbor clustering algorithm.

14. The method of claim 12 , wherein the record distances with respect to the records in the group are distances to the respective cluster's centroid.

15. The method of claim 11 , wherein the identifying further comprises:

calculating a property score for each property in each group, the property score for a given property in a given group being the number of non-empty values for the given property divided by the total number of data records in the given group;

weighting the distances among the values for each of the properties of the records in the respective group with the record distances with respect to the records in the group;

calculating final property reduction scores by dividing the weighted distances by the corresponding property scores; and

determining which property(ies) is/are to be ignored based on a comparison between the calculated final property reduction scores and a predetermined threshold.

16. The method of claim 11 , wherein the predicted value for a given property in a given sub-group is the most common non-null value for the given property among the data records in the given sub-group.

17. A non-transitory computer readable storage medium tangibly storing instructions that, when executed by at least one processor of a computing system, cause the computing system to perform functionality comprising:

receiving over an interface a sparse dataset, the dataset having data records organized in one or more classes, each class having a plurality of properties associated therewith;

dividing the data records in the dataset into a plurality of groups, each group including data records having properties with similar values;

for each group, identifying one or more properties of the records in the respective group that is/are to be ignored based on both record distances with respect to the records in the group, and distances among values for each of the properties of the records in the respective group;

further dividing the data records in the groups into sub-groups without regard to the one or more properties that is/are to be ignored, wherein at least a first subgroup includes a set of data records that is smaller and more cohesive than the data records of the groups from which from which the set of data records of the first sub group were generated;

for each sub-group:

identifying values of the data records therein that are empty but are to be filled in;

based on the data records in the respective sub-group, determining predicted values to be applied to the values identified as being empty but needing to be filled in; and

applying the predicted values to be applied to the values identified as being empty but needing to be filled in; and

providing as output the dataset corrected and/or completed using the applied predicted values.

18. The non-transitory computer readable storage medium of claim 17 , wherein the group are clusters generated via a k-means clustering algorithm and wherein the sub-groups are sub-clusters generated from the clusters using a sequential k-Nearest Neighbor clustering algorithm.

19. The non-transitory computer readable storage medium of claim 18 , wherein the record distances with respect to the records in the group are distances to the respective cluster's centroid.

20. The non-transitory computer readable storage medium of claim 19 , wherein the identifying further comprises:

calculating a property score for each property in each group, the property score for a given property in a given group being the number of non-empty values for the given property divided by the total number of data records in the given group;

weighting the distances among the values for each of the properties of the records in the respective group with the record distances with respect to the records in the group;

calculating final property reduction scores by dividing the weighted distances by the corresponding property scores; and

determining which property(ies) is/are to be ignored based on a comparison between the calculated final property reduction scores and a predetermined threshold.

21. The non-transitory computer readable storage medium of claim 17 , wherein the predicted value for a given property in a given sub-group is the most common non-null value for the given property among the data records in the given sub-group.