Systems and methods for efficient development of a rule-based system using crowd-sourcing
Described herein are methods, systems, apparatuses and products for efficient development of a rule-based system. An aspect provides a method including accessing data records; converting said data records to an intermediate form; utilizing intermediate forms to compute similarity scores for said data records; and selecting as an example to be provided for rule making at least one record of said data records having a maximum dissimilarity score indicative of dissimilarity to already considered examples.
1. A computer program product for data cleansing, said computer program product comprising:
a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising:
computer readable program code configured to access data records;
computer readable program code configured to convert said data records to an intermediate form;
computer readable program code configured to utilize intermediate forms of said data records to compute similarity scores of individual ones of said data records with respect to one another;
computer readable program code configured to provide, from among said data records, at least one example record for rule making; and
computer readable program code configured to thereafter select from among said data records at least one additional example record for rule making;
the additional example record comprising at least one record presenting at least one similarity score which indicates a least similarity with respect to the at least one example record already provided;
the at least one example record and the at least one additional example record comprising a rule set;
computer readable program code configured to employ a difficulty method to select from among said data records at least one training instance for updating the rule set; and
the selected at least one training instance comprising at least one example record presenting at least one similarity score which indicates a least similarity with respect to at least one example record in the rule set.
2. The computer program product according to claim 1 , wherein the intermediate form is a pattern form of a text record.
3. The computer program product according to claim 1 , wherein to employ a difficulty method comprises computing a difficulty measure based on at least one unhandled token from the selected at least one training instance.
4. The computer program product according to claim 1 , wherein said data records were labeled via a crowd-sourcing mechanism.
5. The computer program product according to claim 1 , wherein the data records comprise at least one of postal records, sales records, and transaction records.
6. A system for data cleansing, said system comprising:
at least one processor; and
a memory device operatively connected to the at least one processor;
wherein, responsive to execution of program instructions accessible to the at least one processor, the at least one processor is configured to:
access data records;
convert said data records to an intermediate form;
utilize intermediate forms of said data records to compute similarity scores of individual ones of said data records with respect to one another;
provide, from among said data records, at least one example record for rule making; and
thereafter select from among said data records at least one additional example record for rule making;
the additional example record comprising at least one record presenting at least one similarity score which indicates a least similarity with respect to the at least one example record already provided;
wherein the at least one example record and the at least one additional example record comprising a rule set and, responsive to execution of the program instructions, the at least one processor is further configured to:
employ a difficulty method to select from among said data records at least one training instance for updating the rule set;
the selected at least one training instance comprising at least one example record presenting at least one similarity score which indicates a least similarity with respect to at least one example record in the rule set.
7. The system according to claim 6 , wherein the intermediate form is a pattern form of a text record.
8. The system according to claim 7 , wherein to employ a difficulty method comprises computing a difficulty measure based on at least one unhandled token from the selected at least one training instance.
9. The system according to claim 6 , wherein said data records were labeled via a crowd-sourcing mechanism.
10. The system according to claim 6 , wherein the data records comprise at least one of postal records, sales records, and transaction records.