Method and system for fuzzy matching and alias matching for streaming data sets
A method, system, and computer-usable medium for streaming or processing data streams. Raw text data is cleansed to a standard format. A fuzzy matching algorithm is performed on the text data. For data where domain expertise is required, alias matching is performed. End state categorizing or grouping is provided for the cleansed raw text data.
1. A computer-implementable method for streaming data streams comprising:
cleansing raw text data of the data streams;
applying a fuzzy matching algorithm on input strings of cleansed raw text data to a list of viable end state category or group matches based on common sense level comparisons approximating matching ability and confidence levels of individuals with no domain expertise in a field in which the text data applies, to determine an end state category of the cleansed raw text data;
performing alias matching on the cleansed raw text data that has been applied with the fuzzy matching algorithm, if domain expertise is determined to be used for cleansed raw text data;
separating fuzzy and alias matching into different pipeline stages along expertise boundaries; and
providing the end state category of the cleansed raw text data based on the applying the fuzzy matching algorithm.
2. The method of claim 1 , wherein a configurable software template of a data management platform determines processes that are implemented for the streaming data streams.
3. The method of claim 1 , wherein the raw text data is received from different data sources.
4. The method of claim 1 , wherein the fuzzy matching algorithm is Levenshtein edit distance.
5. The method of claim 1 , wherein the fuzzy matching algorithm implements a confidence level threshold.
6. The method of claim 1 , wherein the alias matching implements a lookup table.
7. The method of claim 1 , wherein a best match is performed for the fuzzy matching and alias matching.
8. A system comprising:
a processor;
a data bus coupled to the processor; and
a non-transitory, computer-readable storage medium embodying computer program code, the non-transitory, computer-readable storage medium being coupled to the data bus, the computer program code interacting with a plurality of computer operations and comprising instructions executable by the processor and configured for:
cleansing raw text data of the data streams;
applying a fuzzy matching algorithm on input strings of cleansed raw text data to a list of viable end state category or group matches based on common sense level comparisons approximating matching ability and confidence levels of individuals with no domain expertise in a field in which the text data applies, to determine an end state category of the cleansed raw text data;
performing alias matching on the cleansed raw text data that has been applied with the fuzzy matching algorithm, if domain expertise is determined to be used for cleansed raw text data;
separating fuzzy and alias matching into different pipeline stages along expertise boundaries; and
providing the end state category of the cleansed raw text data based on the applying the fuzzy matching algorithm.
9. The system of claim 8 , wherein a configurable software template of a data management platform determines processes that are implemented for the streaming data streams.
10. The system of claim 8 , wherein the raw text data is received from different data sources.
11. The system of claim 8 , wherein the fuzzy matching algorithm is Levenshtein edit distance.
12. The system of claim 8 , wherein the fuzzy matching algorithm implements a confidence level threshold.
13. The system of claim 8 , wherein the alias matching implements a lookup table.
14. The system of claim 8 , wherein a best match is performed for the fuzzy matching and alias matching.
15. A non-transitory, computer-readable storage medium embodying computer program code, the computer program code comprising computer executable instructions configured for:
cleansing raw text data of the data streams;
applying a fuzzy matching algorithm on input strings of cleansed raw text data to a list of viable end state category or group matches based on common sense level comparisons approximating matching ability and confidence levels of individuals with no domain expertise in a field in which the text data applies, to determine an end state category of the cleansed raw text data;
performing alias matching on the cleansed raw text data that has been applied with the fuzzy matching algorithm, if domain expertise is determined to be used for cleansed raw text data;
separating fuzzy and alias matching into different pipeline stages along expertise boundaries; and
providing the end state category of the cleansed raw text data based on the applying the fuzzy matching algorithm.
16. The non-transitory, computer-readable storage medium of claim 15 , wherein a configurable software template of a data management platform determines processes that are implemented for the streaming data streams.
17. The non-transitory, computer-readable storage medium of claim 15 , wherein the raw text data is received from different data sources.
18. The non-transitory, computer-readable storage medium of claim 15 , wherein the fuzzy matching algorithm is Levenshtein edit distance.
19. The non-transitory, computer-readable storage medium of claim 15 , wherein the fuzzy matching algorithm implements a confidence level threshold.
20. The non-transitory, computer-readable storage medium of claim 15 wherein the alias matching implements a lookup table.