System and method for reconciliation of data in multiple systems using permutation matching

1. A method comprising:

obtaining, by at least one processing device, a first data set generated by a first computing system and a second data set generated by a second computing system, the first and second data sets configured to be reconciled;

enriching, by the at least one processing device, at least one of the first and second data sets by adding feature values to the at least one data set, the feature values selected to account for a difference in format between the first and second data sets;

generating, by the at least one processing device, matching rules based on the first and second data sets;

using the matching rules, identifying, by the at least one processing device, discrepancies between the first and second data sets; and

outputting, by the at least one processing device, the discrepancies between the first and second data sets to at least one of the first computing system, the second computing system, or a third computing system,

wherein the matching rules include at least one permutation key, each permutation key identifying a subset of data to be grouped together in one of the first and second data sets;

wherein identifying the discrepancies comprises attempting to match one or more first characteristics associated with the grouped subset of data in one of the first and second data sets to one or more second characteristics associated with another of the first and second data sets;

wherein generating the matching rules comprises (i) randomly adding or removing possible permutation keys in multiple simulation iterations, (ii) determining a metric for each simulation iteration, and (iii) selecting the at least one permutation key from the possible permutation keys based on the metrics of the simulation iterations; and

wherein randomly adding or removing the possible permutation keys is constrained to permutation keys with dissimilar frequency distributions in the first and second data sets.

2. The method of claim 1 , wherein the first and second characteristics comprise a key that needs to match exactly between data items in the first and second data sets in order to match the data items in the first and second data sets.

3. The method of claim 1 , wherein the first and second characteristics comprise a value that needs to match in aggregate between groups of data items in the first and second data sets in order to match the groups of data items in the first and second data sets.

4. The method of claim 1 , wherein the first and second characteristics comprise a value that needs to match in aggregate within a tolerance between groups of data items in the first and second data sets in order to match the groups of data items in the first and second data sets.

5. The method of claim 1 , wherein the metric for each simulation iteration is based on a combination of a number of matched data items and a number of matched groups of data items.

6. The method of claim 1 , wherein the matching rules include multiple permutation keys, different permutation keys associated with different subsets of data in different ones of the first and second data sets.

7. An apparatus comprising:

at least one memory configured to store a first data set generated by a first computing system and a second data set generated by a second computing system, the first and second data sets configured to be reconciled; and

at least one processing device configured to:

enrich at least one of the first and second data sets by adding feature values to the at least one data set, the feature values selected to account for a difference in format between the first and second data sets;

generate matching rules based on the first and second data sets;

identify discrepancies between the first and second data sets using the matching rules; and

control the apparatus to output the discrepancies between the first and second data sets to at least one of the first computing system, the second computing system, or a third computing system,

wherein the matching rules include at least one permutation key, each permutation key identifying a subset of data to be grouped together in one of the first and second data sets;

wherein, to identify the discrepancies, the at least one processing device is configured to attempt to match one or more first characteristics associated with the grouped subset of data in one of the first and second data sets to one or more second characteristics associated with another of the first and second data sets;

wherein, to generate the matching rules, the at least one processing device is configured to (i) randomly add or remove possible permutation keys in multiple simulation iterations, (ii) determine a metric for each simulation iteration, and (iii) select the at least one permutation key from the possible permutation keys based on the metrics of the simulation iterations; and

wherein, to randomly add or remove the possible permutation keys, the at least one processing device is constrained to permutation keys with dissimilar frequency distributions in the first and second data sets.

8. The apparatus of claim 7 , wherein the first and second characteristics comprise a key that needs to match exactly between data items in the first and second data sets in order to match the data items in the first and second data sets.

9. The apparatus of claim 7 , wherein the first and second characteristics comprise a value that needs to match in aggregate between groups of data items in the first and second data sets in order to match the groups of data items in the first and second data sets.

10. The apparatus of claim 7 , wherein the first and second characteristics comprise a value that needs to match in aggregate within a tolerance between groups of data items in the first and second data sets in order to match the groups of data items in the first and second data sets.

11. The apparatus of claim 7 , wherein the metric for each simulation iteration is based on a combination of a number of matched data items and a number of matched groups of data items.

12. The apparatus of claim 7 , wherein the matching rules include multiple permutation keys, different permutation keys associated with different subsets of data in different ones of the first and second data sets.

13. A non-transitory computer readable medium containing instructions that when executed cause at least one processor to:

obtain a first data set generated by a first computing system and a second data set generated by a second computing system, the first and second data sets configured to be reconciled;

enrich at least one of the first and second data sets by adding feature values to the at least one data set, the feature values selected to account for a difference in format between the first and second data sets;

generate matching rules based on the first and second data sets;

using the matching rules, identify discrepancies between the first and second data sets; and

output the discrepancies between the first and second data sets to at least one of the first computing system, the second computing system, or a third computing system,

wherein the matching rules include at least one permutation key, each permutation key identifying a subset of data to be grouped together in one of the first and second data sets;

wherein the instructions that when executed cause the at least one processor to identify the discrepancies comprise:

instructions that when executed cause the at least one processor to attempt to match one or more first characteristics associated with the grouped subset of data in one of the first and second data sets to one or more second characteristics associated with another of the first and second data sets;

wherein the instructions that when executed cause the at least one processor to generate the matching rules comprise:

instructions that when executed cause the at least one processor to (i) randomly add or remove possible permutation keys in multiple simulation iterations, (ii) determine a metric for each simulation iteration, and (iii) select the at least one permutation key from the possible permutation keys based on the metrics of the simulation iterations; and

wherein the randomly added or removed possible permutation keys are constrained to permutation keys with dissimilar frequency distributions in the first and second data sets.

14. The non-transitory computer readable medium of claim 13 , wherein the first and second characteristics comprise a key that needs to match exactly between data items in the first and second data sets in order to match the data items in the first and second data sets.

15. The non-transitory computer readable medium of claim 13 , wherein the first and second characteristics comprise a value that needs to match in aggregate between groups of data items in the first and second data sets in order to match the groups of data items in the first and second data sets.

16. The non-transitory computer readable medium of claim 13 , wherein the first and second characteristics comprise a value that needs to match in aggregate within a tolerance between groups of data items in the first and second data sets in order to match the groups of data items in the first and second data sets.

17. The non-transitory computer readable medium of claim 13 , wherein the metric for each simulation iteration is based on a combination of a number of matched data items and a number of matched groups of data items.

18. The non-transitory computer readable medium of claim 13 , wherein the matching rules include multiple permutation keys, different permutation keys associated with different subsets of data in different ones of the first and second data sets.