Data quality management using business process modeling
A business process modeling framework is used for data quality analysis. The modeling framework represents the sources of transactions entering the information processing system, the various tasks within the process that manipulate or transform these transactions, and the data repositories in which the transactions are stored or aggregated. A subset of these tasks is associated as the potential error introduction sources, and the rate and magnitude of various error classes at each such task are probabilistically modeled. This model can be used to predict how changes in transactions volumes and business processes impact data quality at the aggregate level in the data repositories. The model can also account for the presence of error correcting controls and assess how the placement and effectiveness of these controls alter the propagation and aggregation of errors. Optimization techniques are used for the placement of error correcting controls that meet target quality requirements while minimizing the cost of operating these controls. This analysis also contributes to the development of business “dashboards” that allow decision-makers to monitor and react to key performance indicators (KPIs) based on aggregation of the transactions being processed. Data quality estimation in real time provides the accuracy of these KPIs (in terms of the probability that a KPI is above or below a given value), which may condition the action undertaken by the decision-maker.
1. A method of managing data quality with an information processing system, comprising
creating a model of a new or existing business process on a computer, the model representing data whose quality is to be managed using the information processing system;
extending the model to assign tasks within the business process to attributes that model data quality for the data whose quality is to be managed, comprising
assigning tasks characterized by a volume of transactions over a predefined time period and a random variable for each transaction signifying a quantitative value of the transaction a data quality attribute of transaction source,
assigning tasks that operate on incoming transactions and are able to produce errors in them a data quality attribute of error source, any particular transaction error being one of the group comprising valuation error, existence error, and completeness error, and
assigning repositories in the business process where transactions can be stored and retrieved a data quality attribute of audit target;
identifying an anticipated change in a business environment served by the model;
running by the computer error propagation analysis of said anticipated change on said error source tasks operating on said transaction source tasks to estimate error rates and cost of error at the audit targets for the data whose quality is to be managed;
utilizing a control systems model on the computer to associate said error sources with a set of controls;
analyzing by the computer an impact of selected controls of said set of controls responsive to said anticipated change, wherein said selected controls are applied to reduce error rates and cost of error at the audit targets for the data whose quality is to be managed;
optimizing by the computer a selection of controls for the data whose quality is to be managed, using an assessment technique that compares a cost of applying said selected controls with said cost of error at the audit targets,
wherein said optimizing includes programming the computer to use the algorithm
p
(
ɛ
K
i
)
=
p
(
ɛ
)
·
∑
j
=
1
K
i
(
1
-
y
j
p
(
c
s
,
j
❘
ɛ
)
.
ln
p
(
ɛ
K
i
)
=
ln
p
(
ɛ
)
+
∑
j
=
1
K
i
ln
(
1
-
y
j
p
(
c
s
,
j
❘
ɛ
)
)
where
p(ε k i ) is an optimal control strategy k i , for a given target error level,
p(ε) is an error incidence probability of error ε,
K i , is the sequence of controls available for the error source t i , where a control strategy is defined by a fraction of transactions y j ,jε{1,2,. . . , |K i |}, reaching each control c j εK i ; and
applying said selection of controls optimized in said optimizing step to at least some of said tasks within the business process in said information processing system that operate on incoming transactions and are able to produce errors in them.
2. The method of claim 1 , further comprising identifying transaction sources of obtaining or estimating a volume of transactions over a given time period and estimating transaction book values, wherein said anticipated change is a change in said volume of transactions for an identified transaction source.
3. The method of claim 2 , wherein said estimating transaction book values is configured to estimate based on a simple average book value or a probability distribution based on historical transaction data.
4. The method of claim 1 , further comprising identifying error sources of obtaining a probability of errors prior to application of any controls and a taint of the error sources.
5. The method of claim 1 further comprising the step of providing a dashboard of key performance indicators.