Processing method of data quality dynamic information and data processing device
View Patent ↗A data processing device is provided, which includes a storage device and a data quality assessment measurement calculation module. The storage device is configured to obtain change data of data tables from a data center. The data quality assessment measurement calculation module is configured to calculate an updated data quality assessment measurement value and an updated data quality characteristic data according to the change data and a data quality assessment measurement reference value and perform a notification function according to the updated data quality assessment measurement value and a data quality threshold value. The data quality assessment measurement calculation module is configured to compare the updated data quality assessment measurement value with the data quality threshold value to generate a comparison result. The data quality assessment measurement calculation module is configured to generate and send a notification signal to perform a notification function according to the comparison result.
1 . A data processing device, comprising:
a computer-readable storage medium storing computer-executable instructions; and one or more processors to execute the computer-executable instructions to perform a process comprising:
obtaining and storing, in a storage device, change data received from a data center that includes one or more networked computer devices, wherein the change data comprises new data to be added to a plurality of data tables stored in a database or deleted data to be removed from the plurality of data tables;
in response to reading the change data stored in the storage device:
when the change data is the new data to be added to the data tables, automatically calculating, by a data quality assessment measurement calculation module executed on a computer system, a data quality assessment measurement value and data quality characteristic data of the change data according to the change data and based on a function associated with the data quality assessment measurement value before update, wherein the data quality assessment measurement value comprises a population variance measurement value of the change data and a missing value ratio measurement value before update;
when the change data is the deleted data to be removed from the data tables, automatically calculating, by a data quality assessment measurement calculation module, based on a function associated with the data quality assessment measurement value before update an updated data quality assessment measurement value and updated data quality characteristic data according to the data quality assessment measurement value, the data quality characteristic data of the change data and a data quality assessment measurement reference value, wherein the updated data quality assessment measurement value comprises an updated population variance measurement value and an updated missing value ratio measurement value determined according to the missing value ratio measurement value before update;
recording the updated data quality assessment measurement value and the updated data quality characteristic data into a data quality assessment measurement value table; and
automatically performing a notification function according to the updated data quality assessment measurement value and a data quality threshold value, the performing of the notification function comprising:
comparing the updated data quality assessment measurement value with the data quality threshold value to generate a comparison result;
generating and sending a notification signal to a user device to inform a user of the data center that data quality has reached an expected quality level for performing the notification function when the comparison result indicates that the updated data quality assessment measurement value is greater than or equal to the data quality threshold value.
2 . The data processing device of claim 1 ,
the data quality assessment measurement reference value comprises a data quality assessment measurement value before update and data quality characteristic data before update, and the data quality assessment measurement calculation module is configured to calculate the updated data quality assessment measurement value and the updated data quality characteristic data based on a function associated with the data quality assessment measurement value before update, the data quality characteristic data before update, the data quality assessment measurement value of the change data and the data quality characteristic data of the change data.
3 . The data processing device of claim 2 , wherein the data quality assessment measurement value before update comprises a missing value ratio measurement value before update, the data quality assessment measurement value of the change data comprises a missing value ratio measurement value of the change data, the updated data quality assessment measurement value comprises an updated missing value ratio measurement value, the data quality assessment measurement calculation module is configured to calculate the updated missing value ratio measurement value and the updated data quality characteristic data according to the missing value ratio measurement value before update, the data quality characteristic data before update, the missing value ratio measurement value of the change data and the data quality characteristic data of the change data, wherein when the change data is the new data to be added to the plurality of data tables, the updated missing value ratio measurement value and the updated data quality characteristic data are calculated by the data quality assessment measurement calculation module according to the following equations:
M
R
n
e
w
=
N
N
+
k
M
R
O
r
i
+
k
N
+
k
M
R
c
h
a
n
g
e
C
n
e
w
=
C
O
r
i
+
C
c
h
a
n
g
e
where MR new represents the updated missing value ratio measurement value, MR Ori represents the missing value ratio measurement value before update, MR change represents the missing value ratio measurement value of the change data, C new represents the updated data quality characteristic data, C Ori represents the data quality characteristic data before update, C change represents the data quality characteristic data of the change data, N represents the amount of data used to calculate the data quality assessment measurement value before update, wherein C Ori =N, k represents the amount of data used to calculate the data quality assessment measurement value of the change data, wherein C change =k;
wherein when the change data is the deleted data to be removed from the plurality of data tables, the updated missing value ratio measurement value and the updated data quality characteristic data are calculated by the data quality assessment measurement calculation module according to the following equations:
MR
new
=
N
N
-
k
MR
Ori
-
k
N
-
k
MR
change
C
new
=
C
Ori
-
C
change
.
4 . The data processing device of claim 2 , wherein the change data comprises new data to be added to the plurality of data tables or deleted data to be removed from the plurality of data tables, the data quality assessment measurement value before update comprises a mean measurement value before update and a population variance measurement value before update, the data quality assessment measurement value of the change data comprises a mean measurement value and a population variance measurement value of the change data, the updated data quality assessment measurement value comprises an updated mean measurement value and updated population variance measurement value, the data quality assessment measurement calculation module is configured to calculate the updated mean measurement value, the updated population variance measurement value and the updated data quality characteristic data according to the mean measurement value before update, the population variance measurement value before update, the data quality characteristic data before update, the mean measurement value of the change data, the population variance measurement value of the change data and the data quality characteristic data of the change data, wherein when the change data is the new data to be added to the plurality of data tables, the updated mean measurement value, the updated population variance measurement value and the updated data quality characteristic data are calculated by the data quality assessment measurement calculation module according to the following equations:
μ
new
=
N
N
+
k
μ
Ori
+
k
N
+
k
μ
change
σ
new
2
=
N
N
+
k
σ
Ori
2
+
k
N
+
k
σ
change
2
+
(
N
N
+
k
μ
Ori
2
+
k
N
+
k
μ
change
2
-
μ
new
2
)
C
new
=
C
ori
+
C
change
where μ new represents the updated mean measurement value, μ Ori represents the mean measurement value before update, μ change represents the mean measurement value of the change data, σ 2 new represents, the updated population variance measurement value, σ 2 Ori represents the population variance measurement value before update, σ 2 change represents the population variance measurement value of the change data, C new represents the updated data quality characteristic data, C Ori represents the data quality characteristic data before update, C change represents the data quality characteristic data of the change data, N represents the amount of data used to calculate the mean measurement value before update and the population variance measurement value before update, wherein C Ori =N, k represents the amount of data used to calculate the mean measurement value and the population variance measurement value of the change data, wherein C change =k;
wherein when the change data is the deleted data to be removed from the plurality of data tables, the updated mean measurement value, the updated population variance measurement value and the updated data quality characteristic data are calculated by the data quality assessment measurement calculation module according to the following equations:
μ
n
e
w
=
N
N
-
k
μ
Ori
-
k
N
-
k
μ
c
h
a
n
g
e
σ
n
e
w
2
=
N
N
-
k
σ
2
-
k
N
-
k
σ
2
+
(
N
N
-
k
μ
Ori
2
-
k
N
-
k
μ
c
h
a
n
g
e
2
-
μ
n
e
w
2
)
C
n
e
w
=
C
O
r
i
-
C
c
h
a
n
g
e
.
5 . The data processing device of claim 2 , wherein the change data comprises new data to be added to the plurality of data tables or deleted data to be removed from the plurality of data tables, the data quality assessment measurement value before update comprises a population correlation coefficient measurement value before update, the data quality assessment measurement value of the change data comprises a population correlation coefficient measurement value of the change data, the updated data quality assessment measurement value comprises an updated population correlation coefficient measurement value, the data quality assessment measurement calculation module is configured to calculate the updated population correlation coefficient measurement value and the updated data quality characteristic data according to the population correlation coefficient measurement value before update, the data quality characteristic data before update, the population correlation coefficient measurement value of the change data and the data quality characteristic data of the change data, wherein when the change data is the new data to be added to the plurality of data tables, the updated population correlation coefficient measurement value and the updated data quality characteristic data are calculated by the data quality assessment measurement calculation module according to the following equations:
ρ
new
=
1
σ
x_new
σ
y_new
[
N
N
+
k
(
σ
x_Ori
σ
y_Ori
ρ
Ori
+
(
μ
x_Ori
-
μ
y_new
)
(
μ
y_Ori
-
μ
y_new
)
)
+
k
N
+
k
(
σ
x_change
σ
y_change
ρ
change
+
(
μ
x_change
-
μ
y_new
)
(
μ
y_change
-
μ
y_new
)
)
]
where ρ new represents the updated population correlation coefficient measurement value, ρ Ori represents the population correlation coefficient measurement value before update, ρ change represents the population correlation coefficient measurement value of the change data, C new represents the updated data quality characteristic data, x and y represent variables, σ x_new and σ y_new represent updated standard deviations of the variables, σ x_Ori and σ y_Ori represent standard deviations before update of the variables, σ x_change and σ y_change represent standard deviations of the variables of the change data, My new represents an updated mean measurement value of the variables, μ x_Ori and μ y_Ori represent mean measurement values before update of the variables, μ x_change and μ y_change represent mean measurement values of the variables of the change data, wherein C new =(N+k,μ new ,σ 2 new );
wherein when the change data is the deleted data to be removed from the plurality of data tables, the updated mean measurement value, the updated population correlation coefficient measurement value and the updated data quality characteristic data are calculated by the data quality assessment measurement calculation module according to the following equations:
ρ
n
e
w
=
1
σ
x
_
new
σ
y
_
new
[
N
N
-
k
(
σ
x
_
Ori
σ
y
_
Ori
ρ
Ori
+
(
μ
x
_
Ori
-
μ
y
_
new
)
(
μ
y
_
Ori
-
μ
y
_
new
)
)
-
k
N
-
k
(
σ
x
_
change
σ
y
_
change
ρ
c
h
a
n
g
e
+
(
μ
x
_
change
-
μ
y
_
new
)
(
μ
y
_
change
-
μ
y
_
new
)
)
]
.
6 . The data processing device of claim 2 , wherein the data quality assessment measurement value before update comprises a unique value ratio measurement value before update, the data quality assessment measurement value of the change data comprises a unique value ratio measurement value of the change data, the updated data quality assessment measurement value comprises an updated unique value ratio measurement value, the data quality assessment measurement calculation module is configured to calculate the updated unique value ratio measurement value and the updated data quality characteristic data according to the unique value ratio measurement value before update, the data quality characteristic data before update, the unique value ratio measurement value of the change data and the data quality characteristic data of the change data, wherein when the change data is the new data to be added to the plurality of data tables, the updated unique value ratio measurement value and the updated data quality characteristic data are calculated by the data quality assessment measurement calculation module according to the following equations:
UR
new
=
#
(
key
(
C
ori
⋃
C
c
h
a
n
g
e
)
)
N
+
k
C
new
=
C
o
r
i
⋃
C
change
Where UR new represents the updated unique value ratio measurement value, U represents the union operation, key(·) represents a key value, # represents counting the number of elements, MR Ori represents the unique value ratio measurement value before update, MR change represents the unique value ratio measurement value of the change data, C new represents the updated data quality characteristic data, C Ori represents the data quality characteristic data before update, C change represents the data quality characteristic data of the change data, N represents the amount of data used to calculate the data quality assessment measurement value before update;
wherein when the change data is the deleted data to be removed from the plurality of data tables, the updated unique value ratio measurement value and the updated data quality characteristic data are calculated by the data quality assessment measurement calculation module according to the following equations:
UR
new
=
#
(
key
(
C
ori
-
C
c
h
a
n
g
e
)
)
N
-
k
C
new
=
C
o
r
i
-
C
change
.
7 . A processing method of data quality dynamic information, comprising:
obtaining and storing, in a storage device, change data received from a data center that includes one or more networked computer devices, wherein the change data comprises new data to be added to a plurality of data tables stored in a database or deleted data to be removed from the plurality of data tables;
in response to reading the change data stored in the storage device:
when the change data is the new data to be added to the data tables, automatically calculating, by a data quality assessment measurement calculation module executed on a computer system, a data quality assessment measurement value and data quality characteristic data of the change data according to the change data and based on a function associated with the data quality assessment measurement value before update, wherein the data quality assessment measurement value comprises a population variance measurement value of the change data and a missing value ratio measurement value before update;
when the change data is the deleted data to be removed from the data tables, automatically calculating, by a data quality assessment measurement calculation module, based on a function associated with the data quality assessment measurement value before update an updated data quality assessment measurement value and updated data quality characteristic data according to the data quality assessment measurement value, the data quality characteristic data of the change data and a data quality assessment measurement reference value, wherein the updated data quality assessment measurement value comprises an updated population variance measurement value and an updated missing value ratio measurement value determined according to the missing value ratio measurement value before update;
recording the updated data quality assessment measurement value and the updated data quality characteristic data into a data quality assessment measurement value table; and
automatically performing a notification function according to the updated data quality assessment measurement value and a data quality threshold value, the performing of the notification function comprising:
comparing the updated data quality assessment measurement value with the data quality threshold value to generate a comparison result;
generating and sending a notification signal to a user device to inform a user of the data center that data quality has reached an expected quality level for performing the notification function when the comparison result indicates that the updated data quality assessment measurement value is greater than or equal to the data quality threshold value.
8 . The processing method of data quality dynamic information of claim 7 , wherein the data quality assessment measurement reference value comprises a data quality assessment measurement value before update and data quality characteristic data before update, and the processing method further comprising:
calculating the updated data quality assessment measurement value and the updated data quality characteristic data based on a function associated with the data quality assessment measurement value before update, the data quality characteristic data before update, the data quality assessment measurement value of the change data and the data quality characteristic data of the change data.
9 . The processing method of data quality dynamic information of claim 8 , wherein the data quality assessment measurement value before update comprises a missing value ratio measurement value before update, the data quality assessment measurement value of the change data comprises a missing value ratio measurement value of the change data, the updated data quality assessment measurement value comprises an updated missing value ratio measurement value, and the processing method further comprising:
calculating the updated missing value ratio measurement value and the updated data quality characteristic data according to the missing value ratio measurement value before update, the data quality characteristic data before update, the missing value ratio measurement value of the change data and the data quality characteristic data of the change data, wherein when the change data is the new data to be added to the plurality of data tables, the updated missing value ratio measurement value and the updated data quality characteristic data are determined according to the following equations:
MR
new
=
N
N
+
k
MR
Ori
+
k
N
+
k
MR
change
C
new
=
C
o
r
i
+
C
change
where MR new represents the updated missing value ratio measurement value, MR Ori represents the missing value ratio measurement value before update, MR change represents the missing value ratio measurement value of the change data, C new represents the updated data quality characteristic data, C Ori represents the data quality characteristic data before update, C change represents the data quality characteristic data of the change data, N represents the amount of data used to calculate the data quality assessment measurement value before update, wherein C Ori =N, k represents the amount of data used to calculate the data quality assessment measurement value of the change data, wherein C change =k;
wherein when the change data is the deleted data to be removed from the plurality of data tables, the updated missing value ratio measurement value and the updated data quality characteristic data are calculated by the data quality assessment measurement calculation module according to the following equations:
MR
new
=
N
N
-
k
MR
Ori
-
k
N
-
k
MR
change
C
new
=
C
o
r
i
-
C
change
.
10 . The processing method of data quality dynamic information of claim 8 , wherein the change data comprises new data to be added to the plurality of data tables or deleted data to be removed from the plurality of data tables, the data quality assessment measurement value before update comprises a mean measurement value before update and a population variance measurement value before update, the data quality assessment measurement value of the change data comprises a mean measurement value and a population variance measurement value of the change data, the updated data quality assessment measurement value comprises an updated mean measurement value and updated population variance measurement value, and the processing method further comprising:
calculating the updated mean measurement value, the updated population variance measurement value and the updated data quality characteristic data according to the mean measurement value before update, the population variance measurement value before update, the data quality characteristic data before update, the mean measurement value of the change data, the population variance measurement value of the change data and the data quality characteristic data of the change data, wherein when the change data is the new data to be added to the plurality of data tables, the updated mean measurement value, the updated population variance measurement value and the updated data quality characteristic data are determined according to the following equations:
μ
new
=
N
N
+
k
μ
Ori
+
k
N
+
k
μ
change
σ
new
2
=
N
N
+
k
σ
Ori
2
+
k
N
+
k
σ
change
2
+
(
N
N
+
k
μ
Ori
2
+
k
N
+
k
μ
change
2
-
μ
new
2
)
C
new
=
C
ori
+
C
change
where μ new represents the updated mean measurement value, μ Ori represents the mean measurement value before update, μ change represents the mean measurement value of the change data, σ 2 new represents, the updated population variance measurement value, σ 2 Ori represents the population variance measurement value before update, σ 2 change represents the population variance measurement value of the change data, C new represents the updated data quality characteristic data, C Ori represents the data quality characteristic data before update, C change represents the data quality characteristic data of the change data, N represents the amount of data used to calculate the mean measurement value before update and the population variance measurement value before update, wherein C Ori =N, k represents the amount of data used to calculate the mean measurement value and the population variance measurement value of the change data, wherein C change =k;
wherein when the change data is the deleted data to be removed from the plurality of data tables, the updated mean measurement value, the updated population variance measurement value and the updated data quality characteristic data are calculated by the data quality assessment measurement calculation module according to the following equations:
μ
new
=
N
N
+
k
μ
Ori
+
k
N
+
k
μ
change
σ
new
2
=
N
N
+
k
σ
Ori
2
+
k
N
+
k
σ
change
2
+
(
N
N
+
k
μ
Ori
2
+
k
N
+
k
μ
change
2
-
μ
new
2
)
C
new
=
C
ori
+
C
change
.
11 . The processing method of data quality dynamic information of claim 8 , wherein the change data comprises new data to be added to the plurality of data tables or deleted data to be removed from the plurality of data tables, the data quality assessment measurement value before update comprises a population correlation coefficient measurement value before update, the data quality assessment measurement value of the change data comprises a population correlation coefficient measurement value of the change data, the updated data quality assessment measurement value comprises an updated population correlation coefficient measurement value, and the processing method further comprising:
calculating the updated population correlation coefficient measurement value and the updated data quality characteristic data according to the population correlation coefficient measurement value before update, the data quality characteristic data before update, the population correlation coefficient measurement value of the change data and the data quality characteristic data of the change data, wherein when the change data is the new data to be added to the plurality of data tables, the updated population correlation coefficient measurement value and the updated data quality characteristic data are determined according to the following equations:
ρ
new
=
1
σ
x_new
σ
y_new
[
N
N
+
k
(
σ
x_Ori
σ
y_Ori
ρ
Ori
+
(
μ
x_Ori
-
μ
y_new
)
(
μ
y_Ori
-
μ
y_new
)
)
+
k
N
+
k
(
σ
x_change
σ
y_change
ρ
change
+
(
μ
x_change
-
μ
y_new
)
(
μ
y_change
-
μ
y_new
)
)
]
where ρ new represents the updated population correlation coefficient measurement value, ρ Ori represents the population correlation coefficient measurement value before update, ρ change represents the population correlation coefficient measurement value of the change data, C new represents the updated data quality characteristic data, x and y represent variables, σ x_new and σ y_new represent updated standard deviations of the variables, σ x_Ori and σ y_Ori represent standard deviations before update of the variables, σ x_change and σ y_change represent standard deviations of the variables of the change data, μ y_new represents an updated mean measurement value of the variables, μ x_Ori and μ y_Ori represent mean measurement values before update of the variables, μ x_change and μ y_change represent mean measurement values of the variables of the change data, wherein C new =(N+k,μ new ,σ 2 new );
wherein when the change data is the deleted data to be removed from the plurality of data tables, the updated mean measurement value, the updated population correlation coefficient measurement value and the updated data quality characteristic data are calculated by the data quality assessment measurement calculation module according to the following equations:
ρ
n
e
w
=
1
σ
x
_
new
σ
y
_
new
[
N
N
-
k
(
σ
x
_
Ori
σ
y
_
Ori
ρ
Ori
+
(
μ
x
_
Ori
-
μ
y
_
new
)
(
μ
y
_
Ori
-
μ
y
_
new
)
)
-
k
N
-
k
(
σ
x
_
change
σ
y
_
change
ρ
c
h
a
n
g
e
+
(
μ
x
_
change
-
μ
y
_
new
)
(
μ
y
_
change
-
μ
y
_
new
)
)
]
.
12 . The processing method of data quality dynamic information of claim 8 , wherein the change data comprises new data to be added to the plurality of data tables or deleted data to be removed from the plurality of data tables, the data quality assessment measurement value before update comprises a unique value ratio measurement value before update, the data quality assessment measurement value of the change data comprises a unique value ratio measurement value of the change data, the updated data quality assessment measurement value comprises an updated unique value ratio measurement value, and the processing method further comprising:
calculating the updated unique value ratio measurement value and the updated data quality characteristic data according to the unique value ratio measurement value before update, the data quality characteristic data before update, the unique value ratio measurement value of the change data and the data quality characteristic data of the change data, wherein when the change data is the new data to be added to the plurality of data tables, the updated unique value ratio measurement value and the updated data quality characteristic data are determined according to the following equations:
UR
new
=
#
(
key
(
C
ori
⋃
C
c
h
a
n
g
e
)
)
N
+
k
C
new
=
C
o
r
i
⋃
C
change
Where UR new represents the updated unique value ratio measurement value, U represents the union operation, key(·) represents a key value, # represents counting the number of elements, MR Ori represents the unique value ratio measurement value before update, MR change represents the unique value ratio measurement value of the change data, C new represents the updated data quality characteristic data, C Ori represents the data quality characteristic data before update, C change represents the data quality characteristic data of the change data, N represents the amount of data used to calculate the data quality assessment measurement value before update;
wherein when the change data is the deleted data to be removed from the plurality of data tables, the updated unique value ratio measurement value and the updated data quality characteristic data are calculated by the data quality assessment measurement calculation module according to the following equations:
UR
new
=
#
(
key
(
C
ori
-
C
c
h
a
n
g
e
)
)
N
-
k
C
new
=
C
o
r
i
-
C
change
.