Database management systems for managing data with data confidence
Data records and associated data confidence of data in each data record are both stored in a database system. Each data record includes a data confidence. The data confidence indicates an accuracy or reliability level of the data of corresponding data record. The data records with confidence information are constructed in memory before they are stored into the database system. When the data records are retrieved from the database for analysis, the data confidence is retrieved as well. The analysis of data contained in the data records further considers the data confidence. The analysis result is thus affected by the data confidence.
1. A method for managing data storage and retrieval, the method operated within a database management system and comprising:
obtaining a set of data records that is stored or is to be stored in memory of the database management system, wherein a data record of the set of data records includes a set of data columns;
for the data record:
determining a data confidence value regarding data within one or more columns of the set of data columns, to produce one or more data confidence values, wherein the data confidence value includes one or more of an estimate of an accuracy of the data, an estimate of the accuracy of the data record, and an estimate of a reliability level of the data; and
generating one or more additional columns based on the one or more data confidence values, wherein a first additional column of the one or more additional columns is populated with a first data confidence value of the one or more data confidence values; and
generating a set of confidence data records, wherein a confidence data record of the set of confidence data records to includes the one or more additional columns and the set of data columns of the data record.
2. The method for managing data storage and retrieval of claim 1 further comprising:
storing the confidence data record in a segment of a data storage device of a set of data storage devices, wherein the data storage device includes one or more segments, wherein the one or more segments includes the segment, and wherein a corresponding segment from each of the set of data storage devices form a data segment group.
3. The method for managing data storage and retrieval of claim 1 further comprising:
retrieving the set of confidence data records from the memory, wherein the set of confidence data records includes the set of data records and a corresponding one or more data confidence values for each of the data records of the set of data records; and
analyzing the set of data records to produce a result, wherein the result includes a confidence level based on the corresponding one or more data confidence values.
4. The method for managing data storage and retrieval of claim 1 , wherein the confidence data record further comprises:
the one or more additional columns being subsequent to the set of data columns.