Efficient database management system and method for prioritizing analytical calculations on datasets
Analyzing large datasets requires prioritization of analytical calculations to reduce analysis time and resource requirements. The prioritization process includes determining characteristics of a dataset, determining a list of analytical calculations, prioritizing the list of analytical calculations based on the dataset characteristics and characteristics of the analytical calculations, selecting the highest ranked analytical calculation, and applying the selected calculation on the dataset. The prioritization process of a new calculation based on a known set of calculations includes ranking the set of calculations based on their result scores from execution on a given dataset, comparing the new calculation to the list of known set of calculation, determining a similar calculation, and assigning the rank of the similar calculation to the new calculation.
1. A method for execution by a computer of a database management system for prioritizing analytical calculations on a dataset within the database management system, the method comprising:
obtaining a dataset from memory of the database management system, wherein the dataset includes a set of data records, wherein the dataset has a data confidence value, and wherein the data confidence value includes one or more of an estimate of an accuracy of the dataset, an estimate of the accuracy of a data record of the set of data records, and an estimate of a reliability level of the dataset;
determining a set of data characteristics of the dataset;
determining a list of analytical calculations that are able to be executed on the dataset;
determining an analytical calculation characteristic for each analytical calculation of the list of analytical calculations to produce a set of analytical calculation characteristics, wherein the analytical calculation characteristic includes an estimated execution time to perform the analytical calculation;
ranking each analytical calculation of the list of analytical calculations based on the set of data characteristics and the set of analytical calculation characteristics to produce a ranked list of analytical calculations;
selecting an analytical calculation from the ranked list of analytical calculations based on the ranking; and
executing the selected analytical calculation on the dataset to produce an analytical calculation result.
2. The method for prioritizing analytical calculations of claim 1 , wherein a data characteristic of the set of data characteristics includes a number of data records within the dataset.
3. The method for prioritizing analytical calculations of claim 1 , wherein a data characteristic of the set of data characteristics includes a data type of a column of each data record within the dataset.
4. The method for prioritizing analytical calculations of claim 1 , wherein the analytical calculation determines an abnormality of data records within the dataset.
5. The method for prioritizing analytical calculations of claim 4 , wherein the abnormality is one of:
an error;
data distortion; and
data noise.
6. The method for prioritizing analytical calculations of claim 1 further comprises:
determining to perform another analytical calculation on the dataset, wherein the other analytical calculation is not included in the list of analytical calculations;
determining another analytical calculation characteristic for the other analytical calculation;
comparing the other analytical calculation characteristic to the set of analytical calculation characteristics;
when the other analytical calculation characteristic matches a first analytical calculation characteristic of the set of analytical calculation characteristics:
determining the ranking of the analytical calculation associated with the first analytical calculation characteristic;
assigning the ranking of the analytical calculation to the other analytical calculation; and
executing the other analytical calculation on the dataset in accordance with the ranked list of analytical calculations.
7. The method for prioritizing analytical calculations of claim 1 further comprises:
executing each analytical calculation within the list of analytical calculations on the dataset to produce a list of result scores, wherein the list of result scores includes a result score for each analytical calculation; and
updating the ranking of the list of analytical calculations based on the list of result scores.
8. The method for prioritizing analytical calculations of claim 1 , wherein the analytical calculation characteristic further includes:
a type of calculation to be performed by the analytical calculation, wherein the type includes one or more of a linear operation and a square root operation.