Automated detection of null field values and effectively null field values

1. A computer implemented method of detecting and treating one or more field values as null field values in records of an electronic database, the computer implemented method comprising:

selecting a field of the records;

determining, using a programmed computer, a frequency for each field value of the field, the frequency comprising an amount of the records in which the field value of the field appears;

calculating, using a programmed computer, a critical frequency for the field based on the frequency of each field value of the field by calculating f(x) =g(x)−g(x+1), wherein f is a function of the frequency differences based on a function g that is defined by the frequency of each field value of the field wherein the calculating comprises ordering the frequencies of the field values of the field according to frequency along a horizontal axis of a histogram of the one or more field values of the field; and

treating, using a programmed computer, each field value of the field with a frequency that is greater than the critical frequency of the field as a null field value and each field value of the field with a frequency that is less than the critical frequency of the field as a non-null field value.

2. The computer implemented method of claim 1 , wherein the calculating further comprises calculating a difference between adjacent frequencies in the histogram.

3. The computer implemented method of claim 1 , wherein the critical frequency of the field is determined by determining the first point at which a derivative of the f function changes from a first sign to a second sign.

4. The computer implemented method of claim 1 , wherein the critical frequency of the field is determined by determining the first point at which a derivative of the f function changes from a positive sign to a negative sign.

5. The computer implemented method of claim 1 , wherein the critical frequency of the field is determined by determining the first point at which a derivative of the f function changes from a negative sign to a positive sign.

6. The computer implemented method of claim 1 , wherein the critical frequency of the field is determined by determining the first point at which a derivative of the f function equals zero.

7. The computer implemented method of claim 1 , wherein the critical frequency of the field is determined by determining the first point at which a derivative of the f function decreases below a threshold.

8. The computer implemented method of claim 1 , wherein the treating further comprises treating each field value of the field with a frequency that is equal to the critical frequency of the field as a null field value.

9. The computer implemented method of claim 1 , further comprising replacing one or more field values treated as null field values with a canonical null field value.

10. A computer implemented system of detecting and treating one or more field values as null field values in records of an electronic database, the computer implemented system comprising:

one or more computing apparatuses collectively configured to:

select a field of the records;

determine a frequency for each field value of the field, the frequency comprising an amount of the records in which the field value of the field appears;

calculate a critical frequency for the field based on the frequency of each field value of the field by calculating f(x)=g(x)−g(x+1), wherein f is a function of the frequency differences based on a function g that is defined by the frequency of each field value of the field wherein the computing apparatus is configured to calculate by ordering the frequencies of the field values of the field according to frequency along a horizontal axis of a histogram of the one or more field values of the field; and

treat each field value of the field with a frequency that is greater than the critical frequency of the field as a null field value and each field value of the field with a frequency that is less than the critical frequency of the field as a non-null field value.

11. The computer implemented system of claim 10 , wherein the computing apparatus is further configured to calculate by calculating a difference between adjacent frequencies in the histogram.

12. The computer implemented system of claim 10 , wherein the critical frequency of the field is determined by determining the first point at which a derivative of the f function changes from a first sign to a second sign.

13. The computer implemented system of claim 10 , wherein the critical frequency of the field is determined by determining the first point at which a derivative of the f function changes from a positive sign to a negative sign.

14. The computer implemented system of claim 10 , wherein the critical frequency of the field is determined by determining the first point at which a derivative of the f function changes from a negative sign to a positive sign.

15. The computer implemented system of claim 10 , wherein the critical frequency of the field is determined by determining the first point at which a derivative of the f function equals zero.

16. The computer implemented system of claim 10 , wherein the critical frequency of the field is determined by determining the first point at which a derivative of the f function decreases below a threshold.

17. The computer implemented system of claim 10 , wherein the fourth computing apparatus is further configured to treat each field value of the field with a frequency that is equal to the critical frequency of the field as a null field value.

18. The computer implemented system of claim 10 , further comprising a fifth computing apparatus configured to replace one or more field values treated as null field values with a canonical null field value.