System and method for displaying a sample of uniform and outlier rows from a file
A system and method identifies rows in a file as uniform rows or outlier rows based on statistics from the file, and displays a sampling of uniform and outlier rows.
1. A method of displaying data of a file via a user interface, comprising:
receiving a delimiter that delimits at least some of a plurality of rows of the file, each row of the file associated with values corresponding to a plurality of characteristics;
for each of the plurality of characteristics of rows of the file, determining an aggregate value of the characteristic by aggregating characteristic values corresponding to each row of a set of rows corresponding to at least a portion of the plurality of rows of the file;
identifying a criteria for classifying each of the plurality of rows of the file as one of a uniform row or an outlier row, the criteria comprising a comparison of each of one or more characteristic values of the row of the file with the corresponding aggregate value of the characteristic;
classifying at least some of the plurality of rows of the file based on the identified criteria to determine a plurality of uniform rows of the file and a plurality of outlier rows of the file, wherein an outlier row of the file represents a row of the file having one or more characteristic values with more than a threshold difference compared to the corresponding aggregate value of the characteristic;
sampling, using a first sampling technique, to determine a set of uniform rows of the file from the plurality of uniform rows of the file;
sampling, using a second sampling technique, to determine a set of outlier rows of the file from the plurality of outlier rows of the file;
determining a number of rows of the file able to be displayed by the browser window on the user interface;
determining a number of uniform rows of the set of uniform rows to be displayed, based upon (a) a difference between the number of rows able to be displayed by the browser window and a size of the set of outlier rows, and (b) a minimum number of uniform rows;
determining a number of outlier rows of the set of outlier rows to be displayed, based upon a difference between the number of rows able to be displayed by the browser window and the determined number of uniform rows; and
displaying, via the browser window on user interface:
at least some of the set of uniform rows of the file sampled corresponding to the determined number of uniform rows; and
at least some of the set of outlier rows of the file sampled corresponding to the determined number of outlier rows, wherein the outlier rows are selected in order to stratify the outlier portions across a plurality of characteristics associated with the outlier rows.
2. The method of claim 1 , wherein the second sampling technique is congressional sampling, without replacement.
3. The method of claim 1 , wherein at least one of the displaying steps comprises determining whether one of the sampled outlier or uniform rows of the file should be displayed before other of the plurality of the rows of the file have been classified, and displaying said one of the plurality of the rows of the file responsive to said determining.
4. The method of claim 3 , wherein:
responsive to the one of the sampled outlier or uniform rows of the file being classified as a uniform row of the file, the determining step comprises identifying whether a threshold measure of the at least some of the uniform rows of the file have been displayed; and
wherein the displaying the one of the plurality of the rows of the file is responsive to said identifying.
5. The method of claim 3 wherein:
responsive to the one of the sampled outlier or uniform rows of the file being classified as an outlier row of the file, the determining step comprises identifying whether a deviation of a numeric-based characteristic of the one of the sampled outlier rows of the file relative to a plurality of other of the rows of the file exceeds a threshold measure; and
wherein displaying the one of the plurality of the rows of the file responsive to said identifying.
6. The method of claim 3 wherein:
responsive to the one of the sampled outlier or uniform rows of the file being classified as an outlier row of the file, the determining step comprises identifying whether a deviation of a Boolean-based characteristic of the one of the sampled outlier rows of the file is exhibited by less than a threshold number of other of the plurality of rows of the file being displayed; and
wherein displaying the one of the plurality of the rows of the file is responsive to said identifying.
7. The method of claim 1 , further comprising:
determining a size of a window of a user interface displaying data of the file; and
determining a number of uniform rows of the file and a number of outlier rows of the file to display on the user interface based on the size of the window.
8. A computer system for displaying data of a file via a user interface, comprising a processor and a non-transitory computer useable medium storing instructions for:
receiving a delimiter that delimits at least some of a plurality of rows of the file, each row of the file associated with values corresponding to a plurality of characteristics;
for each of the plurality of characteristics of rows of the file, determining an aggregate value of the characteristic by aggregating characteristic values corresponding to each row of a set of rows corresponding to at least a portion of the plurality of rows of the file;
identifying a criteria for classifying each of the plurality of rows of the file as one of a uniform row or an outlier row, the criteria comprising a comparison of each of one or more characteristic values of the row of the file with the corresponding aggregate value of the characteristic;
classifying at least some of the plurality of rows of the file based on the identified criteria to determine a plurality of uniform rows of the file and a plurality of outlier rows of the file, wherein an outlier row of the file represents a row of the file having one or more characteristic values with more than a threshold difference compared to the corresponding aggregate value of the characteristic;
sampling, using a first sampling technique, to determine a set of uniform rows of the file from the plurality of uniform rows of the file;
sampling, using a second sampling technique, to determine a set of outlier rows of the file from the plurality of outlier rows of the file;
determining a number of rows of the file able to be displayed by the browser window on the user interface;
determining a number of uniform rows of the set of uniform rows to be displayed, based upon (a) a difference between the number of rows able to be displayed by the browser window and a size of the set of outlier rows, and (b) a minimum number of uniform rows;
determining a number of outlier rows of the set of outlier rows to be displayed, based upon a difference between the number of rows able to be displayed by the browser window and the determined number of uniform rows; and
displaying, via the browser window on user interface:
at least some of the set of uniform rows of the file sampled corresponding to the determined number of uniform rows; and
at least some of the set of outlier rows of the file sampled corresponding to the determined number of outlier rows, wherein the outlier rows are selected in order to stratify the outlier portions across a plurality of characteristics associated with the outlier rows.
9. The computer system of claim 8 , wherein the second sampling technique comprises congressional sampling, without replacement.
10. The computer system of claim 8 , wherein the instructions are further for:
making a determination whether one of the sampled outlier or uniform rows of the file should be displayed before other of the plurality of the rows of the file have been classified, and for providing at an output an indication of such determination; and
wherein displaying one of the sampled outlier or uniform rows of the file is responsive to the indication of the determination.
11. The computer system of claim 10 , wherein, the determining is in response to the one of the sampled outlier or uniform rows of the file being classified as a uniform row of the file, responsive to whether a threshold measure of the at least some of the uniform rows of the file have been displayed.
12. The computer system of claim 10 wherein, responsive to the one of the sampled outlier or uniform rows of the file being classified as an outlier row of the file, responsive to whether a deviation of a numeric-based characteristic of the one of the sampled outlier rows of the file relative to a plurality of other of the rows of the file exceeds a threshold measure.
13. The computer system of claim 10 wherein responsive to the one of the sampled outlier or uniform rows of the file being classified as an outlier row of the file, responsive to whether a deviation of a Boolean-based characteristic of the one of the sampled outlier rows of the file is exhibited by less than a threshold number of other of the plurality of rows of the file being displayed.
14. A non-transitory computer useable medium having computer readable program code embodied therein for displaying data of a file via a user interface, the computer readable program code comprising instructions that cause a computer system to:
receive a delimiter that delimits at least some of a plurality of rows of the file, each row of the file associated with values corresponding to a plurality of characteristics;
for each of the plurality of characteristics of rows of the file, determine an aggregate value of the characteristic by aggregating characteristic values corresponding to each row of a set of rows corresponding to at least a portion of the plurality of rows of the file;
identify a criteria for classifying each of the plurality of rows of the file as one of a uniform row or an outlier row, the criteria comprising a comparison of each of one or more characteristic values of the row of the file with the corresponding aggregate value of the characteristic;
classify at least some of the plurality of rows of the file based on the identified criteria to determine a plurality of uniform rows of the file and a plurality of outlier rows of the file, wherein an outlier row of the file represents a row of the file having one or more characteristic values with more than a threshold difference compared to the corresponding aggregate value of the characteristic;
sample, using a first sampling technique, to determine a set of uniform rows of the file from the plurality of uniform rows of the file;
sample, using a second sampling technique, to determine a set of outlier rows of the file from the plurality of outlier rows of the file;
determine a number of rows of the file able to be displayed by the browser window on the user interface;
determine a number of uniform rows of the set of uniform rows to be displayed, based upon (a) a difference between the number of rows able to be displayed by the browser window and a size of the set of outlier rows, and (b) a minimum number of uniform rows;
determine a number of outlier rows of the set of outlier rows to be displayed, based upon a difference between the number of rows able to be displayed by the browser window and the determined number of uniform rows; and
display, via the browser window on user interface:
at least some of the set of uniform rows of the file sampled corresponding to the determined number of uniform rows; and
at least some of the set of outlier rows of the file sampled corresponding to the determined number of outlier rows, wherein the outlier rows are selected in order to stratify the outlier portions across a plurality of characteristics associated with the outlier rows.
15. The non-transitory computer useable medium of claim 14 , wherein the second sampling technique is congressional sampling, without replacement.
16. The non-transitory computer useable medium of claim 14 , wherein the computer readable program code comprises instructions that cause the computer system to:
determine whether one of the sampled outlier or uniform rows of the file should be displayed before other of the plurality of the rows of the file have been classified; and
display said one of the plurality of the rows of the file are responsive to said determining.
17. The non-transitory computer useable medium of claim 16 , wherein the instructions cause the computer system to:
responsive to the one of the sampled outlier or uniform rows of the file being classified as a uniform row of the file, the determining comprises identifying whether a threshold measure of the at least some of the uniform rows of the file have been displayed; and
wherein displaying the one of the plurality of the rows of the file are responsive to the identifying.
18. The non-transitory computer useable medium of claim 16 wherein:
responsive to the one of the sampled outlier or uniform rows of the file being classified as an outlier row of the file, the determining comprises identifying whether a deviation of a numeric-based characteristic of the one of the sampled outlier rows of the file relative to a plurality of other of the portions of the file exceeds a threshold measure; and
wherein displaying the one of the plurality of the rows of the file responsive to said determining are responsive to the identifying.
19. The non-transitory computer useable medium non-transitory computer useable medium of claim 16 wherein:
responsive to the one of the sampled outlier or uniform rows of the file being classified as an outlier row of the file, the determining comprises identifying whether a deviation of a Boolean-based characteristic of the one of the sampled outlier rows of the file is exhibited by less than a threshold number of other of the plurality of rows of the file being displayed; and
wherein displaying the one of the plurality of the rows of the file responsive to said determining are responsive to the identifying.