Profiling in a massive parallel processing environment
A computer-implemented method of profiling a data set in a parallel processing environment includes vertically partitioning an initial data set. One or more attribute subsets are then profiled. A list of subjects is generated each corresponding to a specific attribute value identified in the profiling. Values of multiple attributes are extracted for each identified subject, and the sample results are assembled and merged.
1. A computer-implemented method of profiling a data set in a parallel processing environment, comprising:
partitioning an initial data set stored in a row based format vertically according to multiple attribute subsets, wherein partitioning comprises extracting data for the multiple attribute subsets from the initial data set;
storing the extracted data for the multiple attribute subsets in a plurality of staging files, the plurality of staging files including extracted data that is static;
profiling the multiple attribute subsets based on a specific attribute value, wherein different staging files are provided to different machines in a plurality of machines for profiling;
determining a set of row identifiers that satisfy the specific attribute value identified in the profiling of the different staging files processed on the plurality of machines;
extracting, for each row identifier in the set of row identifiers, a row identifier, a column identifier, and a value of an attribute for a set of columns associated with each row identifier;
merging the row identifier, the column identifier, and the value of the attribute for the set of columns associated with each row identifier to form a profiled subset of the initial data set; and
transmitting, displaying or storing the profiled subset of the initial data set, a further processed version, or combinations thereof.
2. The method of claim 1 , wherein the initial data set comprises a table having m rows and n columns.
3. The method of claim 2 , wherein each column corresponds to a specific attribute and each row corresponds to a row identifier.
4. The method of claim 1 , further comprising assigning a different machine in the plurality of machines to each of the attribute subsets.
5. The method of claim 4 , wherein the profiling comprises concurrently profiling the attribute subsets assigned to the different machines.
6. The method of claim 1 , wherein the merging comprises removing duplicate subject sets of row identifiers.
7. The method of claim 1 , further comprising:
storing the row identifier, the column identifier, and the value of the attribute for the set of columns associated with each row identifier in a file; and
using the file to display at least one or more values of attributes stored in the file for a set of columns for a row identifier.
8. The method of claim 1 , wherein the extracted data in the staging files is in a binary format.
9. A computer network including a parallel processing environment, comprising:
a data source;
multiple projection computers; and
one or more client computers connected to the multiple projection computers and having computer-readable code embedded therein for programming the projection computers to perform a method of profiling a data set in the parallel processing environment, wherein the method comprises:
partitioning an initial data set stored in a row based format vertically according to multiple attribute subsets, wherein partitioning comprises extracting data for the multiple attribute subsets from the initial data set;
storing the extracted data for the multiple attribute subsets in a plurality of staging files, the plurality of staging files including extracted data that is static;
profiling the multiple attribute subsets based on a specific attribute value, wherein different staging files are provided to different projection computers in the multiple projection computers for profiling;
determining a set of row identifiers that satisfy the specific attribute value identified in the profiling of the different staging files processed on the multiple projection computers;
extracting, for each row identifier in the set of row identifiers, a row identifier, a column identifier, and a value of an attribute for a set of columns associated with each row identifier;
merging the row identifier, the column identifier, and the value of the attribute for the set of columns associated with each row identifier to form a profiled subset of the initial data set; and
transmitting, displaying or storing the profiled subset of the initial data set, a further processed version, or combinations thereof.
10. The computer network of claim 9 , wherein the initial data set comprises a table having m rows and n columns.
11. The computer network of claim 10 , wherein each column corresponds to a specific attribute and each row corresponds to a row identifier.
12. The computer network of claim 9 , wherein the method further comprises assigning a projection computer in the multiple projection computers to each of the multiple attribute data sets.
13. The computer network of claim 12 , wherein the profiling comprises concurrently profiling the multiple data sets assigned to the different projection computer.
14. The computer network of claim 9 , wherein the merging comprises removing duplicate subject sets of row identifiers.
15. The computer network of claim 9 , wherein the merging comprises sorting the row identifiers.
16. One or more non-transitory processor-readable media having embedded therein processor-readable code to program one or more processors to perform a method of profiling a data set in a parallel processing environment, wherein the method comprises:
partitioning an initial data set stored in a row based format vertically according to multiple attribute subsets, wherein partitioning comprises extracting data for the multiple attribute subsets from the initial data set;
storing the extracted data for the multiple attribute subsets in a plurality of staging files, the plurality of staging files including extracted data that is static;
profiling the multiple attribute subsets based on a specific attribute value, wherein different staging files are provided to different machines in a plurality of machines for profiling;
determining a set of row identifiers that satisfy the specific attribute value identified in the profiling of the different staging files processed on the plurality of machines;
extracting, for each row identifier in the set of row identifiers, a row identifier, a column identifier, and a value of an attribute for a set of columns associated with each row identifier;
merging the row identifier, the column identifier, and the value of the attribute for the set of columns associated with each row identifier to form a profiled subset of the initial data set; and
transmitting, displaying or storing the profiled subset of the initial data set, a further processed version, or combinations thereof.
17. The one or more processor-readable media of claim 16 , wherein the initial data set comprises a table having m rows and n columns.
18. The one or more processor-readable media of claim 17 , wherein each column corresponds to a specific attribute and each row corresponds to a row identifier.
19. The one or more processor-readable media of claim 16 , wherein the method further comprises assigning a different machine in the plurality of machines to each of the attribute subsets.
20. The one or more processor-readable media of claim 19 , wherein the profiling comprises concurrently profiling the attribute subsets assigned to the different machines.
21. The one or more processor-readable media of claim 16 , wherein the merging comprises removing duplicate subject sets of row identifiers.
22. The one or more processor-readable media of claim 16 , wherein the merging comprises sorting the row identifiers.