Data cell cluster identification and table transformation
Various embodiments may operate to access individual lines of information included in a file stored in an electronic storage medium, to detect the existence of data clusters in the file based on neighboring cell content in a horizontal direction (corresponding to the individual lines), and in a vertical direction (orthogonal to the horizontal direction), to identify at least some of the data clusters as being associated with predefined table types (comprising vertical tables, horizontal tables, or cross tables), to merge some of the data clusters into section tables having common properties, and to transform the tables resulting from the merging activity, as well as remaining un-merged data clusters, into a single flat table. The stored file may comprise a spreadsheet file.
1. An apparatus, comprising:
a processing node, implemented by one or more hardware processors, comprising a data reception module to sequentially access individual lines of information included in a file stored in an electronic storage medium; and
a transformation module, implemented by the one or more hardware processors, to couple to the data reception module, the transformation module configured
to detect existence of data clusters in the file, the detecting comprising associating a cell with a new data cluster or an existing data cluster, based on comparing the cell with adjacent cells in a horizontal direction corresponding to the individual lines,
to generate metadata including characteristics associated with each of the data clusters,
to identify at least some of the data clusters as being associated with predefined table types comprising vertical tables, horizontal tables, or cross tables,
to merge one or more of the data clusters having common cluster table schemas into section tables, based on the metadata, and
to transform the vertical tables, the horizontal tables, the cross tables, the section tables, and remaining un-merged data clusters into a single flat table having a vertical or horizontal orientation, based on the metadata.
2. The apparatus of claim 1 , wherein the transformation module is further configured to determine that said one or more of the data clusters have the common cluster table schemas, based on the metadata.
3. The apparatus of claim 1 , wherein the transformation module is further configured to adjust values included in the metadata that identify a size of one of the data clusters, based on additional cells belonging to the one of the data clusters discovered during the detecting.
4. The apparatus of claim 1 , wherein the metadata includes sequence numbers corresponding to the data clusters, the sequence numbers indicating a sequence in which the data clusters are detected.
5. A system, comprising:
a first processing node, implemented by one or more hardware processors, comprising a data reception module to sequentially access individual lines of information included in a file stored in an electronic storage medium; and
a second processing node comprising a transformation module implemented by the one or more hardware processors to couple to the data reception module in the first processing node, the transformation module
to detect existence of data clusters in the file, the detecting comprising associating a cell with a new data cluster or an existing data cluster, based on comparing the cell with adjacent cells in a horizontal direction corresponding to the individual lines,
to generate metadata including characteristics associated with each of the data clusters,
to identify at least some of the data clusters as being associated with predefined table types comprising vertical tables, horizontal tables, or cross tables,
to merge one or more of the data clusters having common cluster table schemas into section tables, based on the metadata, and
to transform the vertical tables, the horizontal tables, the cross tables, the section tables, and remaining un-merged data clusters into a single flat table having a vertical or horizontal orientation, based on the metadata.
6. The system of claim 5 , wherein the transformation module is further configured to determine that said one or more of the data clusters have the common cluster table schemas, based on the metadata.
7. The system of claim 5 , wherein the transformation module is further configured to adjust values included in the metadata that identify a size of one of the data clusters, based on additional cells belonging to the one of the data clusters discovered during the detecting.
8. A processor-implemented method to execute on one or more processors that perform the method, comprising:
accessing, sequentially, individual lines of information included in a file stored in an electronic storage medium;
detecting existence of data clusters in the file, the detecting comprising associating a cell with a new data cluster or an existing data cluster, based on comparing the cell with adjacent cells in a horizontal direction corresponding to the individual lines,
generating metadata including characteristics associated with each of the data clusters,
identifying at least some of the data clusters as being associated with predefined table types comprising vertical tables, horizontal tables, or cross tables,
merging one or more of the data clusters having common cluster table schemas into section tables, based on the metadata, and
transforming the vertical tables, the horizontal tables, the cross tables, the section tables, and remaining un-merged data clusters into a single flat table having a vertical or horizontal orientation, based on the metadata.
9. The method of claim 8 , wherein the metadata includes edge coordinate values identifying the upper edge, lower edge, left edge and right edge of each of the data clusters.
10. The method of claim 8 , wherein the metadata includes sequence numbers corresponding to the data clusters, the sequence numbers indicating a sequence in which the data clusters are detected.
11. The method of claim 8 , wherein the metadata includes section header position information describing a position of a section header associated with at least one of the data clusters.
12. The method of claim 8 , wherein the metadata includes column header labels of at least one of the data clusters.
13. The method of claim 8 , further comprising:
adjusting values included in the metadata identifying a size of one of the data clusters, based on additional cells belonging to the one of the data clusters discovered during the detecting.
14. The method of claim 8 , wherein the merging further comprises:
determining that said one or more of the data clusters have the common cluster table schemas, based on the metadata.
15. The method of claim 14 , wherein the determining further comprises:
determining that left edges and right edges of said one or more of the data clusters are the same, based on edge coordinate values included in the metadata.
16. The method of claim 14 , wherein the determining further comprises:
determining that column header labels of said one or more of the data clusters are the same, based on the metadata.
17. The method of claim 14 , wherein the determining further comprises:
determining that relative section header positions of said one or more of the data clusters are the same, based on section header position information included in the metadata.
18. The method of claim 8 , wherein the transforming further comprises:
generating column headers by automatic assignment for one of the vertical tables upon detecting absence of a column header associated with the one of the vertical tables.
19. A non-transitory computer-readable storage medium containing executable instructions stored thereon which, when executed, result in a processor performing:
accessing, sequentially, individual lines of information included in a file stored in an electronic storage medium;
detecting existence of data clusters in the file, the detecting comprising associating a cell with a new data cluster or an existing data cluster, based on comparing the cell with adjacent cells in a horizontal direction corresponding to the individual lines,
generating metadata including characteristics associated with each of the data clusters,
identifying at least some of the data clusters as being associated with predefined table types comprising vertical tables, horizontal tables, or cross tables,
merging one or more of the data clusters having common cluster table schemas into section tables, based on the metadata, and
transforming the vertical tables, the horizontal tables, the cross tables, the section tables, and remaining un-merged data clusters into a single flat table having a vertical or horizontal orientation, based on the metadata.
20. The non-transitory computer-readable storage medium of claim 19 , containing executable instructions stored thereon which, when executed, result in the processor further performing:
adjusting values included in the metadata identifying a size of one of the data clusters, based on additional cells belonging to the one of the data clusters discovered during the detecting.