Distributed computing on document formats
View Patent ↗According to one embodiment of the present invention, a system includes at least one processor. The system partitions a document into a plurality of data blocks, wherein each data block comprises one or more complete logical units of the document. A plurality of sub-documents is produced from the plurality of data blocks. The sub-documents are processed in parallel by a plurality of processing elements. Embodiments of the present invention further include a method and computer program product for processing a document in parallel in substantially the same manner described above.
1. A system comprising:
at least one processor configured to:
partition a document into a plurality of data blocks of an equal number of bits and store the plurality of data blocks across a plurality of processing elements of a distributed file system;
in response to determining that one or more data blocks are not partitioned on a boundary of complete logical units of the document, adjust the one or more data blocks of the plurality of data blocks to comprise one or more complete logical units of the document, wherein at least one data block of the one or more data blocks is adjusted by removing data from an adjacent data block that is stored on a different processing element from the processing element of the at least one data block, transferring, via a network, the data from the different processing element of the adjacent data block to the processing element of the at least one data block, and inserting the data into the at least one data block to result in the at least one data block including one or more complete logical units;
produce a plurality of sub-documents from the plurality of data blocks including the one or more adjusted data blocks, wherein each sub-document is a stand-alone document that is formatted in a format of the document; and
process the plurality of sub-documents in parallel by the plurality of processing elements.
2. The system of claim 1 , wherein the document includes a format unable to be partitioned based on a size boundary.
3. The system of claim 1 , wherein the one or more complete logical units are defined according to page boundaries.
4. The system of claim 1 , wherein the document includes a non-sequential data type.
5. The system of claim 1 , wherein the at least one processor is further configured to:
extract resource information for one or more shared resources from the document, wherein extracting resource information for the one or more shared resources comprises severing each link between the one or more shared resources and one or more resource pointers in the document.
6. The system of claim 1 , wherein boundaries of complete logical units are indicated by a schema corresponding to the format of the document.
7. A computer program product comprising one or more computer readable storage media collectively having computer readable program code embodied therewith, the computer readable program code executable by at least one processor to cause the at least one processor to:
partition a document into a plurality of data blocks of an equal number of bits and store the plurality of data blocks across a plurality of processing elements of a distributed file system;
in response to determining that one or more data blocks are not partitioned on a boundary of complete logical units of the document, adjust the one or more data blocks of the plurality of data blocks to comprise one or more complete logical units of the document, wherein at least one data block of the one or more data blocks is adjusted by removing data from an adjacent data block that is stored on a different processing element from the processing element of the at least one data block, transferring, via a network, the data from the different processing element of the adjacent data block to the processing element of the at least one data block, and inserting the data into the at least one data block to result in the at least one data block including one or more complete logical units;
produce a plurality of sub-documents from the plurality of data blocks including the one or more adjusted data blocks, wherein each sub-document is a stand-alone document that is formatted in a format of the document; and
process the plurality of sub-documents in parallel by a plurality of processing elements.
8. The computer program product of claim 7 , wherein the document includes a format unable to be partitioned based on a size boundary.
9. The computer program product of claim 7 , wherein the one or more complete logical units are defined according to page boundaries.
10. The computer program product of claim 7 , wherein the document includes a non-sequential data type.
11. The computer program product of claim 7 , wherein the computer readable code further causes the at least one processor to:
extract resource information for one or more shared resources from the document, wherein extracting resource information for the one or more shared resources comprises severing each link between the one or more shared resources and one or more resource pointers in the document.
12. The computer program product of claim 7 , wherein boundaries of complete logical units are indicated by a schema corresponding to the format of the document.