Parallel parsing of file partitions storing a single XML document

1. A method for parsing at least a portion of an XML document in parallel, comprising steps of:

assigning a plurality of file partitions of an XML file that contains said XML document to a plurality of computing nodes by, for each computing node of the plurality of computing nodes, assigning to said each computing node a respective target element of said XML document and a respective file partition of said plurality of file partitions;

in parallel with other computing nodes of said plurality of computing nodes, each computing node of said plurality of the computing nodes:

receiving input specifying the respective target element by name and respective file partition;

scanning the respective file partition for a beginning of the respective target element; and

when the beginning of the respective target element is found, said each computing node parsing the respective target element;

wherein assigning a plurality of file partitions to a plurality of computing nodes includes assigning a first file partition of said plurality of file partitions to a first computing node of said plurality of computing nodes and assigning a second file partition of said plurality of file partitions to a second computing node of said plurality of computing nodes;

wherein the respective target element of the first computing node starts in the first file partition and extends into the second file partition; and

wherein the first computing node parsing the respective target element comprises the first computing node parsing the respective target element from the first file partition and second file partition.

2. The method of claim 1 ,

wherein said XML file is a distributed file stored in a distributed file system comprising a plurality of storage media devices; and

wherein said XML, file comprises segments, wherein each segment of said segments is stored on a separate respective storage media device of said plurality of storage media devices; and

the steps further including establishing a file partition of said file partitions for each segment of said segments.

3. The method of claim 2 , wherein a respective storage media device that stores each segment of said segments is directly accessible by a computing node of said plurality of computing nodes;

wherein other storage media devices of said plurality of storage media devices are not directly accessible by said computing node of said plurality of computing nodes.

4. The method of claim 1 , wherein for each computing node of said plurality of computing nodes, the input specifying the respective target element and respective file partition further specifies one or more names spaces in scope for the respective target element.

5. The method of claim 1 , wherein any target element assigned by said assigning is not nested within any other target element assigned by said assigning.

6. The method of claim 1 ,

wherein for each computing node of said plurality of computing nodes, the input specifying the respective target element by name and respective file partition further specifies names spaces in scope for the respective target element;

wherein said XML document satisfies a set of constraints comprising at least one of:

any CDATA does not contain data marked up as a target element;

any comment does not contain data marked up as a target element; and

any processing instruction does not contain data marked up as a target element.

7. The method of claim 1 , wherein said XML file is stored in distributed file system that comprises said plurality of computing nodes and a plurality of storage media devices.

8. One or more non-transitory storage media storing instructions for parsing at least a portion of an XML document in parallel, said instructions which, when executed by one or more computing devices, cause:

assigning a plurality of file partitions of an XML file that contains said XML document to a plurality of computing nodes by, for each computing node of the plurality of computing nodes, assigning to said each computing node a respective target element of said XML document and a respective file partition of said plurality of file partitions; and

in parallel with other computing nodes of said plurality of computing nodes, each computing node of said plurality of the computing nodes:

receiving input specifying the respective target element by name and respective file partition,

scanning the respective file partition for a beginning of the respective target element, and

when the beginning of the respective target element is found, said each computing node parsing the respective target element;

wherein assigning a plurality of file partitions to a plurality of computing nodes includes assigning a first file partition of said plurality of file partitions to a first computing node of said plurality of computing nodes and assigning a second file partition of said plurality of file partitions to a second computing node of said plurality of computing nodes;

wherein the respective target element of the first computing node starts in the first file partition and extends into the second file partition; and

wherein the first computing node parsing the respective target element comprises the first computing node parsing the respective target element from the first file partition and second file partition.

9. The one or more non-transitory storage media of claim 8 ,

wherein said XML file is a distributed file stored in a distributed file system comprising a plurality of storage media devices; and

wherein said XML, file comprises segments, wherein each segment of said segments is stored on a separate respective storage media device of said plurality of storage media devices; and

wherein said instructions include instructions that, when executed by said one or more computing devices, cause: establishing a file partition of said file partitions for each segment of said segments.

10. The one or more non-transitory storage media of claim 9 , wherein a respective storage media device that stores each segment of said segments is directly accessible by a computing node of said plurality of computing nodes;

wherein other storage media devices of said plurality of storage media devices are not directly accessible by said computing node of said plurality of computing nodes.

11. The one or more non-transitory storage media of claim 8 , wherein for each computing node of said plurality of computing nodes, the input specifying the respective target element and respective file partition further specifies one or more namesspaces in scope for the respective target element.

12. The one or more non-transitory storage media of claim 9 , wherein any target element assigned by said assigning is not nested within any other target element assigned by said assigning.

13. The one or more non-transitory storage media of claim 8 ,

wherein for each computing node of said plurality of computing nodes, the input specifying the respective target element by name and respective file partition further specifies names spaces in scope for the respective target element; and

wherein said XML document satisfies a set of constraints comprising at least one of:

any CDATA does not contain data marked up as a target element,

any comment does not contain data marked up as a target element a target element, and

any processing instruction does not contain data marked up as a target element a target element.

14. The one or more non-transitory storage media of claim 8 ,

wherein said XML file is stored in a distributed file system that comprises said plurality of computing nodes and a plurality of storage media devices.

15. A computer system including one or more processors comprising:

one or more computing nodes configured to parse at least a portion of an XML document in parallel, wherein each of the one or more computing nodes is configured to:

receive assignment of a respective target element of said XML document and a respective file partition of a plurality of file partitions of an XML file that contains said XML document;

in parallel with other computing nodes of said one or more computing nodes:

receive input specifying the respective target element by name and respective file partition;

scan the respective file partition for a beginning of the respective target element; and

parse the respective target element, when the beginning of the respective target element is found;

wherein assigning a plurality of file partitions to a plurality of computing nodes includes assigning a first file partition of said plurality of file partitions to a first computing node of said plurality of computing nodes and assigning a second file partition of said plurality of file partitions to a second computing node of said plurality of computing nodes;

wherein the respective target element of the first computing node starts in the first file partition and extends into the second file partition; and

wherein the first computing node parsing the respective target element comprises the first computing node parsing the respective target element from the first file partition and second file partition.

16. The system of claim 15 ,

wherein said XML file is a distributed file stored in a distributed file system comprising a plurality of storage media devices; and

wherein said XML, file comprises segments, wherein each segment of said segments is stored on a separate respective storage media device of said plurality of storage media devices; and

wherein each of the one or more computing nodes is further configured to: establish a file partition of said file partitions for each segment of said segments.

17. The system of claim 16 , wherein a respective storage media device that stores each segment of said segments is directly accessible by a computing node of said one or more computing nodes;

wherein other storage media devices of said plurality of storage media devices are not directly accessible by said computing node of said plurality of computing nodes.

18. The system of claim 15 , wherein for each computing node of said one or more computing nodes, the input specifying the respective target element and respective file partition further specifies one or more names spaces in scope for the respective target element.

19. The system of claim 15 , wherein any target element assigned by said assigning is not nested within any other target element assigned by said assigning.

20. The system of claim 15 ,

wherein for each computing node of said one or more computing nodes, the input specifying the respective target element by name and respective file partition further specifies names spaces in scope for the respective target element.

wherein said XML document satisfies a set of constraints comprising at least one of:

any CDATA does not contain data marked up as a target element;

any comment does not contain data marked up as a target element a target element; and

any processing instruction does not contain data marked up as a target element a target element.

21. The system of claim 15 , wherein said XML file is stored in a distributed file system that comprises said one or more computing nodes and a plurality of storage media devices.