Efficient conversion of XML data into a model using persistent stores and parallelism
View Patent ↗Methods for product data management and corresponding systems and computer-readable mediums. A method includes receiving an XML document by a parallel parser process, the XML document including a plurality of elements of an XML data structure that corresponds to an object model. The method includes dividing the XML document into a plurality of chunks using the parallel parser process, and parsing the plurality of chunks in parallel using separate parsing tasks to produce objects representing the elements and corresponding attributes. The method includes storing the objects and corresponding attributes in a persistent element store.
1. A method for product data management, the method performed by at least one data processing system and comprising:
receiving an XML document by a parallel parser process, the XML document including a plurality of elements of an XML data structure that corresponds to an object model;
dividing the XML document into a plurality of chunks using the parallel parser process according to a chunking strategy comprising a determination of:
a number of chunks,
a size of each chunk,
an order in which the chunks will be parsed,
an end point of a first chunk of the plurality of chunks, and
a start point of a second chunk of the plurality of chunks, wherein the start point corresponds to the end point, and
wherein the chunking strategy avoids processing the second chunk before the determination of the end point of the first chunk by processing non-adjacent chunks before adjacent chunks;
parsing the plurality of chunks in parallel using separate parsing tasks to produce objects representing the elements and corresponding attributes; and
storing the objects and corresponding attributes in a persistent element store.
2. The method of claim 1 , wherein the data processing system also selectively retrieves the objects and corresponding attributes from the element store;
creates one or more model objects from the retrieved objects and corresponding attributes using a plurality of modeling tasks operating in parallel; and
stores the one or more model objects in a model object store.
3. The method of claim 2 , wherein the data processing system also controls the modeling tasks using a model object build process, and creates an initial structure in the model object store before starting the modeling tasks.
4. The method of claim 1 , wherein the XML document is received from an XML reader process that reads the XML document and passes it to a parallel parser process to produce the plurality of chunks as XML streams.
5. The method of claim 1 , wherein the XML document is divided into the plurality of chunks based on division points in the XML document identified by the parallel parser process.
6. The method of claim 1 , wherein each parsing task operates in a separate parsing thread in a different processor core.
7. The method of claim 1 , wherein the system uses a parsing manager that manages the completion of parsing tasks and instantiates parsing tasks as needed.
8. A data processing system comprising:
a processor; and
an accessible memory, the data processing system particularly configured to
receive an XML document by a parallel parser process, the XML document including a plurality of elements of an XML data structure that corresponds to an object model;
divide the XML document into a plurality of chunks using the parallel parser process according to a chunking strategy comprising a determination of:
a number of chunks,
a size of each chunk,
an order in which the chunks will be parsed,
an end point of a first chunk of the plurality of chunks, and
a start point of a second chunk of the plurality of chunks, wherein the start point corresponds to the end point, and
wherein the chunking strategy avoids processing the second chunk before the determination of the end point of the first chunk by processing non-adjacent chunks before adjacent chunks;
parse the plurality of chunks in parallel using separate parsing tasks to produce objects representing the elements and corresponding attributes; and
store the objects and corresponding attributes in a persistent element store.
9. The data processing system of claim 8 , wherein the data processing system also selectively retrieves the objects and corresponding attributes from the element store;
creates one or more model objects from the retrieved objects and corresponding attributes using a plurality of modeling tasks operating in parallel; and
stores the one or more model objects in a model object store.
10. The data processing system of claim 9 , wherein the data processing system also controls the modeling tasks using a model object build process, and creates an initial structure in the model object store before starting the modeling tasks.
11. The data processing system of claim 8 , wherein the XML document is received from an XML reader process that reads the XML document and passes it to a parallel parser process to produce the plurality of chunks as XML streams.
12. The data processing system of claim 8 , wherein the XML document is divided into the plurality of chunks based on division points in the XML document identified by the parallel parser process.
13. The data processing system of claim 8 , wherein each parsing task operates in a separate parsing thread in a different processor core.
14. The data processing system of claim 8 , wherein the system uses a parsing manager that manages the completion of parsing tasks and instantiates parsing tasks as needed.
15. A non-transitory computer-readable medium encoded with executable instructions that, when executed, cause one or more data processing systems to:
receive an XML document by a parallel parser process, the XML document including a plurality of elements of an XML data structure that corresponds to an object model;
divide the XML document into a plurality of chunks using the parallel parser process according to a chunking strategy comprising a determination of:
a number of chunks,
a size of each chunk,
an order in which the chunks will be parsed,
an end point of a first chunk of the plurality of chunks, and
a start point of a second chunk of the plurality of chunks, wherein the start point corresponds to the end point, and
wherein the chunking strategy avoids processing the second chunk before the determination of the end point of the first chunk by processing non-adjacent chunks before adjacent chunks;
parse the plurality of chunks in parallel using separate parsing tasks to produce objects representing the elements and corresponding attributes; and
store the objects and corresponding attributes in a persistent element store.
16. The computer-readable medium of claim 15 , wherein the data processing system also
selectively retrieves the objects and corresponding attributes from the element store;
creates one or more model objects from the retrieved objects and corresponding attributes using a plurality of modeling tasks operating in parallel; and
stores the one or more model objects in a model object store.
17. The computer-readable medium of claim 16 , wherein the data processing system also controls the modeling tasks using a model object build process, and creates an initial structure in the model object store before starting the modeling tasks.
18. The computer-readable medium of claim 15 , wherein the XML document is received from an XML reader process that reads the XML document and passes it to a parallel parser process to produce the plurality of chunks as XML streams.
19. The computer-readable medium of claim 15 , wherein the XML document is divided into the plurality of chunks based on division points in the XML document identified by the parallel parser process.
20. The computer-readable medium of claim 15 , wherein each parsing task operates in a separate parsing thread in a different processor core and the system uses a parsing manager that manages the completion of parsing tasks and instantiates parsing tasks as needed.