Systems and/or methods for delayed encoding of XML information sets
Certain example embodiments described herein relate to techniques for processing XML documents of potentially very large sizes. For instance, certain example embodiments parse a potentially large XML document, store the parsed data and some associated metadata in multiple independent blocks or partitions, and instantiate only the particular object model object requested by a program. By including logical references rather than physical memory addresses in such pre-parsed partitions, certain example embodiments make it possible to move the partitions through a caching storage hierarchy without necessarily having to adjust or encode memory references, thereby advantageously enabling dynamic usage of the created partitions and making it possible to cache an arbitrarily large document while consuming a limited amount of program memory. Additionally, certain example embodiments relate to, in addition to delaying the instantiation of nodes, delaying the encoding of respective segments of the XML document until associated nodes are referenced.
1. A method of processing an XML document, comprising:
pre-parsing, using at least one processor, an XML document to generate a plurality of XML segments and metadata for each generated XML segment, the XML document including a plurality of objects representable in accordance with an object model, wherein pre-parsing the XML document includes dividing the entire XML document into the plurality of generated XML segments, with each generated XML segment including a portion of raw XML source code of the XML document and a plurality of nodes, and the metadata for the respective generated XML segment identifying a range of node identifiers representing the plurality of nodes in the raw XML source code of the respective generated XML segment;
storing the generated XML segments and the metadata in a data store;
receiving a request for an object from a user program; and
in response to the request received for the object from the user program:
identifying, based on the generated metadata, at least one of the generated XML segments, stored in the data store, having the requested object;
parsing the identified at least one XM′L segment into one or more constituent nodes;
encoding the identified at least one XM′L segment into one or more cacheable partitions, each said cacheable partition including a locator array, a properties array, and a character array, wherein the properties array is configured to store a plurality of encoded nodes, as well as, for each said encoded node: (1) metadata including a respective node type, (2) reference(s) to any familial nodes thereof, and (3) offset(s) into the character array for any attribute or text value(s) associated therewith, wherein the locator array identifies starting positions of nodes encoded in the properties array;
storing the one or more cacheable partitions in the data store;
after the identified at least one XML segment is encoded into the one or more cacheable partitions, releasing the identified at least one XML segment from the data store storing the generated XML segments without releasing the other XML segments stored in the data store until they are encoded into one or more cacheable partitions; and
instantiating said requested object using the one or more cacheable partitions and the one or more constituent nodes in accordance with the object model, wherein the data store is a repository storing data outside of the user program's memory space.
2. A non-transitory computer readable storage medium having instructions stored thereon that, when executed by a computer, cause the computer to perform operations comprising:
pre-parsing an XML document to generate a plurality of XML segments and metadata for each generated XML segment, the XML document including a plurality of objects representable in accordance with an object model, wherein pre-parsing the XML document includes dividing the entire XML document into the plurality of generated XML segments, with each generated XML segment including a portion of raw XML source code of the XML document and a plurality of nodes, and the metadata for the respective generated XML segment identifying a range of node identifiers representing the plurality of nodes in the raw XML source code of the respective generated XML segment;
storing the generated XML segments and the metadata in a data store; and
in response to a request received for an object from a user program:
identifying, based on the generated metadata, at least one of the generated XML segments, stored in the data store, having the requested object;
parsing the identified at least one XML segment into one or more constituent nodes;
encoding the identified at least one XML segment into one or more cacheable partitions, each said cacheable partition including a locator array, a properties array, and a character array, wherein the properties array is configured to store a plurality of encoded nodes, as well as, for each said encoded node: (1) metadata including a respective node type, (2) reference(s) to any familial nodes thereof, and (3) offset(s) into the character array for any attribute or text value(s) associated therewith, wherein the locator array identifies starting positions of nodes encoded in the properties array;
storing the one or more cacheable partitions in the data store;
after the identified at least one XML segment is encoded into the one or more cacheable partitions, releasing the identified at least one XML segment from the data store storing the generated XML segments without releasing the other XML segments stored in the data store until they are encoded into one or more cacheable partitions; and
instantiating said requested object using the one or more cacheable partitions and the one or more constituent nodes in accordance with the object model, wherein the data store is a repository storing data outside of the user program's memory space.
3. A system for processing XML, documents, comprising
processing resources including at least one processor and a memory,
wherein the processing resources are configured to perform operations comprising:
pre-parsing an XML document to generate a plurality of XML segments and metadata for each generated XML segment, the XML document including a plurality of objects representable in accordance with an object model, wherein pre-parsing the XML document includes dividing the entire XML document into the plurality of generated XML segments, with each generated XML segment including a portion of raw XML source code of the XML document and a plurality of nodes, and the metadata for the respective generated XML segment identifying a range of node identifiers representing the plurality of nodes in the raw XML source code of the respective generated XML segment;
storing the generated XML segments and the metadata in a data store; and
in response to a request received for an object from a user program:
identifying, based on the generated metadata, at least one of the generated XML segments, stored in the data store, having the requested object;
parsing the identified at least one XML segment into one or more constituent nodes;
encoding the identified at least one XML segment into one or more cacheable partitions, each said cacheable partition including a locator array, a properties array, and a character array, wherein the properties array is configured to store a plurality of encoded nodes, as well as, for each said encoded node: (1) metadata including a respective node type, (2) reference(s) to any familial nodes thereof, and (3) offset(s) into the character array for any attribute or text value(s) associated therewith, wherein the locator array identifies starting positions of nodes encoded in the properties array;
storing the one or more cacheable partitions in the data store;
after the identified at least one XML segment is encoded into the one or more cacheable partitions, releasing the identified at least one XML segment from the data store storing the generated XML segments without releasing the other XML segments stored in the data store until they are encoded into one or more cacheable partitions; and
instantiating said requested object using the one or more cacheable partitions and the one or more constituent nodes in accordance with the object model, wherein the data store is a repository storing data outside of the user program's memory space.
4. The system of claim 3 , wherein parsing the identified at least one XML segment comprises parsing only the identified at least one XML segment in response to the request.
5. The system of claim 3 , wherein instantiating said requested object comprises instantiating only said requested object in response to the request.
6. The system of claim 3 , wherein pre-parsing an XML document further comprises:
for each generated XML segment from the plurality of the XML segments, generating a stack of XML namespace prefixes as the XML namespace prefixes exist in the XML document when pre-parsing of the segment is started; and
storing the stack of XML namespace prefixes in the metadata associated with the generated XML segment.
7. The system of claim 3 , wherein pre-parsing an XML document further comprises:
for each generated XML segment from the plurality of the XML segments, determining a state before beginning the pre-parsing of the XML document; and
storing the determined state as at least part of the metadata associated with the generated XML segment.
8. The system of claim 3 , wherein the processing resources are further configured to, in response to the request, store the one or more constituent nodes in the cacheable partition having the associated generated XML segment.
9. The system of claim 3 , wherein each said reference is an allocation identifier.
10. The system of claim 3 , wherein the processing resources are further configured to remove partitions of the parsed document from the memory or a non-transitory computer readable storage medium when the user program no longer includes any references to the document or any objects thereof.
11. The system of claim 3 , wherein the XML document is pre-parsed by executing a pre-parsing initialization that includes:
creating a cacheable document node that corresponds to the starting point for user program access to the XML document;
allocating a name dictionary that includes an indexed entry for each unique XML tag name included in the XML document;
allocating a namespace dictionary that includes an index entry for each unique XML namespace included in the XML document; and
allocating a partition table that includes a list of cacheable partitions, XML segments, and an allocation array that allows a node's allocation identifier to be resolved to a specific cacheable partition, each said cacheable partition including metadata from the pre-parsing initialization.
12. The system of claim 11 , wherein the pre-parsing initialization further includes allocating an attribute value dictionary that stores unique occurrences of attribute values included in the XML document, or a text value dictionary that stores unique occurrences of text values included in the XML document.
13. The system of claim 3 , wherein said parsing the identified at least one XML segment comprises:
recognizing parsing events of predefined parsing event types within the XML segment;
creating nodes for the recognized parsing events;
adding the created nodes to a current partition while there is sufficient space therein; and
creating a new partition when there is insufficient space in the current partition for adding created nodes, updating the partition table, and continuing with the adding by treating the newly created partition as the current partition.
14. The system according to claim 13 , wherein the partitions and the partition table are movable from the memory to a non-transitory computer readable storage medium when moving the XML document from session to session.
15. The system of claim 3 , wherein the processing resources are further configured to:
receive an update request to a node in the identified at least one XML segment;
in response to the update request, encode a replacement node with new information requested in the update request into one of the cacheable partitions; and
replace references to an encoded node corresponding to the node for which the update is received with a reference to encoded replacement node.