Processing structured data
The present invention provides a fast and efficient way of processing structured data by utilizing an intermediate file to store the structural information. The structured data may be processed into a Binary mask Format (BMF) file which may serve as a starting point for post-processing. A tree structure built on top of the BMF file may be constructed very quickly, and also takes up less space than a DOM tree. Additionally, BMF records may reside entirely in the memory and contain structural information, allowing SAX-like sequential data access.
1. A method for efficiently processing a structured data file, the structured data file including one or more pieces of content, comprising:
parsing the structured data file by:
creating a first record in an intermediate file, the first record corresponding to a starting tag in the structured data file and having information regarding the starting tag and a content type field identifying the first record as a starting tag type;
creating a second record in the intermediate file, the second record corresponding to a piece of content in the structured data file, the second record containing one or more descriptors containing information regarding the property name, wherein the second record includes an offset indicting a position for said piece of content relative to a point in said structured data file;
creating a third record in the intermediate file, the third record corresponding to a piece of content in the structured data file, the third record containing one or more descriptors containing information regarding the property value, wherein the third record includes an offset indicting a position for said piece of content relative to a point in said structured data file;
creating a fourth record in the intermediate file, the fourth record corresponding to a piece of content in the structured data file, the fourth record containing one or more descriptors containing information regarding the content, wherein the fourth record includes an offset indicting a position for said piece of content relative to a point in said structured data file;
creating a fifth record in the intermediate file, the fifth record corresponding to an ending tag in the structured data file and having information regarding the ending tag and a content type field identifying the fifth record as an ending tag type; and
formatting said intermediate file in a way that allows data from the structured data file to be accessed using both said intermediate file and the structured data file together.
2. The method of claim 1 , wherein said creating a second record comprises:
creating a binary mask format (BMF) record in the intermediate file, the BMF record corresponding to said piece of content.
3. The method of claim 1 , wherein said intermediate file is a BMF file, wherein the BMF file includes one or more BMF records.
4. The method of claim 2 , wherein said creating a second record further comprises:
storing a depth value of said piece of content in said BMF record, said depth value indicating a level in a hierarchy of tags in said structured data file.
5. The method of claim 2 , wherein said creating a second record further comprises:
storing a content type of said piece of content in said BMF record, said content type indicating a type of information for said piece of content.
6. The method of claim 2 , wherein said creating a second record further comprises:
storing a length for said piece of content in said BME record.
7. The method of claim 2 , wherein said offset indicates a starting position for said piece of content relative to a beginning of the structured data file.
8. The method of claim 4 , wherein the formatting includes formatting said record in a way that allows data to be accessed using both said intermediate file and the structured data file without traversing the entire structured data file to determine the depth value.
9. An apparatus for efficiently processing structured data, comprising:
a peripheral component interface (PCI) interface;
a direct memory access (DMA) engine coupled to said PCI interface;
a text processor coupled to said PCI interface, the text processor configured to parse a structured data the by:
creating a first record in an intermediate file, the first record corresponding to a starting tag in the structured data file and having information regarding the starting tag and a content type field identifying the first record as a starting tag type;
creating a second record in the intermediate file, the second record corresponding to a piece of content in the structured data file, the second record containing one or more descriptors containing information regarding the property name, wherein the second record includes an offset indicting a position for said piece of content relative to a point in said structured data file;
creating a third record in the intermediate file, the third record corresponding to a piece of content in the structured data file, the third record containing one or more descriptors containing information regarding the property value, wherein the third record includes an offset indicting a position for said piece of content relative to a point in said structured data file;
creating a fourth record in the intermediate file, the fourth record corresponding to a piece of content in the structured data file, the fourth record containing one or more descriptors containing information regarding the content, wherein the fourth record includes an offset indicting a position for said piece of content relative to a point in said structured data file;
creating a fifth record in the intermediate file, the fifth record corresponding to an ending tag in the structured data file and having information regarding the ending tag and a content type field identifying the fifth record as an ending tag type; and
formatting said intermediate file in a way that allows data from the structured data file to be accessed using both said intermediate file and the structured data file together.
10. The apparatus of claim 9 , wherein said creating a second record comprises:
creating a binary mask format (BMF) record in the intermediate file, the BMF record corresponding to said piece of content.
11. The apparatus of claim 9 , wherein said intermediate file is a BMF file, wherein the BMF file includes one or more BMF records.
12. The apparatus of claim 10 , wherein said creating a second record further comprises:
storing a depth value of said piece of content in said BMF record, said depth value indicating a level in a hierarchy of tags in said structured data file.
13. The apparatus of claim 10 , wherein said creating a second record further comprises:
storing a content type of said piece of content in said BMF record, said content type indicating a type of information for said piece of content.
14. The apparatus of claim 10 , wherein said creating a second record further comprises:
storing a length for said piece of content in said BME record.
15. The apparatus of claim 10 , wherein said offset indicates a starting position for said piece of content relative to a beginning of the structured data file.
16. The apparatus of claim 12 , wherein the formatting includes formatting said record in a way that allows data to be accessed using both said intermediate file and the structured data file without traversing the entire structured data file to determine the depth value.
17. A non-transitory storage device readable by a machine, tangibly embodying a program of instructions executable by the machine to perform a method for efficiently processing a structured data file, the structured data file including one or more pieces of content, the method comprising:
parsing the structured data file by:
creating a first record in an intermediate file, the first record corresponding to a starting tag in the structured data file and having information regarding the starting tag and a content type field identifying the first record as a starting tag type;
creating a second record in the intermediate file, the second record corresponding to a piece of content in the structured data file, the second record containing one or more descriptors containing information regarding the property name, wherein the second record includes an offset indicting a position for said piece of content relative to a point in said structured data file;
creating a third record in the intermediate file, the third record corresponding to a piece of content in the structured data file, the third record containing one or more descriptors containing information regarding the property value, wherein the third record includes an offset indicting a position for said piece of content relative to a point in said structured data file;
creating a fourth record in the intermediate file, the fourth record corresponding to a piece of content in the structured data file, the fourth record containing one or more descriptors containing information regarding the content, wherein the fourth record includes an offset indicting a position for said piece of content relative to a point in said structured data file;
creating a fifth record in the intermediate file, the fifth record corresponding to an ending tag in the structured data file and having information regarding the ending tag and a content type field identifying the fifth record as an ending tag type; and
formatting said intermediate file in a way that allows data from the structured data file to be accessed using both said intermediate file and the structured data file together.
18. The non-transitory storage device of claim 17 , wherein said creating a second record comprises:
creating a binary mask format (BMF) record in the intermediate file, the BMF record corresponding to said piece of content.
19. The non-transitory storage device of claim 17 , wherein said intermediate file is a BMF file, wherein the BMF file includes one or more BMF records.
20. The non-transitory storage device of claim 18 , wherein sad creating a second record further comprises:
storing a depth value of said piece of content in said BMF record, said depth value indicating a level in a hierarchy of tags in said structured data file.