Methods and apparatus for capture and storage of semantic information with sub-files in a parallel computing system

1. A method for storing at least one file generated by a distributed application in a parallel computing system, wherein said file comprises a plurality of sub-files, said method comprising the steps of:

obtaining a user specification of semantic information related to said file;

determining semantically meaningful sub-file boundaries for said plurality of sub-files based on said semantic information;

providing said semantic information as a data structure description to a data formatting library write function, wherein said semantic information is dependent upon a content of said file; and

storing said semantic information related to said file with additional metadata related to said file and with one or more of said sub-files using said determined semantically meaningful sub-file boundaries in one or more storage nodes of said parallel computing system.

2. The method of claim 1 , wherein said semantic information provides a description of data in said file.

3. The method of claim 1 , wherein said sub-files are replicated based on said semantically meaningful boundaries.

4. The method of claim 1 , further comprising the step of processing a query using said semantic information.

5. The method of claim 1 , further comprising the step of performing an analysis of said file using said semantic information.

6. The method of claim 1 , wherein a replication strategy can be specified for each of said plurality of said sub-files.

7. The method of claim 1 , wherein said one or more storage nodes reside on one or more tiers of a multi-tier storage system.

8. The method of claim 1 , wherein said semantic information related to a given sub-file is stored with said corresponding sub-file.

9. An apparatus for storing at least one file generated by a distributed application in a parallel computing system, wherein said file comprises a plurality of sub-files, comprising:

a memory; and

at least one hardware device operatively coupled to the memory and configured to:

obtain a user specification of semantic information related to said file;

determine semantically meaningful sub-file boundaries for said plurality of sub-files based on said semantic information;

provide said semantic information as a data structure description to a data formatting library write function, wherein said semantic information is dependent upon a content of said file; and

store said semantic information related to said file with additional metadata related to said file and with one or more of said sub-files using said determined semantically meaningful sub-file boundaries in one or more storage nodes of said parallel computing system.

10. The apparatus of claim 9 , wherein said semantic information provides a description of data in said file.

11. The apparatus of claim 9 , wherein said sub-files are replicated based on said semantically meaningful boundaries.

12. The apparatus of claim 9 , wherein said at least one hardware device is further configured to process a query using said semantic information.

13. The apparatus of claim 9 , wherein said at least one hardware device is further configured to perform an analysis of said file using said semantic information.

14. The apparatus of claim 9 , wherein said one or more storage nodes reside on one or more tiers of a multi-tier storage system.

15. The apparatus of claim 9 , wherein said semantic information related to a given sub-file is stored with said corresponding sub-file.

16. A data storage system for storing at least one file generated by a distributed application in a parallel computing system, wherein said file comprises a plurality of sub-files, comprising:

a hardware processing device for obtaining a user specification of semantic information related to said file;

determining semantically meaningful sub-file boundaries for said plurality of sub-files based on said semantic information; and for providing said semantic information as a data structure description to a data formatting library write function, wherein said semantic information is dependent upon a content of said file; and

a memory for storing said semantic information related to said file with additional metadata related to said file and with one or more of said sub-files using said determined semantically meaningful sub-file boundaries in one or more storage nodes of said parallel computing system.

17. The data storage system of claim 16 , wherein said semantic information provides a description of data in said file.

18. The data storage system of claim 16 , wherein said sub-files are replicated based on said semantically meaningful boundaries.

19. The data storage system of claim 16 , wherein said processing device is further configured to process a query using said semantic information.

20. The data storage system of claim 16 , wherein said processing device is further configured to perform an analysis of said file using said semantic information.

21. The data storage system of claim 16 , wherein said one or more storage nodes reside on one or more tiers of a multi-tier storage system.

22. The data storage system of claim 16 , wherein said semantic information related to a given sub-file is stored with said corresponding sub-file.