Storing files in a parallel computing system based on user-specified parser function

1. A method for storing a plurality of files created by a distributed source application and sent by said distributed source application to a parallel computing system for storage, wherein said plurality of sent files comprise one or more of a plurality of complete files and a plurality of sub-files, said method comprising the steps of:

obtaining a parser with said plurality of sent files from said distributed source application for processing said plurality of sent files prior to storage;

applying said parser to said plurality of sent files prior to storage to extract metadata from said plurality of sent files;

obtaining said metadata extracted by said parser from one or more of said plurality of sent files; and

storing said obtained metadata with one or more of said plurality of sent files in a file system on one or more storage nodes within said parallel computing system based on said processing by said parser.

2. The method of claim 1 , wherein said storing step further comprises the step of storing only sent files that satisfy one or more semantic requirements of said parser.

3. The method of claim 1 , wherein said extracted metadata comprises one or more key-value pairs from data in one or more of said plurality of sent files.

4. The method of claim 1 , wherein said extracted metadata comprises data-aware semantic information from one or more of said plurality of sent files.

5. The method of claim 1 , further comprising the step of processing a query using said extracted metadata.

6. The method of claim 1 , further comprising the step of searching said extracted metadata to identify one or more sent files associated with said metadata that satisfy one or more semantic requirements.

7. The method of claim 1 , wherein said parser identifies one or more of said storage nodes where said plurality of sent files should be stored.

8. The method of claim 1 , wherein said parser identifies one or more tiers of a multi-tier storage system for storing one or more of said plurality of files.

9. A computer program product comprising a non-transitory tangible processor-readable recordable storage medium having encoded therein executable code of one or more software programs, wherein the one or more software programs when executed by a processor of a processing device implement the steps of the method of claim 1 .

10. An apparatus for storing a plurality of files created by a distributed source application and sent by said distributed source application to a parallel computing system for storage, wherein said plurality of sent files comprise one or more of a plurality of complete files and a plurality of sub-files, comprising:

a memory; and

at least one hardware device operatively coupled to the memory and configured to:

obtain a parser with said plurality of sent files from said distributed source application for processing said plurality of sent files prior to storage;

apply said parser to said plurality of sent files prior to storage to extract metadata from said plurality of sent files;

obtain said metadata extracted by said parser from one or more of said plurality of sent files; and

store said obtained metadata with one or more of said plurality of sent files in a file system on one or more storage nodes within said parallel computing system based on said processing by said parser.

11. The apparatus of claim 10 , wherein said at least one hardware device is further configured to store only sent files that satisfy one or more semantic requirements of said parser.

12. The apparatus of claim 10 , wherein said extracted metadata comprises data-aware semantic information from one or more of said plurality of sent files.

13. The apparatus of claim 10 , wherein said at least one hardware device is further configured to process a query using said extracted metadata.

14. The apparatus of claim 10 , wherein said parser identifies one or more of said storage nodes where said plurality of sent files should be stored.

15. The apparatus of claim 10 , wherein said parser identifies one or more tiers of a multi-tier storage system for storing one or more of said plurality of sent files.

16. The apparatus of claim 10 , wherein said parser executes on one or more nodes of a multi-tier storage system.

17. A data storage system for storing a plurality of files created by a distributed source application and sent by said distributed source application to a parallel computing system for storage, wherein said plurality of sent files comprise one or more of a plurality of complete files and a plurality of sub-files, comprising:

a hardware processing unit for processing said plurality of sent files prior to storage to extract metadata from said plurality of sent files using a parser obtained with said plurality of sent files from said distributed source application and for obtaining metadata extracted by said parser from one or more of said plurality of sent files; and

a storage medium for storing said obtained metadata with one or more of said plurality of sent files in a file system on one or more storage nodes within said parallel computing system based on said processing by said parser.

18. The data storage system of claim 17 , wherein only said sent files that satisfy one or more semantic requirements of said parser are stored.

19. The data storage system of claim 17 , wherein said extracted metadata comprises data-aware semantic information from one or more of said plurality of sent files.

20. The data storage system of claim 17 , wherein said processing unit is further configured to process a query using said extracted metadata.