Full text search capabilities integrated into distributed file systems— incrementally indexing files
View Patent ↗A hierarchical distributed search mechanism is integrated into a distributed file system. Traditional file system APIs (create, open, close, read, write, link, rename, delete, . . . ) and the over-the-wire protocols employed to project these APIs into remote client sites (CIFS, NFS, DDS, Appletalk) are extended to enable the dynamic creation of temporary directories containing links to objects identified by search engines (executing at sites “close” to “their” data) as meeting the search criteria specified by the first parameter of a search function call. The search function, derived from the standard file system API function create, is added to the file system API.
1. A method for incrementally indexing information contained in files within a distributed file system residing upon a virtual file server assembled by integrating a plurality of file servers comprising the steps of:
upon the commencement of a close operation on one of the files of the distributed file system after information contained in the file being closed has been changed:
parsing the information contained in the file; and
creating inverted index entries from the parsed information;
sorting the inverted index entries; and
merging the sorted inverted index entries into inverted file records of an inverted file that is associated with content of the distributed file system;
wherein parsing of the information contained in the file, creating the inverted index entries, sorting of the inverted index entries and merging the sorted inverted index entries into the inverted file records are completed before the close operation is completed.
2. The method of claim 1 further comprising the step of including in each inverted index entry created from the parsed information a global scope object id which uniquely specifies an object contained within the distributed file system.
3. The method of claim 1 wherein closing the file initiates the sequence of parsing of the information contained in the file, creating the inverted index entries, sorting of the inverted index entries and merging the sorted inverted index entries into the inverted file records before the close operation is completed.
4. The method of claim 1 wherein merging the sorted inverted index entries into the inverted file records further comprises the steps of:
differencing the sorted inverted index entries against inverted index entries for a prior version of the file thereby producing a difference file containing inverted index entries that indicate additions and deletions to the file; and
using the inverted index entries of the difference file to update the inverted file that is associated with the content of the distributed file system.
5. A method for incrementally indexing information contained in files within a file system residing upon a file server comprising the steps of:
upon the commencement of a close operation on one of the files of the distributed file system after information contained in the file being closed has been changed, and before the close operation is completed:
parsing the information contained in the file; and
creating inverted index entries from the parsed information;
sorting the inverted index entries;
merging the sorted inverted index entries into inverted file records of an inverted file that is associated with content of the distributed file system; and
after the close operation is completed, generating an indication that the file has been indexed and closed.
6. A method for incrementally indexing information contained in files within a file system residing upon a file server comprising the steps of:
upon the commencement of a close operation on one of the files of the distributed file system after information contained in the file being closed has been changed, and before the close operation is completed, initiating a sequence to:
parse the information contained in the file; and
create inverted index entries from the parsed information;
sort the inverted index entries;
merge the sorted inverted index entries into inverted file records of an inverted file that is associated with content of the distributed file system; and
after the close operation is completed, generating an indication that the file has been closed and is being indexed.