Integrated search for shared storage using index throttling to maintain quality of service
A method of searching shared files stored on a shared storage includes providing a search functionality in the shared storage having a set of search indexes to perform a search based on a search term; providing an event stream of update events associated with the shared files in the shared storage; processing the event stream to generate a processed event stream; and responding to update events in the processed event stream by indexing files referenced in the update events in the processed event stream where the indexing is performed to maintain a predetermined level of quality of service at the shared storage.
1. A method of searching shared files stored on a shared storage comprising:
providing a search functionality in the shared storage, the search functionality having a set of search indexes to perform a search based on a search term;
generating an event stream by the shared storage of update events associated with the shared files in the shared storage, wherein the update events comprise file update events, which include file modification and file creation, and file deletion events;
providing, at the shared storage, the event stream of update events to the search functionality;
processing, at the shared storage, the event stream by performing event filtering, event compression and de-duplication on the stream of update events to generate a processed event stream;
indexing, at the shared storage, files referenced in the update events to update the set of search indexes; and
responding to update events in the processed event stream by throttling the indexing of files referenced in the update events in the processed event stream to maintain a predetermined level of quality of service at the shared storage.
2. The method of claim 1 , wherein throttling the indexing of files referenced in the update events in the processed event stream to maintain a predetermined level of quality of service at the shared storage comprises:
throttling the indexing of files referenced in the update events to achieve the predetermined level of quality of service for shared storage requests including at least read and write requests.
3. The method of claim 1 , wherein throttling the indexing of files referenced in the update events in the processed event stream to maintain a predetermined level of quality of service at the shared storage comprises:
reducing the indexing of files referenced in the update events to achieve the predetermined level of quality of service at the shared storage.
4. The method of claim 3 , wherein reducing the indexing of files referenced in the update events to achieve the predetermined level of quality of service at the shared storage comprises:
filtering the update events in the processed event stream based on file type; and
discarding update events in the processed event stream for files having a first file type.
5. The method of claim 4 , wherein discarding update events in the processed event stream for files having a first file type comprises:
discarding update events in the processed event stream for files having a file type indicative of a temporary file.
6. The method of claim 3 , wherein reducing the indexing of files referenced in the update events to achieve the predetermined level of quality of service at the shared storage comprises:
filtering the update events in the processed event stream based on a business value of the files referenced in the update events; and
detaining update events in an event queue for files having a low business value,
wherein the business value of files is determined from one or more factors including a file location, a file format, a file size and/or a file name.
7. The method of claim 1 , wherein:
generating, at the shared storage, an event stream of update events associated with the shared files in the shared storage comprises generating, at the shared storage, one or more event streams of update events associated with the shared files; and
processing, at the shared storage, the event stream to generate a processed event stream comprises processing, at the shared storage, the one or more event streams using a plurality of event queues to generate one or more processed event streams, each processed event stream being assigned a priority level.
8. The method of claim 7 , wherein throttling the indexing of files referenced in the update events in the processed event stream to maintain a predetermined level of quality of service at the shared storage comprises:
throttling the indexing of files referenced in the update events based on the priority level of the processed event streams; and
detaining in an event queue the indexing of files referenced in the update events in a processed event stream having a low priority level.
9. The method of claim 1 , wherein processing, at the shared storage, the event stream to generate a processed event stream comprises:
processing, at the shared storage, the event stream using an event queue;
identifying each update event using a file identification uniquely identifying a file in the shared storage;
receiving a first update event having a first file identification; and
generating the processed event stream by removing a duplicate update event having the same file identification as the first update event.
10. The method of claim 9 , wherein generating the processed event stream by removing a duplicate update event having the same file identification as the first update event comprises:
locating an entry in the event queue associated with the duplicate update event; and
overwriting the entry in the event queue using the first update event.
11. The method of claim 9 , wherein generating the processed event stream by removing a duplicate update event having the same file identification as the first update event comprises:
locating an entry in the event queue associated with the duplicate update event;
removing the entry in the event queue associated with the duplicate update event; and
placing the first update event at the end of the event queue.
12. A method of searching shared files stored on a shared storage comprising:
providing a search functionality in the shared storage, the search functionality having a set of search indexes to perform a search based on a search term;
generating an event stream by the shared storage of update events associated with the shared files in the shared storage, wherein the update events comprise file update events, which include file modification and file creation, and file deletion events;
providing, at the shared storage, the event stream of update events to the search functionality;
processing, at the shared storage, the event stream by performing event filtering, event compression and de-duplication on the stream of update events to generate a processed event stream; indexing, at the shared storage, files referenced in the update events to update the set of search indexes; and
responding to update events in the processed event stream by indexing files referenced in the update events in the processed event stream, the indexing being performed to maintain a predetermined level of quality of service at the shared storage,
wherein processing, at the shared storage, the event stream to generate a processed event stream comprises:
processing the event stream using an event queue;
identifying each update event using a file identification uniquely identifying a file in the shared storage;
receiving a first update event having a first file identification; and
generating the processed event stream by removing a duplicate update event having the same file identification as the first update event.
13. A system for searching shared files stored on a shared storage comprising:
a processor; and
a memory coupled with the processor, wherein the memory is configured to provide the processor with instructions which when executed cause the processor to:
provide a search functionality in the shared storage, the search functionality having a set of search indexes to perform a search based on a search term;
generate an event stream by the shared storage of update events associated with the shared files in the shared storage, wherein the update events comprise file update events, which include file modification and file creation, and file deletion events;
provide, at the shared storage, the event stream of update events to the search functionality;
process, at the shared storage, the event stream by performing event filtering, event compression and de-duplication on the stream of update events to generate a processed event stream;
index, at the shared storage, files referenced in the update events to update the set of search indexes; and
respond to update events in the processed event stream by throttling the indexing of files referenced in the update events in the processed event stream to maintain a predetermined level of quality of service at the shared storage.
14. The system recited in claim 13 , wherein the memory is further configured to provide the processor with instructions which when executed cause the processor to:
throttle the indexing of files referenced in the update events to achieve the predetermined level of quality of service for shared storage requests including at least read and write requests.
15. The system recited in claim 13 , wherein the memory is further configured to provide the processor with instructions which when executed cause the processor to:
reduce the indexing of files referenced in the update events to achieve the predetermined level of quality of service at the shared storage.
16. The system recited in claim 15 , wherein the memory is further configured to provide the processor with instructions which when executed cause the processor to:
filter the update events in the processed event stream based on file type; and
discard update events in the processed event stream for files having a first file type.
17. The system recited in claim 15 , wherein the memory is further configured to provide the processor with instructions which when executed cause the processor to:
filter the update events in the processed event stream based on a business value of the files referenced in the update events; and
detain update events in an event queue for files having a low business value,
wherein the business value of files is determined from one or more factors including a file location, a file format, a file size and/or a file name.
18. The system recited in claim 13 , wherein the memory is further configured to provide the processor with instructions which when executed cause the processor to:
generate, at the shared storage, one or more event streams of update events associated with the shared files; and
process, at the shared storage, the one or more event streams using a plurality of event queues to generate one or more processed event streams, each processed event stream being assigned a priority level.