Systems and methods for controller-worker architecture for searching a storage system
A controller is provided to manage searching a storage system. A search query including a search term is received. The controller generates a set of tasks associated with conducting a search of the storage system for any objects in the storage system having the search term. These tasks are pushed onto a message queue. A set of workers are allocated to retrieve and process the tasks. A worker processes a task in parallel with another worker processing another task.
1. A method comprising:
providing a controller to manage searching objects of a storage system;
receiving a search query comprising a search term;
splitting an object to be searched into a plurality of pieces;
generating, by the controller, a plurality of tasks associated with conducting a search of the storage system for any objects in the storage system having the search term, the plurality of tasks comprising first and second tasks, the first task specifying a first piece of the object to be searched for the search term, and the second task specifying a second piece of the object to be searched for the same search term;
pushing the tasks onto a message queue; and
allocating a plurality of workers to retrieve and process the plurality of tasks on the message queue, wherein a worker processes a task in parallel with another worker processing another task, the first task thereby being processed by a first worker while the second task is processed by a second worker, wherein the object comprises an object recipe referencing first and second slice recipes, each slice recipe comprising a sequence of fingerprints corresponding to segments that make up the object,
wherein the first piece of the object corresponds to a first slice recipe and the second piece of the object corresponds to a second slice recipe, and
wherein the search comprises a first worker having been assigned the first task searching a first subset of segments making up the object according to the first slice recipe, and
a second worker having been assigned the second task searching a second subset of segments making up the object according to the second slice recipe.
2. The method of claim 1 wherein the objects comprise logical objects having references to data units in the storage system and the method further comprises:
receiving filter criteria with the search query; and
before generating the plurality of tasks, filtering out logical objects that do not meet the filter criteria.
3. The method of claim 1 wherein the objects comprise logical objects having references to data units in the storage system, and the task of the worker comprises reading a logical object and data units referenced by the logical object to determine whether the search term can be found in the data units referenced by the logical object.
4. The method of claim 1 further comprising:
dynamically adjusting a number of the plurality of workers based on a desired time to complete the search.
5. The method of claim 1 further comprising:
receiving status updates from the workers about the tasks being processed;
logging the status updates in persistent storage;
after an interruption of the search, restarting the search by accessing the persistent storage to determine which tasks were completed before the interruption; and
pushing, onto the message queue, tasks that were not completed before the interruption and not pushing tasks onto the message queue that were completed before the interruption.
6. The method of claim 1 wherein the search further comprises:
upon the first worker finding a first partial match of the search term in a segment referenced by a last fingerprint in the first slice recipe, allowing the first worker to search a segment referenced by an initial fingerprint in the second slice recipe to determine whether a second partial match of the search term can be found that combines with the first partial match to form a complete match of the search term.
7. A system comprising: a processor, and memory storing one or more sequences of instructions which, when executed by the processor, cause the processor to carry out the steps of:
providing a controller to manage searching objects of a storage system;
receiving a search query comprising a search term;
splitting an object to be searched into a plurality of pieces;
generating, by the controller, a plurality of tasks associated with conducting a search of the storage system for any objects in the storage system having the search term, the plurality of tasks comprising first and second tasks, the first task specifying a first piece of the object to be searched for the search term, and the second task specifying a second piece of the object to be searched for the same search term;
pushing the tasks onto a message queue; and
allocating a plurality of workers to retrieve and process the plurality of tasks on the message queue, wherein a worker processes a task in parallel with another worker processing another task, the first task thereby being processed by a first worker while the second task is processed by a second worker, wherein the object comprises an object recipe referencing first and second slice recipes, each slice recipe comprising a sequence of fingerprints corresponding to segments that make up the object,
wherein the first piece of the object corresponds to a first slice recipe and the second piece of the object corresponds to a second slice recipe, and
wherein the search comprises a first worker having been assigned the first task searching a first subset of segments making up the object according to the first slice recipe, and
a second worker having been assigned the second task searching a second subset of segments making up the object according to the second slice recipe.
8. The system of claim 7 wherein the objects comprise logical objects having references to data units in the storage system and the processor further carries out the steps of:
applying filter settings received with the search query to the metadata to obtain a subset of fingerprint listings to be read, the first fingerprint listing that is read thereby being a fingerprint listing of the first object that has meet the filter settings.
9. The system of claim 7 wherein the objects comprise logical objects having references to data units in the storage system, and the task of the worker comprises reading a logical object and data units referenced by the logical object to determine whether the search term can be found in the data units referenced by the logical object.
10. The system of claim 7 wherein the processor further carries out the steps of:
dynamically adjusting a number of the plurality of workers based on a desired time to complete the search.
11. The system of claim 7 wherein the processor further carries out the steps of:
receiving status updates from the workers about the tasks being processed;
logging the status updates in persistent storage;
after an interruption of the search, restarting the search by accessing the persistent storage to determine which tasks were completed before the interruption; and
pushing, onto the message queue, tasks that were not completed before the interruption and not pushing tasks onto the message queue that were completed before the interruption.
12. The system of claim 7 wherein the search further comprises:
upon the first worker finding a first partial match of the search term in a segment referenced by a last fingerprint in the first slice recipe, allowing the first worker to search a segment referenced by an initial fingerprint in the second slice recipe to determine whether a second partial match of the search term can be found that combines with the first partial match to form a complete match of the search term.
13. A computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein, the computer-readable program code adapted to be executed by one or more processors to implement a method comprising:
providing a controller to manage searching objects of a storage system;
receiving a search query comprising a search term;
splitting an object to be searched into a plurality of pieces;
generating, by the controller, a plurality of tasks associated with conducting a search of the storage system for any objects in the storage system having the search term, the plurality of tasks comprising first and second tasks, the first task specifying a first piece of the object to be searched for the search term, and the second task specifying a second piece of the object to be searched for the same search term;
pushing the tasks onto a message queue; and
allocating a plurality of workers to retrieve and process the plurality of tasks on the message queue, wherein a worker processes a task in parallel with another worker processing another task, the first task thereby being processed by a first worker while the second task is processed by a second worker, wherein the object comprises an object recipe referencing first and second slice recipes, each slice recipe comprising a sequence of fingerprints corresponding to segments that make up the object,
wherein the first piece of the object corresponds to a first slice recipe and the second piece of the object corresponds to a second slice recipe, and
wherein the search comprises a first worker having been assigned the first task searching a first subset of segments making up the object according to the first slice recipe, and
a second worker having been assigned the second task searching a second subset of segments making up the object according to the second slice recipe.
14. The computer program product of claim 13 wherein the objects comprise logical objects having references to data units in the storage system and the method further comprises:
receiving filter criteria with the search query; and
before generating the plurality of tasks, filtering out logical objects that do not meet the filter criteria.
15. The computer program product of claim 13 wherein the objects comprise logical objects having references to data units in the storage system, and the task of the worker comprises reading a logical object and data units referenced by the logical object to determine whether the search term can be found in the data units referenced by the logical object.
16. The computer program product of claim 13 wherein the method further comprises:
dynamically adjusting a number of the plurality of workers based on a desired time to complete the search.
17. The computer program product of claim 13 wherein the method further comprises:
receiving status updates from the workers about the tasks being processed;
logging the status updates in persistent storage;
after an interruption of the search, restarting the search by accessing the persistent storage to determine which tasks were completed before the interruption; and
pushing, onto the message queue, tasks that were not completed before the interruption and not pushing tasks onto the message queue that were completed before the interruption.
18. The computer program product of claim 13 wherein the search further comprises:
upon the first worker finding a first partial match of the search term in a segment referenced by a last fingerprint in the first slice recipe, allowing the first worker to search a segment referenced by an initial fingerprint in the second slice recipe to determine whether a second partial match of the search term can be found that combines with the first partial match to form a complete match of the search term.