System and method of managing capacity of search index partitions
A search system can maintain a search index of metadata and text for objects in a repository, repositories or distributed across a network. The search index can be divided into partitions with a partition assigned a first capacity utilization threshold and a second capacity utilization threshold. If the capacity utilization of the partition is below the first threshold, the system can add, update and delete information in the partition. If the capacity utilization of the partition is above the first threshold, the system can update and delete information in the partition, but cannot add information for new objects to the partition. If the capacity utilization of the partition is above the second threshold, the system can enter a rebalancing mode in which it seeks to rebalance capacity utilization between partitions. The behavior of the system can change depending upon the size of a partition relative to its configurable thresholds.
1. A search system, comprising:
a computer processor;
a storage unit coupled to the computer processor;
an indexing engine executing on the computer processor; and
a search index coupled to the indexing engine,
the indexing engine configured for acting on indexing requests to update the search index,
the search index stored in the storage unit coupled to the computer processor,
the search index comprising a plurality of search index partitions,
each search index partition having a capacity utilization and characterized by an update capacity threshold and a rebalance capacity threshold,
wherein, as a result of the indexing engine acting on the indexing requests, the capacity utilization of the each search index partition changes and causes the indexing engine to change behavior,
the change in behavior of the indexing engine comprising the indexing engine changing its mode of operation depending upon what state the each search index partition is in,
wherein:
in a first state of a source partition of the plurality of search index partitions in the search index when the capacity utilization of the source partition does not exceed the update capacity threshold of the source partition, the indexing engine is able to update or delete an object in the source partition; and
in a second state of the source partition of the plurality of search index partitions in the search index when the capacity utilization of the source partition meets at least the rebalance capacity threshold of the source partition, the indexing engine is able to delete an object in the source partition or move an object in the source partition that is subject to an update to a target partition of the plurality of search index partitions in the search index.
2. The search system of claim 1 , wherein the rebalance capacity threshold is greater than the update capacity threshold.
3. The search system of claim 1 , wherein in the first state of the source partition, the indexing engine disallows an object not in the source partition to be added to the source partition.
4. The search system of claim 1 , wherein in the second state of the source partition, the indexing engine disallows an object not in the source partition to be added to the source partition.
5. The search system of claim 1 , wherein each partition is further characterized by an add capacity threshold, wherein in a third state of the source partition, the indexing engine allows an object not in the source partition to be added to the source partition when the capacity utilization of the source partition does not exceed the add capacity threshold.
6. The search system of claim 5 , wherein the target partition in the search index to which the object in the source partition that is subject to the update is moved to is in a first state in which the indexing engine allows objects not in the target partition to be added to the target partition.
7. The search system of claim 1 , wherein the rebalance capacity threshold includes an enter rebalance capacity threshold and an exit rebalance capacity threshold, the enter rebalance capacity threshold greater than the exit rebalance capacity threshold.
8. The search system of claim 7 , wherein the state of the source partition transitions from the first state to the second state when the capacity utilization of the source partition reaches the enter rebalance capacity threshold.
9. The search system of claim 7 , wherein the state of the source partition transitions from the second state to the first state when the utilization capacity reaches the exit rebalance capacity threshold.
10. The search system of claim 1 , wherein the search index comprises metadata for documents in a repository and text for documents in the repository, further comprising:
a search engine coupled to the search index and configured to search for documents in the repository based on the metadata and/or the text in the search index.
11. The search system of claim 1 , wherein the indexing engine is configured to receive indexing requests to add, modify, or delete objects in the search index.
12. The search system of claim 11 , further comprising:
an index controller, wherein the search index comprises metadata for documents in a repository and text for documents in the repository, the index controller configured to determine whether an indexing request is associated with metadata, text, or a combination thereof.
13. A method, comprising:
responsive to indexing requests, updating a search index,
the updating performed by an indexing engine at a computer having a processor, a storage unit coupled to the processor, the indexing engine executing on the processor, and a search index coupled to the indexing engine,
the search index stored in the storage unit coupled to the processor and comprising a plurality of search index partitions,
each search index partition having a capacity utilization and characterized by an update capacity threshold and a rebalance capacity threshold,
wherein, as a result of the indexing engine acting on the indexing requests, the capacity utilization of the each search index partition changes and causes the indexing engine to change behavior,
the change in behavior of the indexing engine comprising:
the indexing engine changing its mode of operation depending upon what state the each search index partition is in,
wherein:
in a first state of a source partition of the plurality of search index partitions in the search index when the capacity utilization of the source partition does not exceed the update capacity threshold of the source partition, the indexing engine is able to update or delete an object in the source partition; and
in a second state of the source partition of the plurality of search index partitions in the search index, the indexing engine is able to delete an object in the source partition or move an object in the source partition that is subject to an update to a target partition of the plurality of search index partitions in the search index.
14. The method according to claim 13 , wherein the rebalance capacity threshold is greater than the update capacity threshold.
15. The method according to claim 13 , wherein in the first state of the source partition, the indexing engine disallows an object not in the source partition to be added to the source partition.
16. The method according to claim 13 , wherein in the second state of the source partition, the indexing engine disallows an object not in the source partition to be added to the source partition.
17. The method according to claim 13 , wherein each partition is further characterized by an add capacity threshold, wherein in a third state of the source partition, the indexing engine allows an object not in the source partition to be added to the source partition when the capacity utilization of the source partition does not exceed the add capacity threshold.
18. The method according to claim 17 , wherein the target partition in the search index to which the object in the source partition that is subject to the update is moved to is in a first state in which the indexing engine allows objects not in the target partition to be added to the target partition.
19. The method according to claim 13 , wherein the rebalance capacity threshold includes an enter rebalance capacity threshold and an exit rebalance capacity threshold, the enter rebalance capacity threshold greater than the exit rebalance capacity threshold.
20. The method according to claim 19 , wherein the state of the source partition transitions from the first state to the second state when the capacity utilization of the source partition reaches the enter rebalance capacity threshold.
21. The method according to claim 19 , wherein the state of the source partition transitions from the second state to the first state when the utilization capacity reaches the exit rebalance capacity threshold.
22. The method according to claim 13 , wherein the search index comprises metadata for documents in a repository and text for documents in the repository, further comprising:
a search engine coupled to the search index and configured to search for documents in the repository based on the metadata and/or the text in the search index.
23. The method according to claim 13 , wherein the indexing engine is configured to receive indexing requests to add, modify, or delete objects in the search index.
24. The method according to claim 23 , wherein the search index comprises metadata for documents in a repository and text for documents in the repository, further comprising determining whether an indexing request is associated with metadata, text, or a combination thereof.