Search and analytics engine with object store persistence

1 . A search system comprising:

at least one processor; and

at least one memory device storing executable instructions that cause the at least one processor to execute:

an indexing tier configured to:

generate a segment of an index structure based on data received from at least one first computing device;

store the segment in a local storage device associated with the indexing tier; and

transmit the segment to an external object store for storage thereon; and

a search tier configured to:

receive a search query from a second computing device;

retrieve at least a portion of the segment from the external object store; and

execute the search query using at least the portion of the segment.

2 . The search system of claim 1 , wherein the indexing tier is configured to:

in response to the segment being transmitted to the external object store, delete the segment from the local storage device.

3 . The search system of claim 1 , wherein the indexing tier is configured to:

determine whether a size of a plurality of segments stored in the local storage device satisfy a threshold level, the plurality of segments including the segment; and

in response to the size being determined as satisfying the threshold level, transmit a batched payload to the external object store, the batched payload including the plurality of segments.

4 . The search system of claim 1 , wherein the indexing tier is configured to:

determine whether a plurality of segments are stored in the local storage device for a period of time that satisfies a threshold time interval, the plurality of segments including the segment; and

in response to the period of time being determined as satisfying the threshold time interval, transmit a batched payload to the external object store, the batched payload including the plurality of segments.

5 . The search system of claim 1 , wherein the indexing tier is configured to:

in response to receiving the data from the at least one first computing device, generate a transaction record identifying a change to be made to the index structure;

in advance of the segment being generated, transmit the transaction record to the external object store for storage thereon;

in response to a restart event or a failure event, retrieve the transaction record from the external object store; and

generate the segment of the index structure using the transaction record.

6 . The search system of claim 5 , wherein the indexing tier is configured to:

initiate deletion of the transaction record from the external object store in response to a determination that the segment has been stored in the external object store and the segment includes the change identified in the transaction record.

7 . The search system of claim 1 , wherein the executable instructions include instructions that cause the at least one processor to execute:

a controller configured to:

monitor a first volume of the data received by the indexing tier and a second volume of search queries received by the search tier;

in response to the first volume satisfying a first threshold level, adjust a number of indexing nodes of the indexing tier; and

in response to the second volume satisfying a second threshold level, adjust a number of search nodes of the search tier.

8 . The search system of claim 1 , wherein the segment is a first segment, wherein the indexing tier is configured to:

identify a second segment in the external object store that is not referenced by an index manifest representing a searchable state of the index structure; and

initiate deletion of the second segment from the external object store.

9 . The search system of claim 8 , wherein the indexing tier is configured to:

identify a set of active searchers associated with the search tier; and

in response to a determination that the second segment is not accessed by the set of active searchers, initiate deletion of the second segment from the external object store.

10 . The search system of claim 1 , wherein the local storage device associated with the indexing tier is a first local storage device, wherein the search query is a first search query, wherein the search tier is configured to:

in response to the portion of the segment being retrieved from the external object store, store the portion of the segment in a second local storage device associated with the search tier;

receive a second search query;

retrieve the portion of the segment from the second local storage device; and

execute the second search query using at least the portion of the segment.

11 . The search system of claim 1 , wherein the local storage device associated with the indexing tier is a first local storage device, the segment is a first segment, and the search query is a first search query, wherein the search tier is configured to:

identify a pattern of segment access by analyzing a plurality of search queries;

identify a second segment of the index structure based on the pattern of segment access;

retrieve the second segment from the external object store; and

in advance of receiving a second search query that uses the second segment, store the second segment in a second local storage device associated with the search tier.

12 . A non-transitory computer-readable medium storing executable instructions that when executed by at least one processor cause the at least one processor to execute operations, the operations comprising:

generating a segment of an index structure based on data received from at least one first computing device;

storing the segment in a local storage device associated with an indexing tier;

transmitting the segment to an external object store for storage thereon;

receiving a search query from a second computing device;

retrieving at least a portion of the segment from the external object store; and

executing the search query using at least the portion of the segment.

13 . The non-transitory computer-readable medium of claim 12 , wherein the operations further comprise:

determining whether a size of a plurality of segments stored in the local storage device satisfy a threshold level, the plurality of segments including the segment;

determining whether the plurality of segments are stored in the local storage device for a period of time that satisfies a threshold time interval; and

in response to the size being determined as satisfying the threshold level and the period of time being determined as satisfying the threshold time interval, transmitting, a batched payload to the external object store, the batched payload including the plurality of segments.

14 . The non-transitory computer-readable medium of claim 12 , wherein the operations further comprise:

in response to receiving the data from the at least one first computing device, generating a transaction record identifying a change to be made to the index structure;

in advance of generating the segment, transmitting the transaction record to the external object store for storage thereon;

in response to a restart event or a failure event, retrieving the transaction record from the external object store; and

generating the segment of the index structure using the transaction record.

15 . The non-transitory computer-readable medium of claim 14 , wherein the operations further comprise:

initiating deletion of the transaction record from the external object store in response to a determination that the segment has been stored in the external object store and the segment includes the change identified in the transaction record.

16 . The non-transitory computer-readable medium of claim 12 , wherein the operations further comprise:

in response to the segment being transmitted to the external object store, deleting the segment from the local storage device.

17 . A method comprising:

generating a segment of an index structure based on data received from at least one first computing device;

storing the segment in a first local storage device associated with an indexing tier;

transmitting the segment to an external object store for storage thereon;

receiving a search query from a second computing device;

determining whether the segment is stored in a second local storage device associated with a search tier, the second local storage device configured to store one or more segments previously retrieved by the search tier;

in response to the segment being determined as not stored in the second local storage device, retrieving at least a portion of the segment from the external object store; and

executing the search query using at least the portion of the segment.

18 . The method of claim 17 , further comprising:

determining whether a size of a plurality of segments stored in the first local storage device satisfy a threshold level, the plurality of segments including the segment;

determining whether the plurality of segments are stored in the first local storage device for a period of time that satisfies a threshold time interval; and

in response to the size being determined as satisfying the threshold level and the period of time being determined as satisfying the threshold time interval, transmitting a batched payload to the external object store, the batched payload including the plurality of segments.

19 . The method of claim 17 , further comprising:

storing the segment in the second local storage device; and

deleting the segment from the second local storage device in response to an access frequency of the segment satisfying a threshold condition.

20 . The method of claim 17 , further comprising:

identifying a plurality of segments stored in the external object store that satisfy a merge threshold;

retrieving the plurality of segments from the external object store;

merging the plurality of segments into a consolidated segment;

transmitting the consolidated segment to the external object store; and

updating an index manifest to reference the consolidated segment instead of the plurality of segments.