PROCESSING DATA USING NODES IN A SCALABLE ENVIRONMENT
Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives raw machine data at an indexing system, and stores at least a portion of the raw machine data in buckets using containerized indexing nodes instantiated in a containerized environment. The data intake and query system stores the buckets in a shared storage system.
1 . (canceled)
2 . A method comprising:
receiving a query from a client device, the query including criteria for identifying search results from a set of data comprising messages published to a publish-subscribe messaging system; and
defining, by one or more computing devices, a query processing scheme for obtaining and processing the set of data, the query processing scheme including a dynamic allocation of partitions to process the messages to identify the search results, wherein defining the query processing scheme includes:
determining a number of messages queues on the publish-subscribe messaging system in which the messages are placed;
dynamically allocating individual queues of the message queues to individual partitions, the individual partitions to repeatedly collect messages from a corresponding allocated individual queue over a plurality of time windows of a specified duration, and at the end of each time window, cause the messages received during a time window to be processed by a processing partition; and
executing the query based on the query processing scheme.
3 . The method of claim 2 , wherein each time window is a predetermined length of time.
4 . The method of claim 2 , wherein dynamically allocating individual queues of the message queues to individual partitions comprises evenly allocating the individual queues among one or more worker node computing devices.
5 . The method of claim 2 , wherein a number of the individual partitions is equal to a number of message queues, and wherein dynamically allocating individual queues to individual partitions comprises allocating a single queue to each individual partition.
6 . The method of claim 2 , wherein a number of the individual partitions is less than a number of the message queues, and wherein dynamically allocating individual queues to individual processors comprises allocating multiple queues to each of the individual partitions.
7 . The method of claim 2 , wherein at least two of the individual partitions are implemented within a common worker node computing device.
8 . The method of claim 2 , wherein each of the individual partitions is implemented within a different worker node computing device.
9 . The method of claim 2 , wherein the set of data corresponds to a topic of the publish-subscribe messaging system.
10 . The method of claim 2 , wherein the set of data corresponds to a topic of the publish-subscribe messaging system, and wherein determining the number of messages queues on the publish-subscribe messaging system in which the messages are placed comprises querying the publish-subscribe messaging system for identifying information of one or more message queues corresponding to the topic.
11 . The method of claim 2 further comprising identifying, from the criteria, that the query is directed to data on the publish-subscribe messaging system based at least in part on the criteria of the query.
12 . The method of claim 2 further comprising identifying, from the criteria, that the query is directed to data on the publish-subscribe messaging system based at least in part on the criteria of the query.
13 . The method of claim 2 further comprising transmitting search results generated from a collection of messages to a data destination and, subsequent to transmitting the search results search results generated from the collection of messages, transmitting an acknowledgement of each message within the collection of messages to the publish-subscribe messaging system.
14 . The method of claim 2 further comprising transmitting search results generated from a collection of messages to a data destination and, subsequent to transmitting the search results search results generated from the collection of messages, transmitting an acknowledgement of each message within the collection of messages to the publish-subscribe messaging system, wherein transmitting the acknowledgement of each message within the collection of messages to the publish-subscribe messaging system comprises instructing individual partitions that collected each message within the collection of messages to transmit acknowledgements for each message to the publish-subscribe messaging system.
15 . The method of claim 2 , wherein the messages are associated with a topic on the publish-subscribe messaging system, wherein the method further comprising transmitting the search to the publish-subscribe messaging system as messages for a second topic.
16 . The method of claim 2 , wherein the publish-subscribe messaging system queues the messages for delivery to indexers that index data within the messages.
17 . The method of claim 2 , wherein the publish-subscribe messaging system queues the messages for delivery to indexers that index data within the messages, and wherein the indexers place indexed data generated from the messages within a common storage also accessible to the processors.
18 . A system comprising:
a data store including computer-executable instructions; and
one or more processors configured to execute the computer-executable instructions, wherein execution of the computer-executable instructions causes the system to:
receive a query from a client device, the query including criteria for identifying search results from a set of data comprising messages published to a publish-subscribe messaging system; and
define, by one or more computing devices, a query processing scheme for obtaining and processing the set of data, the query processing scheme including a dynamic allocation of partitions to process the messages to identify the search results, wherein defining the query processing scheme includes:
determining a number of messages queues on the publish-subscribe messaging system in which the messages are placed;
dynamically allocating individual queues of the message queues to individual partitions, the individual partitions to repeatedly collect messages from a corresponding allocated individual queue over a plurality of time windows of a specified duration, and at the end of each time window, cause the messages received during a time window to be processed by a processing partition; and
executing the query based on the query processing scheme.
19 . The system of claim 18 , wherein execution of the computer-executable instructions causes the system to transmit search results generated from a collection of messages to a data destination, and subsequent to transmitting the search results generated from the collection of messages, transmit an acknowledgement of each message within the collection of messages to the publish-subscribe messaging system.
20 . Non-transitory computer-readable media including computer-executable instructions that, when executed by a computing system, cause the computing system to:
receive a query from a client device, the query including criteria for identifying search results from a set of data comprising messages published to a publish-subscribe messaging system; and
define, by one or more computing devices, a query processing scheme for obtaining and processing the set of data, the query processing scheme including a dynamic allocation of partitions to process the messages to identify the search results, wherein defining the query processing scheme includes:
determining a number of messages queues on the publish-subscribe messaging system in which the messages are placed;
dynamically allocating individual queues of the message queues to individual partitions, the individual partitions to repeatedly collect messages from a corresponding allocated individual queue over a plurality of time windows of a specified duration, and at the end of each time window, cause the messages received during a time window to be processed by a processing partition; and
executing the query based on the query processing scheme.
21 . The non-transitory computer-readable media of claim 20 , wherein execution of the computer-executable instructions causes the computing system to transmit search results generated from a collection of messages to a data destination, and subsequent to transmitting the search results generated from the collection of messages, transmit an acknowledgement of each message within the collection of messages to the publish-subscribe messaging system.