Timestamp-based processing of messages using message queues
Systems and methods for priority-based processing of messages received from multiple servers. An example method comprises: receiving a plurality of network packets from one or more servers; processing the plurality of network packets to produce a first message associated with a first timestamp and a second message associated with a second timestamp; writing the first message to a first message queue of a plurality of message queues; writing the second message to a second message queue of the plurality of message queues; and retrieving, from the plurality of message queues, the first message and the second message in an order of their respective associated timestamps.
1. A method, comprising:
receiving a plurality of network packets from one or more servers;
processing the plurality of network packets to produce a first message associated with a first timestamp and a second message associated with a second timestamp;
writing the first message to a first message queue of a plurality of message queues;
writing the second message to a second message queue of the plurality of message queues; and
retrieving, from the plurality of message queues, the first message and the second message in an order of their respective associated timestamps.
2. The method of claim 1 , further comprising:
producing a first memory data structure based on the first message and a second memory data structure based on the second message; and
placing the first memory data structure and the second memory data structure into a result queue.
3. The method of claim 1 , wherein receiving the network packets is performed over a plurality of transport layer connections.
4. The method of claim 1 , wherein receiving the network packets comprises processing the network packets in an order of receiving the network packets over a plurality of transport layer connections.
5. The method of claim 1 , wherein the network packets are associated with a sub-application layer protocol provided by one of: a transport layer protocol, a session layer protocol, or a presentation layer protocol.
6. The method of claim 1 , wherein the network packets are associated with a sub-application layer protocol provided by HTTP protocol.
7. The method of claim 1 , wherein receiving the plurality of network packets is performed responsive to transmitting a request to the servers.
8. The method of claim 1 , further comprising:
responsive to determining that a total size of messages in the first message queue exceeds a certain threshold, causing a first processing thread to suspend receiving packets.
9. The method of claim 1 , further comprising:
responsive to determining that a total size of messages in the first message queue falls below a certain threshold, causing a first processing thread to resume receiving packets.
10. The method of claim 1 , wherein each server represents a search peer of a data aggregation and analysis system.
11. The method of claim 1 , wherein each server performs map operations of a map-reduce search, to return partial results based on a subset of source data.
12. The method of claim 1 , wherein the message comprises one or more search results.
13. The method of claim 1 , wherein the message comprises one or more events derived from time-series source data.
14. The method of claim 1 , wherein the method is performed by a search head that performs map operations of a map-reduce search.
15. A computer system, comprising:
a memory; and
one or more processing devices, coupled to the memory, to:
receive a plurality of network packets from one or more servers;
process the plurality of network packets to produce a first message associated with a first timestamp and a second message associated with a second timestamp;
write the first message to a first message queue of a plurality of message queues;
write the second message to a second message queue of the plurality of message queues; and
retrieve, from the plurality of message queues, the first message and the second message in an order of their respective associated timestamps.
16. The computer system of claim 15 , wherein the processing devices are further to:
produce a first memory data structure based on the first message and a second memory data structure based on the second message; and
place the first memory data structure and the second memory data structure into a result queue.
17. The computer system of claim 15 , wherein receiving the network packets comprises processing the network packets in an order of receiving the network packets over a plurality of transport layer connections.
18. The computer system of claim 15 , wherein the network packets are associated with a sub-application layer protocol provided by one of: a transport layer protocol, a session layer protocol, or a presentation layer protocol.
19. The computer system of claim 15 , wherein the network packets are associated with a sub-application layer protocol provided by HTTP protocol.
20. A computer-readable non-transitory storage medium comprising executable instructions that, when executed by a computer system, cause the computer system to perform operations comprising:
receiving a plurality of network packets from one or more servers;
processing the plurality of network packets to produce a first message associated with a first timestamp and a second message associated with a second timestamp;
writing the first message to a first message queue of a plurality of message queues;
writing the second message to a second message queue of the plurality of message queues; and
retrieving, from the plurality of message queues, the first message and the second message in an order of their respective associated timestamps.