Producing search results by aggregating messages from multiple search peers
View Patent ↗Asynchronous processing of messages that are received from multiple servers is disclosed. An example method may include transmitting, by a computer system, a search request to a plurality of search peers of a data aggregation and analysis system. The method may further include receiving a plurality of sub-application layer protocol packets from the plurality of search peers. The method may further include parsing, by a first processing thread of the computer system, one or more sub-application layer protocol packets of the plurality of sub-application layer protocol packets, to produce an application layer message representing a partial response to the search request. The method may further include processing, by a second processing thread of the computer system, the application layer message to produce a memory data structure representing an aggregated response to the search request.
1. A method, comprising:
transmitting, by a computer system, a search request to a plurality of search peers of a data aggregation and analysis system;
receiving a plurality of sub-application layer protocol packets from the plurality of search peers;
parsing, by a first processing thread of the computer system, one or more sub-application layer protocol packets of the plurality of sub-application layer protocol packets, to produce an application layer message representing a partial response to the search request; and
processing, by a second processing thread of the computer system, the application layer message to produce a memory data structure representing an aggregated response to the search request.
2. The method of claim 1 , wherein processing the application layer message further comprises: splitting a payload of the application layer message into two or more parts based on at least one of: a defined set of bit position or a defined separator.
3. The method of claim 1 , wherein processing the application layer message further comprises: encoding a payload of the application layer message according to a defined encoding rule.
4. The method of claim 1 , wherein processing the application layer message further comprises allocating the second processing thread from a thread pool.
5. The method of claim 1 , wherein receiving the plurality of sub-application layer protocol packets is performed by the first processing thread.
6. The method of claim 1 , wherein receiving the plurality of sub-application layer protocol packets is performed by a third processing thread asynchronously with the respect to at least one of: the first processing thread or the second processing thread.
7. The method of claim 1 , wherein parsing one or more sub-application layer protocol packets is performed in an order of receiving the sub-application layer protocol packets over a plurality of transport layer connections.
8. The method of claim 1 , wherein the sub-application layer protocol is provided by one of: a transport layer protocol, a session layer protocol, or a presentation layer protocol.
9. The method of claim 1 , wherein the sub-application layer protocol is provided by HTTP protocol.
10. The method of claim 1 , wherein receiving the plurality of sub-application layer protocol packets is performed over a plurality of transport layer connections.
11. The method of claim 1 , further comprising:
writing the application layer message to a message queue; and
responsive to determining that a total size of messages in the message queue exceeds a certain threshold, causing the first processing thread to suspend receiving sub-application layer protocol packets.
12. The method of claim 11 , further comprising:
responsive to determining that a total size of messages in the message queue falls below a certain threshold, notifying the first processing thread to resume receiving sub-application layer protocol packets.
13. The method of claim 1 , wherein each search peer of the plurality of search peers performs map operations of a map-reduce search, to return partial results based on a subset of source data.
14. The method of claim 1 , wherein the application layer message comprises one or more events derived from time-series source data.
15. The method of claim 1 , further comprising:
inserting a timestamp into the application layer message.
16. The method of claim 1 , wherein the method is performed by a search head that performs map operations of a map-reduce search.
17. A computer system, comprising:
a memory; and
one or more processing devices, coupled to the memory, to:
transmit a search request to a plurality of search peers of a data aggregation and analysis system;
receive a plurality of sub-application layer protocol packets from the plurality of search peers;
parse, by a first processing thread of the computer system, one or more sub-application layer protocol packets of the plurality of sub-application layer protocol packets, to produce an application layer message representing a partial response to the search request; and
process, by a second processing thread of the computer system, the application layer message to produce a memory data structure representing an aggregated response to the search request.
18. The system of claim 17 , wherein the sub-application layer protocol is provided by one of: a transport layer protocol, a session layer protocol, or a presentation layer protocol.
19. The system of claim 17 , wherein the sub-application layer protocol is provided by HTTP protocol.
20. The system of claim 17 , wherein the one or more processing devices are further to:
write the application layer message to a message queue; and
responsive to determining that a total size of messages in the message queue exceeds a certain threshold, cause the first processing thread to suspend receiving sub-application layer protocol packets.
21. The system of claim 17 , wherein each search peer of the plurality of search peers performs map operations of a map-reduce search, to return partial results based on a subset of source data.
22. The system of claim 17 , wherein the application layer message comprises one or more events derived from time-series source data.
23. The system of claim 17 , wherein the one or more processing devices are further to:
insert a timestamp into the application layer message.
24. The system of claim 17 , wherein the system implements a search head that performs map operations of a map-reduce search.
25. A computer-readable non-transitory storage medium comprising executable instructions that, when executed by a computer system, cause the computer system to:
transmit a search request to a plurality of search peers of a data aggregation and analysis system;
receive a plurality of sub-application layer protocol packets from the plurality of search peers;
parse, by a first processing thread of the computer system, one or more sub-application layer protocol packets of the plurality of sub-application layer protocol packets, to produce an application layer message representing a partial response to the search request; and
process, by a second processing thread of the computer system, the application layer message to produce a memory data structure representing an aggregated response to the search request.