Utilizing distributed tasks to retrieve supplemental job information for performing a job
View Patent ↗A method to assist with processing distributed jobs by retrieving and/or synchronizing supplemental job data. The method includes receiving a request to perform a job and opening a first connection (e.g., persistent connection) between a primary machine and a secondary machine, and transmitting by the primary machine a request pertaining to the job to the secondary machine using a second connection, the job to be performed by the secondary machine. The method also includes receiving by the primary machine using the second connection a task request for supplemental information pertaining to the job, transmitting by the primary machine a task response including the supplemental information to the secondary machine, and receiving a job result for the job using the second connection.
1. A method comprising:
receiving, by a primary machine, a request to perform a job;
opening a first connection between the primary machine and a secondary machine, the first connection being a persistent connection;
transmitting, by the primary machine, a request pertaining to the job to the secondary machine using a second connection;
receiving, by the primary machine using the first connection, a task request for supplemental information pertaining to the job;
transmitting, by the primary machine, a task response including the supplemental information to the secondary machine; and
receiving a job result for the job using the second connection.
2. The method of claim 1 , further comprising: synchronizing a portion of a first data store on the primary machine with a second data store on the secondary machine using the supplemental information of the task response.
3. The method of claim 1 , wherein data received over the persistent connection comprises a stream of separate task requests in a Java Script Object Notation (JSON) format.
4. The method of claim 1 , wherein the first connection and the second connection are Hyper Text Transfer Protocol (HTTP) connections transmitting data in accordance with a Representational State Transfer (REST) architecture, and wherein the first connection is a persistent HTTP connection initiated by an HTTP GET call.
5. The method of claim 1 , wherein the job pertains to a search query, and the job result is a search result produced using a late-binding schema.
6. The method of claim 1 , wherein the primary machine includes a lookup-table having an entry with translation data that links data identified in the task request to different data, wherein the supplemental information included within the task response comprises the translation data.
7. The method of claim 1 , wherein the primary machine includes a data store and the secondary machine includes a secondary data store, wherein the method further comprises synchronizing a portion of the secondary data store with the data store of the primary machine using a one-way pull synchronization implemented using task requests and task responses.
8. The method of claim 1 , wherein the secondary machine is prevented from initiating connections with the primary machine, and the primary machine initiates an opening of the persistent connection before initiating an opening of the second connection, and the persistent connection persists beyond a closing of the second connection.
9. The method of claim 1 , wherein the persistent connection is initiated by the primary machine during a machine setup procedure performed by an administrator and the second connection is opened after the job has been created from a user request.
10. The method of claim 1 , wherein the transmitting the task response comprises:
opening a third connection with the secondary machine after receiving the task request;
transmitting the task response using the third connection; and
closing the third connection.
11. The method of claim 1 , wherein task requests are transmitted over the persistent connection without being transmitted over any other connection.
12. The method of claim 1 , wherein the persistent connection is dedicated to receiving task requests from the secondary machine without transmitting task requests or task responses to the secondary machine.
13. The method of claim 1 , wherein the secondary machine comprises a plurality of secondary machines and the persistent connection with the secondary machine comprises a plurality of persistent connections, each persistent connection being between the primary machine and one of the plurality of secondary machines, at least one of the plurality of secondary machines having persistent connections with multiple primary machines.
14. The method of claim 1 , wherein the job pertains to a search query for searching source data that comprises raw machine data.
15. The method of claim 1 , wherein the job pertains to a search query for searching source data that comprises time-stamped events, and wherein each of the time-stamped events includes at least a portion of raw machine data.
16. A system comprising:
a memory; and
one or more processing devices coupled to the memory to:
receive, by a primary machine, a request to perform a job;
open a first connection between the primary machine and a secondary machine, the first connection being a persistent connection;
transmit, by the primary machine, a request pertaining to the job to the secondary machine using a second connection;
receive, by the primary machine using the first connection, a task request for supplemental information pertaining to the job;
transmit, by the primary machine, a task response including the supplemental information to the secondary machine; and
receive a job result for the job using the second connection.
17. The system of claim 16 , wherein the one or more processing devices further to:
synchronize a portion of a first data store on the primary machine with a second data store on the secondary machine using the supplemental information of the task response.
18. The system of claim 16 , wherein data received over the first connection comprises a stream of separate task requests in a Java Script Object Notation (JSON) format.
19. The system of claim 16 , wherein the first connection and the second connection are Hyper Text Transfer Protocol (HTTP) connections transmitting data in accordance with a Representational State Transfer (REST) architecture, and wherein the first connection is a persistent HTTP connection initiated by an HTTP GET call.
20. The system of claim 16 , wherein the job pertains to a search query, and the job result is a search result produced using a late-binding schema.
21. The system of claim 16 , wherein the primary machine includes a lookup-table having an entry with translation data that links data identified in the task request to different data, wherein the supplemental information included within the task response comprises the translation data.
22. The system of claim 16 , wherein the secondary machine is prevented from initiating connections with the primary machine, and the primary machine initiates an opening of the first connection before initiating an opening of the second connection, and the first connection persists beyond the closing of the second connection.
23. The system of claim 16 , wherein the opening the first connection is initiated by the primary machine during a machine setup procedure performed by an administrator and the second connection is opened after the job has been created from a user request.
24. A non-transitory computer readable storage medium encoding instructions thereon that, in response to execution by a processing device, cause the processing device to perform operations comprising:
receiving, by a primary machine, a request to perform a job;
opening a first connection between the primary machine and a secondary machine, the first connection being a persistent connection;
transmitting, by the primary machine, a request pertaining to the job to the secondary machine using a second connection;
receiving, by the primary machine using the first connection, a task request for supplemental information pertaining to the job;
transmitting, by the primary machine, a task response including the supplemental information to the secondary machine; and
receiving a job result for the job using the second connection.
25. The non-transitory computer readable storage medium of claim 24 , further comprising: synchronizing a portion of a first data store on the primary machine with a second data store on the secondary machine using the supplemental information of the task response.
26. The non-transitory computer readable storage medium of claim 24 , wherein data received over the first connection comprises a stream of separate task requests in a Java Script Object Notation (JSON) format.
27. The non-transitory computer readable storage medium of claim 24 , wherein the first connection and the second connection are Hyper Text Transfer Protocol (HTTP) connections transmitting data in accordance with a Representational State Transfer (REST) architecture, and wherein the first connection is a persistent HTTP connection initiated by an HTTP GET call.
28. The non-transitory computer readable storage medium of claim 24 , wherein the job pertains to a search query, and the job result is a search result produced using a late-binding schema.
29. The transitory computer readable storage medium of claim 24 , wherein the primary machine includes a lookup-table having an entry with translation data that links data identified in the task request to different data, wherein the supplemental information included within the task response comprises the translation data.
30. The non-transitory computer readable storage medium of claim 24 , wherein the secondary machine is prevented from initiating connections with the primary machine, and the primary machine initiates an opening of the first connection before initiating an opening of the second connection, and the first connection persists beyond the closing of the second connection.