Mixed mode ERP process executing a mapreduce task
A search request received at a computer of a search support system is processed by analyzing the received search request to identify request parameters and connecting to a system index of the search support system that is referenced in the request parameters. An external result provider (ERP) process is initiated that establishes communication between the search support system and a data source external to the search support system, for a virtual index referenced in the request parameters. Thus, the ERP process provides an interface between the search support system and external data sources, such as by third parties. The ERP process can operate in a streaming mode (providing real-time search results with minimal processing) and/or a reporting mode (providing results with a greater delay and processing extent) and can switch between modes. The search request results are received from the connected system indexes and the referenced virtual indexes.
1. A method comprising:
receiving, by a first device from an external data system, files that are responsive to a search request;
employing a streaming mode to begin streaming the files from the first device to a second device that is employable by a local data system;
employing a reporting mode to cause the first device to execute a MapReduce task, while concurrently employing the streaming mode, to create a report for the files; and
in response to completing the MapReduce task, causing the first device to stop streaming the files and to send the report to the second device.
2. The method of claim 1 , wherein causing the first device to execute the MapReduce task comprises causing the first device to execute a MapReduce job comprising a plurality of MapReduce tasks creating a plurality reports for the files, the method further comprising, upon completion of each of the MapReduce tasks, causing the first device to pause streaming the files and send a corresponding one of the reports to the second device.
3. The method of claim 1 , wherein causing the first device to stop streaming the files comprises pausing the streaming while sending the report to the second device.
4. The method of claim 1 , further comprising determining to communicate the report to the second device.
5. The method of claim 1 , wherein the MapReduce task is one of a plurality of MapReduce tasks creating a plurality reports, the method further comprising determining to send each of the plurality of reports upon completion of a corresponding one of the plurality of MapReduce tasks.
6. The method of claim 1 , the method further comprising causing the second device to switch from using the files from the streaming mode to using results from the reporting mode based on a determination that using the results from the reporting mode is more efficient than using the streaming mode.
7. The method of claim 1 , wherein the external data system is a HADOOP data system.
8. The method of claim 1 , further comprising determining to cache the report as a cached search result for the search request such that the cached search result is usable as a response in a subsequent received search request.
9. The method of claim 1 , further comprising determining that a cached search result produced by a previous received search request is available to satisfy at least a portion of the search request.
10. The method of claim 1 , further comprising determining to employ the reporting mode based on the search request.
11. One or more non-transitory computer-readable storage media, storing instructions, which when executed by one or more processors cause performance of operations comprising:
receiving, by a first device from an external data system, files that are responsive to a search request;
employing a streaming mode to begin streaming the files from the first device to a second device that is employable by a local data system;
employing a reporting mode to cause the first device to execute a MapReduce task, while concurrently employing the streaming mode, to create a report for the files; and
in response to completing the MapReduce task, causing the first device to stop streaming the files and to send the report to the second device.
12. The one or more non-transitory computer-readable storage media of claim 11 , wherein causing the first device to execute the MapReduce task comprises causing the first device to execute a MapReduce job comprising a plurality of MapReduce tasks creating a plurality reports for the files, the operations further comprising, upon completion of each of the MapReduce tasks, causing the first device to pause streaming the files and send a corresponding one of the reports to the second device.
13. The one or more non-transitory computer-readable storage media of claim 11 , wherein causing the first device to stop streaming the files comprises pausing the streaming while sending the report to the second device.
14. The one or more non-transitory computer-readable storage media of claim 11 , the operations further comprising determining to communicate the report to the second device.
15. The one or more non-transitory computer-readable storage media of claim 11 , wherein the MapReduce task is one of a plurality of MapReduce tasks creating a plurality reports, the operations further comprising determining to send each of the plurality of reports upon completion of a corresponding one of the plurality of MapReduce tasks.
16. The one or more non-transitory computer-readable storage media of claim 11 , the operations further comprising causing the second device to switch from using the files from the streaming mode to using results from the reporting mode based on a determination that using the results from the reporting mode is more efficient than using the streaming mode.
17. The one or more non-transitory computer-readable storage media of claim 11 , wherein the external data system is a HADOOP data system.
18. The one or more non-transitory computer-readable storage media of claim 11 , the operations further comprising determining to cache the report as a cached search result for the search request such that the cached search result is usable as a response in a subsequent received search request.
19. The one or more non-transitory computer-readable storage media of claim 11 , the operations further comprising determining that a cached search result produced by a previous received search request is available to satisfy at least a portion of the search request.
20. The one or more non-transitory computer-readable storage media of claim 11 , the operations further comprising determining to employ the reporting mode based on the search request.
21. A computing system comprising:
one or more processors; and
a memory coupled with the one or more processors, the memory having instructions stored thereon, which, when executed by the one or more processors, cause the computing system to perform operations comprising:
receiving, by a first device from an external data system, files that are responsive to a search request;
employing a streaming mode to begin streaming the files from the first device to a second device that is employable by a local data system;
employing a reporting mode to cause the first device to execute a MapReduce task, while concurrently employing the streaming mode, to create a report for the files; and
in response to completing the MapReduce task, causing the first device to stop streaming the files and to send the report to the second device.
22. The computing system of claim 21 , wherein causing the first device to execute the MapReduce task comprises causing the first device to execute a MapReduce job comprising a plurality of MapReduce tasks creating a plurality reports for the files, the operations further comprising, upon completion of each of the MapReduce tasks, causing the first device to pause streaming the files and send a corresponding one of the reports to the second device.
23. The computing system of claim 21 , wherein causing the first device to stop streaming the files comprises pausing the streaming while sending the report to the second device.
24. The computing system of claim 21 , the operations further comprising determining to communicate the report to the second device.
25. The computing system of claim 21 , wherein the MapReduce task is one of a plurality of MapReduce tasks creating a plurality reports, the operations further comprising determining to send each of the plurality of reports upon completion of a corresponding one of the plurality of MapReduce tasks.
26. The computing system of claim 21 , the operations further comprising causing the second device to switch from using the files from the streaming mode to using results from the reporting mode based on a determination that using the results from the reporting mode is more efficient than using the streaming mode.
27. The computing system of claim 21 , wherein the external data system is a HADOOP data system.
28. The computing system of claim 21 , the operations further comprising determining to cache the report as a cached search result for the search request such that the cached search result is usable as a response in a subsequent received search request.
29. The computing system of claim 21 , the operations further comprising determining that a cached search result produced by a previous received search request is available to satisfy at least a portion of the search request.
30. The computing system of claim 21 , the operations further comprising determining to employ the reporting mode based on the search request.