Searching raw data from an external data system using a dual mode search system
A search request received at a computer of a search support system is processed by analyzing the received search request to identify request parameters and connecting to a system index of the search support system that is referenced in the request parameters. An external result provider (ERP) process is initiated that establishes communication between the search support system and a data source external to the search support system, for a virtual index referenced in the request parameters. Thus, the ERP process provides an interface between the search support system and external data sources, such as by third parties. The ERP process can operate in a streaming mode (providing real-time search results with minimal processing) and/or a reporting mode (providing results with a greater delay and processing extent) and can switch between modes. The search request results are received from the connected system indexes and the referenced virtual indexes.
1. A method, comprising:
receiving, by a first device, a first search request from a second device;
communicating, by the first device, a second search request to an external data system, the search request based at least in part on the first search request;
receiving, by the first device, raw data from the external data system obtained from raw data accessible by the external data system, the raw data fulfills at least a portion of the first search request;
processing, by the first device, the raw data, as the raw data is being received, responsive to the first search request;
while the first device is processing the raw data, streaming at least a portion of the raw data to the second device;
wherein the processing the raw data further comprises:
creating, by the first device, at least one interim search result from the raw data based on the first search request;
sending the at least one interim search result to the second device.
2. The method of claim 1 , wherein the first device stops streaming the at least a portion of the raw data to the second device upon creating the at least one interim search result.
3. The method of claim 1 , wherein streaming the at least a portion of the raw data to the second device is stopped upon determining that sending the at least one interim search result to the second device is more efficient than streaming the at least a portion of the raw data to the second device.
4. The method of claim 1 , wherein the raw data includes unprocessed data from the external data system as the unprocessed data is retrieved by the external data system.
5. The method of claim 1 , wherein the at least one interim search result for the first search request comprises at least one of: statistics, aggregates, or filtered results.
6. The method of claim 1 , wherein the processing of the raw data includes applying a late binding schema to the raw data.
7. The method as in claim 1 , wherein the external data system is a HADOOP data system.
8. The method as in claim 1 , further comprising:
determining whether to cache at least a portion of the at least one interim search result for the first search request such that the at least one cached interim search result is usable as a response in a subsequent received search request.
9. The method as in claim 1 , further comprising:
determining whether cached search results produced by a previous received search request are available to help satisfy at least a portion of the first search request.
10. One or more non-transitory computer-readable storage media, storing instructions, which when executed by one or more processors cause performance of:
receiving, by a first device, a first search request from a second device;
communicating, by the first device, a second search request to an external data system, the search request based at least in part on the first search request;
receiving, by the first device, raw data from the external data system obtained from raw data accessible by the external data system, the raw data fulfills at least a portion of the first search request;
processing, by the first device, the raw data, as the raw data is being received, responsive to the first search request;
while the first device is processing the raw data, streaming at least a portion of the raw data to the second device;
wherein the processing the raw data further comprises:
creating, by the first device, at least one interim search result from the raw data based on the first search request;
sending the at least one interim search result to the second device.
11. The one or more non-transitory storage media of claim 10 , wherein the first device stops streaming the at least a portion of the raw data to the second device upon creating the at least one interim search result.
12. The one or more non-transitory storage media of claim 10 , wherein streaming the at least a portion of the raw data to the second device is stopped upon determining that sending the at least one interim search result to the second device is more efficient than streaming the at least a portion of the raw data to the second device.
13. The one or more non-transitory storage media of claim 10 , wherein the raw data includes unprocessed data from the external data system as the unprocessed data is retrieved by the external data system.
14. The one or more non-transitory storage media of claim 10 , wherein the at least one interim search result for the first search request comprises at least one of:
statistics, aggregates, or filtered results.
15. The one or more non-transitory storage media as in claim 10 , wherein the external data system is a HADOOP data system.
16. An apparatus, comprising:
a search request receiving device, at a first device, implemented at least partially in hardware, that receives a first search request from a second device;
a search request sending device, at the first device, implemented at least partially in hardware, that communicates a second search request to an external data system, the search request based at least in part on the first search request;
a raw data receiving device, at the first device, implemented at least partially in hardware, that receives raw data from the external data system obtained from raw data accessible by the external data system, the raw data fulfills at least a portion of the first search request;
a raw data processor, at the first device, implemented at least partially in hardware, that processes the raw data, as the raw data is being received, responsive to the first search request;
a raw data streaming device, at a first device, implemented at least partially in hardware, that while the raw data processor is processing the raw data, streams at least a portion of the raw data to the second device;
wherein the raw data processor further:
creates at least one interim search result from the raw data based on the first search request;
sends the at least one interim search result to the second device.
17. The apparatus of claim 16 , wherein the raw data streaming device stops streaming the at least a portion of the raw data to the second device upon creating the at least one interim search result.
18. The apparatus of claim 16 , wherein the raw data streaming device stops streaming the at least a portion of the raw data to the second device upon the raw data processor determining that sending the at least one interim search result to the second device is more efficient than streaming the at least a portion of the raw data to the second device.
19. The apparatus of claim 16 , wherein the raw data includes unprocessed data from the external data system as the unprocessed data is retrieved by the external data system.
20. The apparatus of claim 16 , wherein the at least one interim search result for the first search request comprises at least one of: statistics, aggregates, or filtered results.