DATA PROCESSING SYSTEM INCLUDING A SEARCH ENGINE
A search engine receives a search trigger for a task. In response to identifying data that is responsive to the search trigger, a notification of the identified data is sent to the task, to cause the task to process the identified data. The search engine receives a notification of result data produced by the task based on the processing of the identified data.
1 . A method comprising:
receiving, by a search engine in a data processing system including a processor, search triggers for respective tasks executable in the system;
in response to identifying data that is responsive to at least one of the search triggers, causing, by the search engine, sending of a notification of the identified data to at least one of the tasks associated with the at least one search trigger, to cause the at least one task to process the identified data; and
receiving, by the search engine, a notification of result data produced by the at least one task based on the processing of the identified data.
2 . The method of claim 1 , further comprising:
updating, by the search engine, an index based on the result data.
3 . The method of claim 2 , further comprising:
determining, by the search engine using the updated index, that the result data is responsive to a second of the search triggers; and
sending, by the search engine, a notification of the result data to a corresponding one of the tasks associated with the second search trigger.
4 . The method of claim 1 , further comprising:
receiving, by the system, a new task created by a user; and
registering, by the system, a search trigger for the new task with the search engine.
5 . The method of claim 1 , further comprising:
determining a topology of the tasks based on relationships determined from the search triggers.
6 . The method of claim 5 , wherein the relationships are indicated by the search triggers indicating which task has registered an interest in result data from another of the tasks.
7 . The method of claim 5 , further comprising:
displaying the topology of the tasks in a user interface.
8 . The method of claim 1 , further comprising:
displaying a provenance of a particular task in a user interface, the provenance identifying object references to data objects that contributed to processing performed by the particular task.
9 . The method of claim 1 , further comprising:
converting, using a converter, between a schema of data provided by the search engine and a schema of data provided by one of the tasks.
10 . A data processing system comprising:
at least one processor;
an analytic task executable by the at least one processor to:
register a search trigger with a search engine of the data processing system;
receive, from the search engine, a notification of data responsive to the search trigger;
process the data to produce result data; and
send, to the search engine, a notification of the result data, to cause the search engine to store information associated with the result data.
11 . The system of claim 10 , wherein the data responsive to the search trigger is produced by another analytic task in response to notification of data provided by the search engine to the another analytic task.
12 . An article comprising at least one non-transitory machine-readable storage medium to store instructions that upon execution cause a data processing system to:
receive, by a search engine in the data processing system, search triggers for respective tasks executable in the system;
in response to identifying data that is responsive to a first of the search triggers, send a notification of the identified data to a first of the tasks associated with the first search trigger, to cause the first task to process the identified data;
receive, by the search engine, a notification of result data produced by the first task based on the processing of the identified data;
determine, by the search engine, that the result data is responsive to a second of the search triggers; and
send, by the search engine, a notification of the result data to a second of he tasks associated with the second search trigger.
13 . The article of claim 12 , wherein the instructions upon execution cause the system to further:
update an index based on receiving the result data,
wherein the determining that the result data is responsive to the second search trigger uses the updated index.
14 . The article of claim 12 , wherein instructions upon execution cause the system to further:
determine, based on relationships indicated by the search triggers, a topology of the tasks, wherein the topology of the tasks indicate which tasks are related to which other tasks.
15 . The article of claim 12 , wherein identifying the data that is responsive to the first search trigger includes identifying a new type of data not previously present in the system.