Processing unstructured data streams using continuous queries
Systems and methods for continuous processing of unstructured data streams are provided. Information may be stored in memory regarding a query, including associated search results and statistics derived at an identified time. After the identified time, unstructured data may be received from a plurality of streams over a communication network. A full-text search may be conducted on the received unstructured data based on the query to yield one or more matches. The stored statistics associated with the query may be retrieved from memory and updated based on the search results of the unstructured data received at the subsequent time. The updated statistics may then be stored in memory for retrieval at a subsequent time.
1. A method for continuous processing of unstructured data streams, the method comprising:
storing information in memory regarding an entity and an entity-specific query to be continuously applied, wherein the entity-specific query is associated with search results for one or more specified keywords in a context indicative of a specified sentiment and associated statistics derived at an identified time;
receiving unstructured data from a plurality of different continuous streams from different stream sources over a communication network, wherein the unstructured data is received a subsequent time after the identified time; and
executing instructions stored in memory, wherein execution of the instructions by a processor:
filters the received unstructured data based on entity-specific query, wherein filtering the received unstructured data comprises conducting a full-text search of the received unstructured data based on the keywords specified by the entity-specific query, and filtering results of the conducted search to yield one or more matches responsive to the sentiment specified by the entity-specific query,
retrieves the statistics that were derived at the identified time based on analytics for the matches responsive to the entity-specific query,
updates the associated statistics for the unstructured data received at the subsequent time, wherein the updated associated statistics are based on filtering the unstructured data received at the subsequent time based on the entity-specific query, and
indexes the unstructured data from the plurality of different continuous streams from the different stream sources based on the keywords specified by the entity-specific query, wherein the indexed data is tagged based on the sentiment specified by the entity-specific query, and wherein the indexed data and the associated updated statistics are merged and stored in memory for retrieval in association with the entity and a time period after the identified time.
2. The method of claim 1 , wherein the entity-specific query is continuously applied to the unstructured data in real-time, and wherein the unstructured data is indexed in real-time and associated with the entity.
3. The method of claim 1 , wherein the entity-specific query identifies a period of time of interest to the entity.
4. The method of claim 1 , wherein the entity-specific query is associated with a predetermined period of interest to the entity.
5. The method of claim 1 , wherein the entity-specific query further specifies a metric of interest to the entity.
6. The method of claim 1 , wherein the tag is further based on the identified context of interest to the entity.
7. The method of claim 1 , further comprising filtering the indexed data based on the tag.
8. The method of claim 7 , further comprising merging the filtered data with a stored list of previously filtered data to generate a new list, wherein the new list is stored in association with the entity and the identified time.
9. A system for continuous processing of unstructured data streams, the system comprising:
memory for storing information regarding an entity and an entity-specific query to be continuously applied, wherein the query is associated with search results for one or more specified keywords in a context indicative of a specified sentiment and associated statistics derived at an identified time;
a communication interface for receiving unstructured data from a plurality of different continuous streams from different stream sources over a communication network, wherein the unstructured data is received at a subsequent after the identified time; and
a processor for executing instructions stored in memory, wherein execution of the instructions by a processor:
filters the received unstructured data based on entity-specific query, wherein filtering the received unstructured data comprises conducting a full-text search of the received unstructured data based on the keywords specified by the entity-specific query, and filtering results of the conducted search to yield one or more matches responsive to the sentiment specified by the entity-specific query,
retrieves the statistics that were derived at the identified time based on analytics for the matches responsive to the entity-specific query,
updates the associated statistics for the unstructured data received at the subsequent time, wherein the updated associated statistics are based on filtering the unstructured data received at the subsequent time based on the sentiment specified by the entity-specific query, and
indexes the unstructured data from the plurality of different continuous streams from the different stream sources based on the keywords specified by the entity-specific query, wherein the indexed data is tagged based on the sentiment specified by the entity-specific query, wherein memory stores the indexed data merged with the associated updated statistics for retrieval in association with the entity and a time period after the identified time.
10. The system of claim 9 , wherein the entity-specific query is continuously applied to the unstructured data in real-time, and wherein the unstructured data is indexed in real-time and associated with the entity.
11. The system of claim 9 , wherein the entity-specific query identifies a period of time of interest to the entity.
12. The system of claim 9 , wherein the entity-specific query is associated with a predetermined period of interest to the entity.
13. The system of claim 9 , wherein the entity-specific query further specifies a metric of interest to the entity.
14. A non-transitory computer-readable storage medium, having embodied thereon a program executable by a processor to perform a method for continuous processing of unstructured data streams, the method comprising:
storing information in memory regarding an entity and an entity-specific query to be continuously applied, wherein the entity-specific query is associated with search results for one or more specified keywords in a context indicative of a specified sentiment and associated statistics derived at an identified time;
receiving unstructured data from a plurality of different continuous streams from different stream sources over a communication network, wherein the unstructured data is received at a subsequent time after the identified time;
filtering received unstructured data based on entity-specific query, wherein filtering the received unstructured data comprises conducting a full-text search of the received unstructured data based on the keywords specified by the entity-specific query, and filtering results of the conducted search to yield one or more matches responsive to the sentiment specified by the entity-specific query;
retrieving the statistics that were derived at the identified time based on analytics for the matches responsive to the entity-specific query;
updating the associated statistics for the unstructured data received at a subsequent time, wherein the updated associated statistics are based on filtering the unstructured data received at the subsequent time based on the entity-specific query; and
indexing the unstructured data from the plurality of different streams from the different stream sources based on the keywords specified by the entity-specific query, wherein the indexed data is tagged based on the sentiment specified by the entity-specific query, and wherein the indexed data and the associated updated statistics are merged and stored in memory for retrieval in association with the entity and a time period after the identified time.