Massively-parallel system architecture and method for real-time extraction of high-value information from data streams

1. A method for real-time extraction of high-value information from data streams, comprising:

at a computer system including a plurality of processors and memory storing programs for execution by the processors,

receiving a plurality of data streams, wherein each of the data streams includes a plurality of posts, wherein each of the posts includes a content portion and one or more source characteristics;

in real time, for each post in a particular data stream:

assigning the post a post identifier;

generating a content packet, wherein the content packet includes:

a respective source identifier corresponding to a respective source profile, wherein the respective source profile includes information obtained from previously received posts associated the respective source identifier, and

content information corresponding to the content portion of the post;

querying the memory to access a source profile using the respective source identifier;

correlating the content packet with information from the source profile to produce a correlated content packet, the information including at least some of the information obtained from the previously received posts associated the respective source identifier;

broadcasting the correlated content packet to a plurality of filter graph definitions, wherein each of the filter graph definitions is configured to identify posts with high value information according to the respective filter graph definition, wherein posts are identified at least in part on based on both the information obtained from the previously received posts associated the respective source identifier and the content information corresponding to the content portion of the post, and each of the filter graph definitions is configured to execute on at least a subset of the plurality of processors.

2. The method of claim 1 , further including:

indexing each post in the data stream; and

storing each post in the data stream.

3. The method of claim 1 , wherein each of the filter graph definitions includes a plurality of filter nodes, each of which is configured to accept or reject individual posts in a data stream based on content and/or metadata information associated with the individual posts, and wherein the filter nodes included in a respective filter graph definition are combined according to a predefined arrangement so as to identify the individual posts with high value information according to the respective filter graph definition, wherein configuring the filter graph definitions to execute on at least a subset of the plurality of processors includes preparing the filter nodes for execution on respective ones of the processors.

4. The method of claim 1 , further including:

assigning each of the one or more source characteristics a respective source identifier; and

generating one or more source packets, wherein the one or more source packets each include the post identifier as well as source information corresponding to a respective source characteristic.

5. The method of claim 4 , further including, for each post in a particular data stream, after broadcasting the correlated content packet, updating the source profile using the one or more source packets.

6. The method of claim 1 , wherein the one or more source characteristics include one or more of author information and publisher information.

7. The method of claim 6 , further comprising normalizing the author information according to a standard author format.

8. The method of claim 6 , further comprising normalizing the author information and the publisher information and associating the author information and the publisher information with respective posts associated with the same author and/or publisher using the respective source identifier.

9. The method of claim 8 , wherein the filter nodes include a plurality of natural language filters.

10. The method of claim 9 , wherein the natural language filters are specified lexically using regular expressions.

11. The method of claim 10 , wherein the regular expressions are implemented as deterministic finite automatons.

12. The method of claim 1 , wherein:

the computer system includes a source profile caching sub-system with one or more cache levels including at least a first-level cache storing a plurality of first source profiles; and

for each post in a particular data stream, querying the memory to access the source profile using the respective source identifier further includes:

transmitting the respective source identifier to the first-level cache; and

querying the first-level cache to access the source profile using the respective source identifier; and

when the querying of the first-level cache returns a result corresponding to a first-level cache hit, returning the source profile.

13. The method of claim 12 , wherein the source profile caching sub-system further includes a second-level cache storing a plurality of second source profiles; and

querying the memory to access the source profile using the respective source identifier further includes:

when querying of the first-level cache returns a result corresponding to a first-level cache-miss, automatically transmitting the respective source identifier to the second-level cache; and

querying the second-level cache to access the source profile using the respective source identifier.

14. The method of claim 13 , further including, when the second-level cache returns a result corresponding to a second-level cache hit, transferring the source profile to the first-level cache memory, thereby adding the source profile to the first source profiles; and

discarding, from the first source profiles, respective ones of the first source profiles according to least-recently posted criteria.

15. The method of claim 14 , wherein the first-level cache is of sufficient size to store, on average, all source profiles for which a corresponding packet has been received within a previous month.

16. The method of claim 1 , further including:

harvesting, using a third-party data aggregator, at least one first post in the plurality of posts from a first web site; and

harvesting, using the third-party data aggregator, at least one second post in the plurality of posts from a second web site.

17. The method of claim 1 , further including harvesting, using a direct crawler associated with a third website, one or more third posts in the plurality of posts from the third web site.

18. The method of claim 1 , further including harvesting, using an application program interface associated with a fourth website, one or more fourth posts in the plurality of posts from the fourth web site.

19. A computer system for identifying high-value information in data streams, comprising:

one or more processors;

memory storing one or more programs to be executed by the one or more processors;

the one or more programs comprising instructions for:

receiving a plurality of data streams, wherein each of the data streams includes a plurality of posts, wherein each of the posts includes a content portion and one or more source characteristics;

in real time, for each post in a particular data stream:

assigning the post a post identifier;

generating a content packet, wherein the content packet includes:

a respective source identifier corresponding to a respective source profile, wherein the respective source profile includes information obtained from previously received posts associated the respective source identifier, and

content information corresponding to the content portion of the post;

querying the memory to access a source profile using the respective source identifier;

correlating the content packet with information from the source profile to produce a correlated content packet, the information including at least some of the information obtained from the previously received posts associated the respective source identifier;

broadcasting the correlated content packet to a plurality of filter graph definitions, wherein each of the filter graph definitions is configured to identify posts with high value information according to the respective filter graph definition, wherein posts are identified at least in part on based on both the information obtained from the previously received posts associated the respective source identifier and the content information corresponding to the content portion of the post, and each of the filter graph definitions is configured to execute on at least a subset of the plurality of processors.

20. A non-transitory computer readable storage medium storing one or more programs configured for execution by a computer system, the one or more programs comprising instructions for:

receiving a plurality of data streams, wherein each of the data streams includes a plurality of posts, wherein each of the posts includes a content portion and one or more source characteristics;

in real time, for each post in a particular data stream:

assigning the post a post identifier;

generating a content packet, wherein the content packet includes:

a respective source identifier corresponding to a respective source profile, wherein the respective source profile includes information obtained from previously received posts associated the respective source identifier, and

content information corresponding to the content portion of the post;

querying the memory to access a source profile using the respective source identifier;

correlating the content packet with information from the source profile to produce a correlated content packet, the information including at least some of the information obtained from the previously received posts associated the respective source identifier;

broadcasting the correlated content packet to a plurality of filter graph definitions, wherein each of the filter graph definitions is configured to identify posts with high value information according to the respective filter graph definition, wherein posts are identified at least in part on based on both the information obtained from the previously received posts associated the respective source identifier and the content information corresponding to the content portion of the post, and each of the filter graph definitions is configured to execute on at least a subset of the plurality of processors.