Method and apparatus for monitoring a data stream to detect a pattern of data elements using bloom filters
A system that determines whether a pattern of data elements was observed in a stream of data. During operation, the system receives a query which seeks to determine whether the pattern of data elements was observed in the stream of data. In response to the query, the system performs a number of lookups in an enhanced Bloom filter to determine whether the pattern of data elements was observed in the stream of data, wherein the enhanced Bloom filter includes multiple instances of a Bloom filter, and wherein each instance of the Bloom filter is associated with a different time interval. If so, the system generates a notification that the pattern of data elements was observed.
1. A method for using a computer to determine whether a pattern of data elements was observed in a stream of data, comprising:
observing the stream of data;
recording data elements observed in the stream of data in an enhanced Bloom filter that comprises multiple instances of a Bloom filter, each instance of the Bloom filter being associated with a separate and different time interval, wherein recording a data element comprises updating a record in corresponding elements in each instance of the Bloom filter;
receiving a query which seeks to determine whether the pattern of data elements was observed in the stream of data;
in response to the query, performing a number of lookups in the enhanced Bloom filter to determine whether the pattern of data elements was observed in the stream of data; and
if so, generating a notification that the pattern of data elements was observed.
2. The method of claim 1 ,
wherein the pattern of data elements comprises a data element which is observed in multiple time intervals; and
wherein determining whether the pattern of data elements was observed involves determining whether the data element was observed in multiple instances of the Bloom filter within the enhanced Bloom filter.
3. The method of claim 1 ,
wherein the pattern of data elements comprises multiple data elements which are observed in multiple time intervals; and
wherein determining whether the pattern of data elements was observed involves determining whether the multiple data elements were observed in multiple corresponding instances of the Bloom filter within the enhanced Bloom filter.
4. The method of claim 1 ,
wherein the pattern of data elements comprises multiple data elements which are observed in a given time interval; and
wherein determining whether the pattern of data elements was observed involves determining whether the multiple data elements were observed in a corresponding instance of the Bloom filter within the enhanced Bloom filter.
5. The method of claim 1 , wherein the data elements are network packets.
6. The method of claim 5 ,
wherein the network packets are generated during an interaction between a user and an Internet advertisement; and
wherein the query is used to identify instances of click fraud, which involves automatically or a manually clicking on an Internet advertisement to fraudulently generate a clickthrough for the Internet advertisement.
7. The method of claim 5 ,
wherein the network packets are generated by a computer virus attempting to infect computer systems; and
wherein the query is used to identify a cluster of packets sent by the computer virus from an infected computer system to multiple ports on other computer systems.
8. The method of claim 5 ,
wherein the network packets are generated by inter blog communications; and
wherein the query is used to identify spam pings and spam comments, which are comments posted on spam blogs that are automatically propagated to other blogs with the intent to direct traffic to a specified website.
9. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for determining whether a pattern of data elements was observed in a stream of data, wherein the method comprises:
observing the stream of data;
recording data elements observed in the stream of data in an enhanced Bloom filter that comprises multiple instances of a Bloom filter, each instance of the Bloom filter being associated with a separate and different time interval, wherein recording a data element comprises updating a record in corresponding elements in each instance of the Bloom filter;
receiving a query which seeks to determine whether the pattern of data elements was observed in the stream of data;
in response to the query, performing a number of lookups in the enhanced Bloom filter to determine whether the pattern of data elements was observed in the stream of data; and
if so, generating a notification that the pattern of data elements was observed.
10. The computer-readable storage medium of claim 9 ,
wherein the pattern of data elements comprises a data element which is observed in multiple time intervals; and
wherein determining whether the pattern of data elements was observed involves determining whether the data element was observed in multiple instances of the Bloom filter within the enhanced Bloom filter.
11. The computer-readable storage medium of claim 9 ,
wherein the pattern of data elements comprises multiple data elements which are observed in multiple time intervals; and
wherein determining whether the pattern of data elements was observed involves determining whether the multiple data elements were observed in multiple corresponding instances of the Bloom filter within the enhanced Bloom filter.
12. The computer-readable storage medium of claim 9 ,
wherein the pattern of data elements comprises multiple data elements which are observed in a given time interval; and
wherein determining whether the pattern of data elements was observed involves determining whether the multiple data elements were observed in a corresponding instance of the Bloom filter within the enhanced Bloom filter.
13. The computer-readable storage medium of claim 9 , wherein the data elements are network packets.
14. The computer-readable storage medium of claim 13 ,
wherein the network packets are generated during an interaction between a user and an Internet advertisement; and
wherein the query is used to identify instances of click fraud, which involves automatically or a manually clicking on an Internet advertisement to fraudulently generate a clickthrough for the Internet advertisement.
15. The computer-readable storage medium of claim 13 ,
wherein the network packets are generated by a computer virus attempting to infect computer systems; and
wherein the query is used to identify a cluster of packets sent by the computer virus from an infected computer system to multiple ports on other computer systems.
16. The computer-readable storage medium of claim 13 ,
wherein the network packets are generated by inter blog communications; and
wherein the query is used to identify spam pings and spam comments, which are comments posted on spam blogs that are automatically propagated to other blogs with the intent to direct traffic to a specified website.
17. An apparatus that determines whether a pattern of data elements was observed in a stream of data, comprising:
a processor;
a memory;
a storage device; and
a data monitoring mechanism configured to:
observe the stream of data;
record data elements observed in the stream of data in an enhanced Bloom filter that comprises multiple instances of a Bloom filter, each instance of the Bloom filter being associated with a separate and different time interval, wherein recording a data element comprises updating a record in corresponding elements in each instance of the Bloom filter;
receive a query which seeks to determine whether the pattern of data elements was observed in the stream of data;
in response to the query, to perform a number of lookups in the enhanced Bloom filter to determine whether the pattern of data elements was observed in the stream of data; and
if so, to generate a notification that the pattern of data elements was observed.