Parallel detection of updates to a domain name system record system using a common filter
In an embodiment, a computer-implemented method detects updates to a domain name system record system. In the method, a stream of data points is received with each data point describing a record of a domain name system. For respective data points, a processor is selected from a plurality of processors. The processors are each configured to apply a data point against a common filter that assesses whether the data point is in a set. At the selected processor, the respective data point is applied to the common filter to determine whether the record is included in the set. When the record is determined not to be included in the set, a message is provided to indicate that the domain name system includes a new record, and the common filter is updated to include the data point in the set.
1. A computer-implemented method for detecting updates to a domain name system record system, comprising:
receiving a stream of data points each describing a record of a domain name system;
for respective data points in the stream of data points:
selecting a processor from a plurality of processors, each processor configured to apply a data point against a common filter that assesses whether the data point is in a set;
applying, at the selected processor, the respective data point to the common filter to determine whether the record is included in the set;
when the record is determined not to be included in the set:
providing a message to indicate that the domain name system includes a new record; and
updating the common filter to include the data point in the set,
wherein the common filter is a Bloom filter.
2. The method of claim 1 , further comprising:
organizing the data points into a plurality of packets, wherein the selecting comprises selecting a processor for respective packets in the plurality of packets; and
transmitting the packet to the selected processor to apply the common filter to each data point in the packet.
3. The method of claim 1 , further comprising:
for respective data points in the stream of data points, storing the record in a database; and
updating the common filter based on the database.
4. The method of claim 3 , further comprising:
when the record is determined not to be included in the set, storing the record and data from the database into a queue; and
retrieving data from the queue to update the common filter.
5. The method of claim 1 , further comprising, in response to the message, blocking a domain associated with the record.
6. The method of claim 1 , wherein the stream of data points includes: (i) a new base domain name from the domain name system, (ii) a fully qualified domain name from the domain name system, (iii) an indication of a resource record type of a fully qualified domain name from the domain name system, (iv) an identification of a resource record of a fully qualified domain name from the domain name system, and (v) an identification of a set of a resource record of a fully qualified domain name from the domain name system.
7. A non-transitory computer-readable medium having instructions stored thereon that, when executed by at least one computing device, causes the at least one computing device to perform a method for detecting updates to a domain name system record system, the method comprising:
receiving a stream of data points each describing a record of a domain name system;
for respective data points in the stream of data points:
selecting a processor from a plurality of processors, each processor configured to apply a data point against a common filter that assesses whether the data point is in a set;
applying, at the selected processor, the respective data point to the common filter to determine whether the record is included in the set;
when the record is determined not to be included in the set:
providing a message to indicate that the domain name system includes a new record; and
updating the common filter to include the data point in the set,
wherein the common filter is a Bloom filter.
8. The non-transitory computer-readable medium of claim 7 , the method further comprising:
organizing the data points into a plurality of packets, wherein the selecting comprises selecting a processor for respective packets in the plurality of packets; and
transmitting the packet to the selected processor to apply the common filter to each data point in the packet.
9. The non-transitory computer-readable medium of claim 7 , the method further comprising:
for respective data points in the stream of data points, storing the record in a database; and
updating the common filter based on the database.
10. The non-transitory computer-readable medium of claim 9 , the method further comprising:
when the record is determined not to be included in the set, storing the record and data from the database into a queue; and
retrieving data from the queue to update the common filter.
11. The non-transitory computer-readable medium of claim 9 , the method further comprising, in response to the message, blocking a domain associated with the record.
12. The non-transitory computer-readable medium of claim 9 , wherein the stream of data points includes: (i) a new base domain name from the domain name system, (ii) a fully qualified domain name from the domain name system, (iii) an indication of a resource record type of a fully qualified domain name from the domain name system, (iv) an identification of a resource record of a fully qualified domain name from the domain name system, and (v) an identification of a set of a resource record of a fully qualified domain name from the domain name system.
13. A system for detecting updates to a domain name system record system, comprising:
a computing device;
a stream receiver, implemented on the computing device, that receives a stream of data points each describing a record of a domain name system;
a plurality of processors, each processor configured to apply a data point against a common filter that assesses whether the data point is in a set;
a change detector, implemented on the computing device, that, for respective data points in the stream of data points, selects a processor from the plurality of processors, wherein the selected processor applies the respective data point to the common filter to determine whether the record is included in the set;
a stream outputter, implemented on the computing device, that, when the record is determined not to be included in the set, provider a message to indicate that the domain name system includes a new record; and
an updater, implemented on the computing device, that, when the record is determined not to be included in the set, updates the common filter to include the data point in the set,
wherein the common filter is a Bloom filter.
14. The system of claim 13 , further comprising:
a packetizer that organizes the data points into a plurality of packets and transmits the packet to the selected processor to apply the common filter to each data point in the packet, wherein the change detector selects a processor for respective packets in the plurality of packets.
15. The system of claim 13 , further comprising:
a database that stores the data points in the stream of data points, wherein the updater updates the common filter based on the database.
16. The system of claim 15 , further comprising:
a queue that stores the record and data from the database into; and
a writer that retrieves data from the queue to update the common filter.
17. The system of claim 13 , further comprising an response policy zone generator that, in response to the message, blocks a domain associated with the record.
18. The system of claim 13 , wherein the stream of data points includes: (i) a new base domain name from the domain name system, (ii) a fully qualified domain name from the domain name system, (iii) an indication of a resource record type of a fully qualified domain name from the domain name system, (iv) an identification of a resource record of a fully qualified domain name from the domain name system, and (v) an identification of a set of a resource record of a fully qualified domain name from the domain name system.