Identifying signatures for data sets

1. A non-transitory machine-readable storage medium encoded with instructions executable by a hardware processor of a computing device for identifying signatures for data sets, the machine-readable storage medium comprising instructions to cause the hardware processor to:

for each of a plurality of first data sets, obtain a data set signature;

generate a first data structure for storing each data set signature that is distinct from each other data set signature obtained for the plurality of first data sets;

for each of a plurality of second data sets, obtain at least one data subset;

generate a second data structure for storing each data subset that is obtained from the second data sets;

remove, from the first data structure, each data set signature that matches a data subset included in the second data structure; and

for each data set signature removed from the first data structure, identify each first data set from which the data set signature was obtained; and

for each identified first data set, obtain a new data set signature.

2. The storage medium of claim 1 , wherein the instructions further cause the hardware processor to:

receive a new data set; and

determine, using the first data structure, whether the new data set matches a data set signature.

3. The storage medium of claim 1 , wherein each data set signature has a signature length that matches a subset length of each data subset.

4. The storage medium of claim 1 , wherein the first data structure is a deterministic acyclic finite state automata.

5. The storage medium of claim 1 , wherein:

each of the plurality of first data sets has a plurality of data set signatures, and for each first data set, data set signatures are obtained based on a measure of complexity associated with each data set signature.

6. The storage medium of claim 5 , wherein:

data set signatures of each of the plurality of first data sets are ranked based the measures of complexity associated with the data set signatures.

7. The storage medium of claim 1 , wherein the instructions further cause the hardware processor to:

obtain a collection of first data sets;

group each first data set included in the collection into one of a plurality of groups based on measures of similarity between first data sets;

select, from each of the plurality of groups, one first data set as a representative data set for the group, and wherein each the plurality of first data sets includes the selected first data sets.

8. The storage medium of claim 1 , wherein the instructions further cause the hardware processor to:

determine that each data set signature of a plurality of data set signatures of a particular first data set matches a data subset included in the second data structure; and

in response to the determination, select one of the plurality of data set signatures of the particular first data set for inclusion in the first data structure, the selection being based on a number of second data sets that include a data subset that matches the selected data set signature.

9. A method for identifying byte signatures for byte streams, implemented by a hardware processor, the method comprising:

obtaining, for each of a plurality of malicious byte streams, a byte signature;

generating a first data structure for storing each obtained byte signature that is distinct from each other obtained byte signature;

obtaining, for each of a plurality of second byte streams, at least one byte subset, the plurality of second byte streams comprising malicious, benign, and unknown byte streams;

generating a second data structure for storing each obtained byte subset;

removing, from the first data structure, each byte signature that matches a byte subset included in the second data structure; and

identifying, for each byte signature removed from the first data structure, a malicious byte stream from which the byte signature was obtained; and

obtaining, for each identified malicious byte stream, a new byte signature.

10. The method of claim 9 , wherein the second data structure indicates, for each byte subset, a number of the second byte streams that included the byte subset.

11. The method of claim 10 , further comprising:

identifying a particular malicious byte stream for which each of a plurality of candidate byte signatures match a byte subset included in the second data structure; and

adding, to the first data structure, a distinct candidate byte signature of the plurality of candidate byte signatures of the particular malicious byte stream, the distinct candidate byte signature being the candidate byte signature having a matching byte subset for which a least cost of second byte stream matches is indicated.