State grouping methodologies to compress transitions in a deterministic automata
A hardware system for signature matching in a distributed network is disclosed. The hardware system comprises a network processor and a memory. The network processor is configured to perform horizontal compression on a state table using bitmaps, wherein the state table has a plurality of states and state transitions. The processor is also configured to perform a first grouping of states of the state table using the bitmaps to generate a first one or more sets of states, perform a second grouping of states of the state table based on the first one or more sets of states and a transition threshold to generate a second one or more sets of states, perform a conquer step grouping of the states of the state table based on the second one or more sets of states and conquer criteria to generate third one or more sets of states, and generate a two dimensioned compressed state table based on the third one or more sets of states. The memory circuit is configured to store the two dimensioned compressed state table.
1. A hardware system for signature matching in a distributed network system, the hardware system comprising:
a network processor configured to:
perform horizontal compression on a state table using bitmaps, wherein the state table has a plurality of states and state transitions;
perform a first grouping of states of the state table using the bitmaps to generate a first one or more sets of states;
perform a second grouping of states of the state table based on the first one or more sets of states and a transition threshold to generate a second one or more sets of states;
perform a conquer step grouping of the states of the state table based on the second one or more sets of states and conquer criteria to generate third one or more sets of states; and
generate a two dimensioned compressed state table based on the third one or more sets of states; and
a memory circuit configured to store the two dimensioned compressed state table.
2. The system of claim 1 , wherein the network processor is further configured to identify an enhanced leader state for each of the third one or more sets of states and one or more combinations of the third one or more sets of states by identifying the state that provides the lowest number of uncompressed transitions per set.
3. The system of claim 1 , wherein the conquer criteria includes to combine a first set and a second set of the second one or more sets if the first set and the second set have the same bitmap.
4. The system of claim 1 , wherein the conquer criteria includes to combine a first set and a second set of the second one or more sets based on a number of uncompressed transitions for a combination of the first set and the second set.
5. The system of claim 4 , wherein the network processor is configured to determine the number of uncompressed transitions for the combination of the first set and the second set by identifying an enhanced leader state.
6. The system of claim 5 , wherein the enhanced leader state is selected that provides a least number of uncompressed transitions for the combination.
7. The system of claim 1 , wherein the network processor is configured to perform a plurality of iterations of the conquer step grouping to generate the third one or more sets of states.
8. The system of claim 1 , wherein the transition threshold indicates that a threshold percentage of transitions of the states in a set are similar to each other.
9. The system of claim 8 , wherein the transition threshold is 85 percent.
10. The system of claim 1 , wherein the network processor is configured to perform a similarity check based on the transition threshold, wherein the similarity check identifies states that have similar transitions at indexes.
11. The system of claim 1 , wherein the network processor is configured to generate a bitmask for each member state in reference to a leader state of a set.
12. The system of claim 11 , wherein the bitmask includes a ‘1’ to indicate that a transition is stored and a ‘0’ to indicate that a transition is not stored.
13. The system of claim 1 , wherein the network processor is configured to determine a number of uncompressed transitions for a set of the second one or more sets of states by determining uncompressed transitions for a leader state and determining transitions in member states that differ from the leader state at an index.
14. A method of operating a network processor, the method comprisii providing a state table having a plurality of states and state transitions;
performing horizontal compression on the state table;
performing first stage grouping of the plurality of states to generate first one or more sets of states;
performing conquer step grouping of the first one or more sets of states using conquer criteria to generate final one or more sets of states; and
performing vertical compression of the state table using the final one or more states to generate a compressed state table that represents a database of digital signatures.
15. The method of claim 14 , wherein performing the conquer step grouping includes identifying an enhanced leader state for each of the first one or more sets of states.
16. The method of claim 14 , further comprising performing deep packet inspection (DPI) of network traffic using the compressed state table.