Layer 7 network attack detection using machine learning feature contribution

1. A method comprising:

analyzing a plurality of attacks detected by a machine learning (ML) web application firewall (WAF) to determine a set of attacks of the plurality of attacks that were not identified as an attack by a rule-based WAF;

for each attack of the set of attacks that were not identified as an attack by the rule-based WAF, determining feature contribution data of the attack;

grouping, using a clustering algorithm, the set of attacks into one or more clusters based on feature contribution data of each of the set of attacks; and

for each of the one or more clusters:

determining, by a processing device, one or more features that have a high feature contribution to an attack probability of at least a threshold number of attacks of the cluster, wherein the high feature contribution of the one or more features are identified either as

a) top contributing features in a number of split classification requests that is higher than a threshold number of the split classification requests, or

b) features whose contribution to the attack probability is higher than a mean contribution of all of the one or more features in the number of the split classification requests that is higher than the threshold number of the split classification requests;

identifying, by the processing device, a new attack vector based on the one or more features of each attack in the cluster; and

generating a new rule for use by the rule-based WAF to identify the new attack vector.

2. The method of claim 1 , wherein determining feature contribution data of an attack comprises:

determining a feature set of the attack, the feature set of the attack indicating features that the attack is comprised of;

determining, using the ML WAF, an attack probability of the attack based on the feature set; and

decomposing the attack probability of the attack into a feature contribution made by each feature of the feature set of the attack to determine the feature contribution data of the attack.

3. The method of claim 1 , wherein generating the new rule comprises:

generating a rule indicating that a request having features matching the determined one or more features corresponds to an attack.

4. The method of claim 2 , wherein a feature set of an attack comprises one or more of: a set of RCE keywords, a set of query string N-grams, a client from which the attack was generated, a number of slash characters in a uniform resource locator (URL) of the attack, a location from which the attack originated, and a length of the URL of the attack.

5. The method of claim 1 , further comprising:

updating a set of security rules of the rule-based WAF with the new rule generated for each of the one or more clusters.

6. The method of claim 5 , further comprising:

receiving a request; and

in response to determining, by the rule-based WAF, that a feature set of the request includes features matching the new rule generated for any of the one or more clusters, taking one or more preventative actions.

7. A system comprising:

a memory; and

a processing device operatively coupled to the memory, the processing device to:

determine a plurality of requests that are each identified by both a machine learning (ML) web application firewall (WAF) and a rule-based WAF as an attack,

wherein the rule-based WAF uses a set of rules to identify attacks;

for each of the plurality of attacks, determine feature contribution data of the attack;

group, using a clustering algorithm, the plurality of attacks into one or more clusters based on feature contribution data;

determine one or more features that have a high feature contribution to an attack probability of at least a threshold number of attacks of the cluster, wherein the high feature contribution of the one or more features are identified either as

a) top contributing features in a number of split classifacation requests that is higher than a threshold number of the split classifcation requests, or

b) features whose contribution to the attack probability is higher than a mean contribution of all of the one or more features in the number of the split classifacation requests that is higher than the threshold number of the split classification requests;

identify a new attack vector based on the one or more features of each attack in the cluster; and

generate a new rule for use by the rule-based WAF to identify the new attack vector.

8. The system of claim 7 , wherein the processing device is further to:

in response to receiving a request that is identified by the ML WAF as an attack but is not identified as an attack by the rule-based WAF, determine by the ML WAF, feature contribution data of the request; and

determine, using the rule inference ML model, a particular rule among the set of rules that applies to the feature contribution data of the request.

9. The system of claim 8 , wherein the processing device is further to:

provide an indication that the particular rule requires modification.

10. The system of claim 9 , wherein the processing device is further to:

receive an updated version of the particular rule; and

update the set of security rules of the rule-based WAF with the updated version of the particular rule.

11. The system of claim 10 , wherein the processing device is further to:

receive a subsequent request; and

in response to determining, by the rule-based WAF, that a feature set of the subsequent request includes features matching the updated version of the particular rule, take one or more preventative actions.

12. The system of claim 7 , wherein to determine feature contribution data of an attack, the processing device is to:

determine a feature set of the attack, the feature set of the attack indicating features that the attack is comprised of;

determine, using the ML WAF, an attack probability of the attack based on the feature set; and

decompose the attack probability of the attack into a feature contribution made by each feature of the feature set of the attack to determine the feature contribution data of the attack.

13. The system of claim 12 , wherein a feature set of an attack comprises one or more of: a set of RCE keywords, a set of query string N-grams, a client from which the attack was generated, a number of slash characters in a uniform resource locator (URL) of the attack, a location from which the attack originated, and a length of the URL of the attack.

14. A non-transitory computer-readable medium having instructions stored thereon which, when executed by a processing device, cause the processing device to:

analyze a plurality of attacks detected by a machine learning (ML) web application firewall (WAF) to determine a set of attacks of the plurality of attacks that were not identified as an attack by a rule-based WAF;

for each attack of the set of attacks that were not identified as an attack by the rule-based WAF, determine feature contribution data of the attack;

group, using a clustering algorithm, the set of attacks into one or more clusters based on feature contribution data of each of the set of attacks; and

for each of the one or more clusters:

determining, by the processing device, one or more features that have a high feature contribution to an attack probability of at least a threshold number of attacks of the cluster, wherein the high feature contribution of the one or more features are identified either as

a) top contributing features in a number of split classification requests that is higher than a threshold number of the split classification requests, or

b) features whose contribution to the attack probability is higher than a mean contribution of all of the one or more features in the number of the split classification requests that is higher than the threshold number of the split classification requests;

identify, by the processing device, a new attack vector based on the one or more features of each attack in the cluster; and

generate a new rule for use by the rule-based WAF to identify the new attack vector.

15. The non-transitory computer-readable medium of claim 14 ,

wherein to determine feature contribution data of an attack, the processing device is to:

determine a feature set of the attack, the feature set of the attack indicating features that the attack is comprised of;

determine, using the ML WAF, an attack probability of the attack based on the feature set; and

decompose the attack probability of the attack into a feature contribution made by each feature of the feature set of the attack to determine the feature contribution data of the attack.

16. The non-transitory computer-readable medium of claim 14 ,

wherein to generate the new rule, the processing device is to:

generate a rule indicating that a request having features matching the determined one or more features corresponds to an attack.

17. The non-transitory computer-readable medium of claim 15 ,

wherein a feature set of an attack comprises one or more of: a set of RCE keywords, a set of query string N-grams, a client from which the attack was generated, a number of slash characters in a uniform resource locator (URL) of the attack, a location from which the attack originated, and a length of the URL of the attack.

18. The non-transitory computer-readable medium of claim 14 ,

wherein the processing device is further to:

update a set of security rules of the rule-based WAF with the new rule generated for each of the one or more clusters.