Granted Patent B2

US 12,645,516 · App. 17/661,497 · Granted Jun 2, 2026

Inventors: Jonathan R. Azaria (Beit Gamliel, IL); Ori Nakar (Tel Aviv, IL); Matan Lion (Tel Aviv, IL)

Assignee: Imperva, Inc.

G06F9/547G06F16/9566

Abstract

Claims (101)

1 . A method performed by one or more computing devices to automatically discover endpoints of a web service, the one or more computing devices being connected between a web service server and one or more web service clients, the method comprising:

deploying a proxy between the one or more web service clients and the web service server to perform attack detection for traffic being sent between the one or more web service clients and the web service server, wherein performing the attack detection comprises:

obtaining a plurality of web service requests originated by the one or more web service clients to access the web service provided by the web service server;

determining levels of similarities between pairs of web service requests in the plurality of web service requests based on whether a pair of web service requests generate a same or similar response body in their corresponding web service responses;

grouping, by executing a clustering algorithm, the plurality of web service requests into a plurality of groups based on the levels of similarities, wherein the plurality of groups includes at least a first group and a second group, wherein each URL path in the first group exhibits higher similarity with other URL paths in the first group as compared to URL paths in the second group;

determining whether uniform resource locator (URL) paths of web service requests included in the first group include a parameter by analyzing the URL paths within the first group, wherein determining comprises applying a heuristic to a tree data structure representing the URL paths of the web service requests in the first group to determine common pattern among the URL paths of web service requests included in the first group, wherein the tree data structure comprises a root node representing the web service, and each node stemming from the root node represents a URL part of the URL paths;

responsive to a determination that the URL paths of the web service requests included in the first group do not include the parameter, determining that each of the URL paths of the web service requests included in the first group refers to a separate end point of the web service;

determining whether URL paths of web service requests included in the second group include a parameter;

responsive to a determination that the URL paths of the web service requests included in the second group include the parameter:

determining that the URL paths of the web service requests included in the second group refer to a single endpoint of the web service, and

determining a location of the parameter and a type of the parameter in the URL paths;

generating a URL path pattern representing the location of the parameter and the type of the parameter for the URL paths of the web service requests included in the second group;

updating a profile of the web service to indicate that the URL paths of the web service requests included in the first group each represent an endpoint of the web service and the URL path pattern refers to an endpoint of the web service; and

performing the attack detection based on the updated profile, wherein performing the attack detection based on the updated profile comprises discovering the end points of the web service accurately as compared to conventional endpoint discovery solutions, wherein the end points of the web service are discovered by updating the profile to include the URL paths of the web service requests and/or the URL path pattern that are determined to be referred to the endpoints of the web service.

2 . The method of claim 1 , wherein the levels of similarities are further determined based on one or more of:

whether URL paths of a pair of web service requests include a same URL part in a same location,

whether URL paths of a pair of web service requests include URL parts having a same length or type in a same location, and

whether URL paths of a pair of web service requests include a same number of URL parts.

3 . The method of claim 2 , wherein the levels of similarities are further determined based on one or more of:

whether a pair of web service requests include a same or similar post body or query string,

whether a pair of web service requests generate a same or similar parameter in their corresponding web service responses,

whether a pair of web service requests or their corresponding web service responses include a same or similar header, and

whether a pair of web service requests include a same or similar parameter.

4 . The method of claim 1 , wherein the clustering algorithm is a density-based spatial clustering of applications with noise (DBSCAN) algorithm.

5 . The method of claim 1 , wherein the determination that the URL paths of the web service requests included in the second group include the parameter is based on one or more of: a number of distinct URL paths in the URL paths of the web service requests included in the second group, a ratio of the number of distinct URL paths to a number of web service requests included in the second group, and an existence of a predefined type of value in the URL paths of the web service requests included in the second group.

6 . The method of claim 1 , wherein the determination that the URL paths of the web service requests included in the first group do not include the parameter is based on a determination that the first group stays relatively consistent over multiple iterations of grouping.

7 . A method performed by one or more computing devices to automatically discover endpoints of a web service, the one or more computing devices being connected between a web service server and one or more web service clients, the method comprising:

deploying a proxy between the one or more web service clients and the web service server to perform attack detection for traffic being sent between the one or more web service clients and the web service server, wherein performing the attack detection comprises:

obtaining a plurality of web service requests originated by the one or more web service clients to access the web service provided by the web service server;

generating a tree data structure representing uniform resource locator (URL) paths of the plurality of web service requests, wherein the tree data structure comprises a root node representing the web service, and each node stemming from the root node represents a URL part of the URL paths;

applying a heuristic to the tree data structure to group URL paths that include a parameter;

pruning the tree data structure by merging nodes based on a result of applying the heuristic to generate a pruned tree data structure;

generating a distance matrix based on the pruned tree data structure, the distance matrix indicating distances between the URL paths of the plurality of web service requests;

grouping, by executing a clustering algorithm, URL paths into a plurality of groups based on the distance matrix, wherein each URL path in a group of the plurality of groups exhibits higher similarity with other URL paths within the same group than as compared to URL paths in other groups of the plurality of groups;

for each of the plurality of groups, attempting to extract one or more endpoints from the group and if the attempt is successful, updating a profile of the web service to indicate the one or more endpoints, otherwise if the attempt is not successful, designating the group as a leftover group; and

performing the attack detection based on the updated profile, wherein performing the attack detection based on the updated profile comprises discovering the end points of the web service accurately as compared to conventional endpoint discovery solutions, wherein the end points of the web service are discovered by updating the profile to include the URL paths of the web service requests and/or the URL path pattern that are determined to be referred to the endpoints of the web service.

8 . The method of claim 7 , further comprising:

obtaining a second plurality of web service requests originated by the one or more web service clients to access the web service provided by the web service server;

grouping URL paths of the second plurality of web service requests that do not match endpoints indicated by the profile of the web service and URL paths included in groups designated as leftover groups into a second plurality of groups; and

for each of the second plurality of groups, attempting to extract one or more endpoints from the group and if the attempt is successful, updating the profile of the web service to indicate the one or more endpoints, otherwise if the attempt is not successful, designating the group as a leftover group.

9 . The method of claim 8 , wherein each URL path included in a group is designated as referring to an endpoint of the web service if the group stays relatively consistent over multiple iterations of grouping.

10 . A set of one or more non-transitory machine-readable storage media storing instructions which, when executed by one or more processors of one or more computing devices, causes the one or more computing devices to perform operations for automatically discovering endpoints of a web service, the one or more computing devices being connected between a web service server and one or more web service clients, the operations comprising:

deploying a proxy between the one or more web service clients and the web service server to perform attack detection for traffic being sent between the one or more web service clients and the web service server, wherein performing the attack detection comprises:

obtaining a plurality of web service requests originated by the one or more web service clients to access the web service provided by the web service server;

determining levels of similarities between pairs of web service requests in the plurality of web service requests based on whether a pair of web service requests generate a same or similar response body in their corresponding web service responses;

grouping, by executing a clustering algorithm, the plurality of web service requests into a plurality of groups based on the levels of similarities, wherein the plurality of groups includes at least a first group and a second group, wherein each URL path in the first group exhibits higher similarity with other URL paths in the first group as compared to URL paths in the second group;

determining whether uniform resource locator (URL) paths of web service requests included in the first group include a parameter by analyzing the URL paths within the first group, wherein determining comprises applying a heuristic to a tree data structure representing the URL paths of the web service requests in the first group to determine common pattern among the URL paths of web service requests included in the first group, wherein the tree data structure comprises a root node representing the web service, and each node stemming from the root node represents a URL part of the URL paths;

responsive to a determination that the URL paths of the web service requests included in the first group do not include the parameter, determining that each of the URL paths of the web service requests included in the first group refers to a separate end point of the web service;

determining whether URL paths of web service requests included in the second group include a parameter;

responsive to a determination that the URL paths of the web service requests included in the second group include the parameter:

determining that the URL paths of the web service requests included in the second group refer to a single endpoint of the web service, and

determining a location of the parameter and a type of the parameter in the URL paths;

generating a URL path pattern representing the location of the parameter and the type of the parameter for the URL paths of the web service requests included in the second group;

updating a profile of the web service to indicate that the URL paths of the web service requests included in the first group each represent an endpoint of the web service and the URL path pattern refers to an endpoint of the web service; and

performing the attack detection based on the updated profile, wherein performing the attack detection based on the updated profile comprises discovering the end points of the web service accurately as compared to conventional endpoint discovery solutions, wherein the end points of the web service are discovered by updating the profile to include the URL paths of the web service requests and/or the URL path pattern that are determined to be referred to the endpoints of the web service.

11 . The set of one or more non-transitory machine-readable storage media of claim 10 , wherein the levels of similarities are further determined based on one or more of:

whether URL paths of a pair of web service requests include a same URL part in a same location,

whether URL paths of a pair of web service requests include URL parts having a same length or type in a same location, and

whether URL paths of a pair of web service requests include a same number of URL parts.

12 . The set of one or more non-transitory machine-readable storage media of claim 10 , wherein the determination that the URL paths of the web service requests included in the second group include the parameter is based on one or more of: a number of distinct URL paths in the URL paths of the web service requests included in the second group, a ratio of the number of distinct URL paths to a number of web service requests included in the second group, and an existence of a predefined type of value in the URL paths of the web service requests included in the second group.

13 . A set of one or more non-transitory machine-readable storage media storing instructions which, when executed by one or more processors of one or more computing devices, causes the one or more computing devices to perform operations for automatically discovering endpoints of a web service, the one or more computing devices being connected between a web service server and one or more web service clients, the operations comprising:

deploying a proxy between the one or more web service clients and the web service server to perform attack detection for traffic being sent between the one or more web service clients and the web service server, wherein performing the attack detection comprises:

obtaining a plurality of web service requests originated by the one or more web service clients to access the web service provided by the web service server;

generating a tree data structure representing uniform resource locator (URL) paths of the plurality of web service requests, wherein the tree data structure comprises a root node representing the web service, and each node stemming from the root node represents a URL part of the URL paths;

applying a heuristic to the tree data structure to group URL paths that include a parameter;

pruning the tree data structure by merging nodes based on a result of applying the heuristic to generate a pruned tree data structure;

generating a distance matrix based on the pruned tree data structure, the distance matrix indicating distances between the URL paths of the plurality of web service requests;

grouping, by executing a clustering algorithm, URL paths into a plurality of groups based on the distance matrix, wherein each URL path in a group of the plurality of groups exhibits higher similarity with other URL paths within the same group as compared to URL paths in other groups of the plurality of groups;

for each of the plurality of groups, attempting to extract one or more endpoints from the group and if the attempt is successful, updating a profile of the web service to indicate the one or more endpoints, otherwise if the attempt is not successful, designating the group as a leftover group; and

performing the attack detection based on the updated profile, wherein performing the attack detection based on the updated profile comprises discovering the end points of the web service accurately as compared to conventional endpoint discovery solutions, wherein the end points of the web service are discovered by updating the profile to include the URL paths of the web service requests and/or the URL path pattern that are determined to be referred to the endpoints of the web service.

14 . The set of one or more non-transitory machine-readable storage media of claim 13 , wherein the operations further comprise:

obtaining a second plurality of web service requests originated by the one or more web service clients to access the web service provided by the web service server;

grouping URL paths of the second plurality of web service requests that do not match endpoints indicated by the profile of the web service and URL paths included in groups designated as leftover groups into a second plurality of groups; and

for each of the second plurality of groups, attempting to extract one or more endpoints from the group and if the attempt is successful, updating the profile of the web service to indicate the one or more endpoints, otherwise if the attempt is not successful, designating the group as a leftover group.

15 . The set of one or more non-transitory machine-readable storage media of claim 14 , wherein each URL path included in a group is designated as referring to an endpoint of the web service if the group stays relatively consistent over multiple iterations of grouping.

16 . A computing device configured to detect endpoints of a web service, the computing device being connected between a web service server and one or more web service clients, the computing device comprising:

one or more processors; and

a non-transitory machine-readable storage medium having instructions stored therein, which when executed by the one or more processors, causes the computing device to deploy a proxy between the one or more web service clients and the web service server to perform attack detection for traffic being sent between the one or more web service clients and the web service server, wherein, to perform the attack detection, the computing device is caused to:

obtain a plurality of web service requests originated by the one or more web service clients to access the web service provided by the web service server;

determine levels of similarities between pairs of web service requests in the plurality of web service requests based on whether a pair of web service requests generate a same or similar response body in their corresponding web service responses;

group, by executing a clustering algorithm, the plurality of web service requests into a plurality of groups based on the levels of similarities, wherein the plurality of groups includes at least a first group and a second group, wherein each URL path in the first group exhibits higher similarity with other URL paths in the first group than as compared to URL paths in the second group;

determine whether uniform resource locator (URL) paths of web service requests included in the first group include a parameter by analyzing the URL paths within the first group, wherein the computing device is caused to apply a heuristic to a tree data structure representing the URL paths of the web service requests in the first group to determine common pattern among the URL paths of web service requests included in the first group, wherein the tree data structure comprises a root node representing the web service, and each node stemming from the root node represents a URL part of the URL paths;

responsive to a determination that the URL paths of the web service requests included in the first group do not include the parameter, determine that each of the URL paths of the web service requests included in the first group refers to a separate end point of the web service;

determine whether URL paths of web service requests included in the second group include a parameter;

responsive to a determination that the URL paths of the web service requests included in the second group include the parameter:

determine that the URL paths of the web service requests included in the second group refer to a single endpoint of the web service, and

determine a location of the parameter and a type of the parameter in the URL paths;

generate a URL path pattern representing the location of the parameter and the type of the parameter for the URL paths of the web service requests included in the second group;

update a profile of the web service to indicate that the URL paths of the web service requests included in the first group each represent an endpoint of the web service and the URL path pattern refers to an endpoint of the web service; and

perform the attack detection based on the updated profile, wherein, to perform the attack detection based on the updated profile, the computing device is caused to discover the end points of the web service accurately as compared to conventional endpoint discovery solutions, wherein the end points of the web service are discovered by updating the profile to include the URL paths of the web service requests and/or the URL path pattern that are determined to be referred to the endpoints of the web service.

17 . A computing device configured to detect endpoints of a web service, the computing device being connected between a web service server and one or more web service clients, the computing device comprising:

one or more processors; and

a non-transitory machine-readable storage medium having instructions stored therein, which when executed by the one or more processors, causes the computing device to deploy a proxy between the one or more web service clients and the web service server to perform attack detection for traffic being sent between the one or more web service clients and the web service server, wherein, to perform the attack detection, the computing device is caused to:

obtain a plurality of web service requests originated by the one or more web service clients to access the web service provided by the web service server;

generate a tree data structure representing uniform resource locator (URL) paths of the plurality of web service requests, wherein the tree data structure comprises a root node representing the web service, and each node stemming from the root node represents a URL part of the URL paths;

apply a heuristic to the tree data structure to group URL paths that include a parameter;

prune the tree data structure by merging nodes based on a result of applying the heuristic to generate a pruned tree data structure;

generate a distance matrix based on the pruned tree data structure, the distance matrix indicating distances between the URL paths of the plurality of web service requests;

group, by executing a clustering algorithm, URL paths into a plurality of groups based on the distance matrix, wherein each URL path in a group of the plurality of groups exhibits higher similarity with other URL paths within the same group as compared to URL paths in other groups of the plurality of groups;

for each of the plurality of groups, attempt to extract one or more endpoints from the group and if the attempt is successful, update a profile of the web service to indicate the one or more endpoints, otherwise if the attempt is not successful, designate the group as a leftover group; and

perform the attack detection based on the updated profile, wherein, to perform the attack detection based on the updated profile, the computing device is caused to discover the end points of the web service accurately as compared to conventional endpoint discovery solutions, wherein the end points of the web service are discovered by updating the profile to include the URL paths of the web service requests and/or the URL path pattern that are determined to be referred to the endpoints of the web service.

Assignments (1)

Continuity (2)

References Cited (13)