Bot detection and mitigation using dynamic web flows built via machine learning
An overlay network bot detection service is augmented to include a content generation service that dynamically generates dummy web pages that are served (along with real site content) to a requesting user, This content is built using machine learning models trained on a target website's content, or that otherwise leverage generative AI to create site content that mimics the site's real content. The generated content is preferably built dynamically during an actual interaction session with the requesting user, is designed to “look” and “feel” like actual content of the website, and inclusion of the content acts to trap a requesting user's browser in one or more non-productive (fake) navigation loops within the site. This facilitates the overall bot detection because such content and such loops are not actually part of the real site, and thus the navigation of these unproductive pages is highly indicative of bot activity.
1 . A method of protecting a website, comprising:
during an interaction with the website initiated by a requesting client:
serving a set of one or more first pages that constitute actual content of the website;
selectively serving a set of one or more second pages that differ from the set of one or more first pages in that the one or more second pages do not constitute the actual content of the website but are configured to appear to the requesting client as though the one or more second pages do constitute the actual content of the website, the one or more second pages having been built dynamically during the interaction and based at least in part on one or more machine learning content models associated with the website; and
based at least in part on receiving telemetry indicating navigation by the requesting client through the set of one or more second pages, characterizing the requesting client as a bot; and
taking a given mitigation action against the requesting client to protect the website.
2 . The method as described in claim 1 wherein the one or more machine learning content models comprises a set of models, wherein each of the set of models is trained on a category of content.
3 . The method as described in claim 2 , wherein the category of content is one of: text, images, page templates, and site metadata associated with site navigation.
4 . The method as described in claim 1 , wherein at least one second page of the set of one or more second pages has a page design that is generated based at least in part on a measure of visual similarity to at least a portion of a first page of the set of one or more first pages.
5 . The method as described in claim 1 , further including the one or more machine learning content models are trained out-of-band relative to the interaction.
6 . The method as described in claim 1 , wherein at least one machine learning content model of the one or more machine learning content models is built using generative-AI.
7 . The method as described in claim 1 , wherein at least one machine learning content model of the one or more machine learning content models is a neural network.
8 . The method as described in claim 7 , wherein the neural network is a graph neural network (GNN).
9 . The method as described in claim 1 , wherein at least one machine learning content model of the one or more machine learning content models is a model that has been updated out-of-band relative to the interaction.
10 . The method as described in claim 1 , wherein at least one machine learning content model of the one or more machine learning content models is trained on content of the website.
11 . The method as described in claim 1 , wherein the one or more first pages and the one or more second pages are served by an edge server of an overlay network.
12 . The method as described in claim 11 , wherein the given mitigation action is determined by a bot detection service associated with the overlay network.
13 . The method as described in claim 1 , wherein the requesting client is a page scrapper.
14 . The method as described in claim 1 , further including the website is configured for use without a captcha.
15 . The method as described in claim 1 , wherein the given mitigation action is one of: blocking an action requested by the requesting client, logging the interaction, issuing a notification, and tar-pitting or sand-boxing the requesting client.
16 . An apparatus configured to protect a website, comprising: one or more hardware processors;
computer memory holding computer program code executed by the one or more hardware processors, the computer program code configured during an interaction with the website initiated by a requesting client to:
serve a set of one or more first pages that constitute actual content of the website;
selectively serve a set of one or more second pages that differ from the set of one or more first pages in that the one or more second pages do not constitute the actual content of the website but are configured to appear to the requesting client as though the one or more second pages do constitute the actual content of the website, the one or more second pages having been built dynamically during the interaction and based at least in part on one or more machine learning content models associated with the website;
based at least in part on receiving telemetry indicating navigation by the requesting client through the set of one or more second pages, characterize the requesting client as a bot; and
take a given mitigation action against the requesting client to protect the website.
17 . The apparatus as described in claim 16 , wherein the given mitigation action is one of: blocking an action requested by the requesting client, logging the interaction, issuing a notification, and tar-pitting or sand-boxing the requesting client.
18 . A computer program product in a non-transitory computer-readable medium, the computer program product comprising computer program code executable by one or more hardware processors to protect a website, the computer program code configured during an interaction with the website initiated by a requesting client to:
serve a set of one or more first pages that constitute actual content of the website;
selectively serve a set of one or more second pages that differ from the set of one or more first pages in that the one or more second pages do not constitute the actual content of the website but are configured to appear to the requesting client as though the one or more second pages do constitute the actual content of the website, the one or more second pages having been built dynamically during the interaction and based at least in part on one or more machine learning content models associated with the website;
based at least in part on receiving telemetry indicating navigation by the requesting client through the set of one or more second pages, characterize the requesting client as a bot; and
take a given mitigation action against the requesting client to protect the website.
19 . The computer program product as described in claim 18 , wherein the given mitigation action is one of: blocking an action requested by the requesting client, logging the interaction, issuing a notification, and tar-pitting or sand-boxing the requesting client.
20 . The computer program product as described in claim 18 , wherein the given mitigation action is taken at an edge server of an overlay network.