Granted Patent B1

US 12,273,392 · App. 18/669,980 · Granted Apr 8, 2025

Inventors: Siying Yang (Saratoga, CA); Krishna Narayanaswamy (Saratoga, CA)

Assignee: Netskope, Inc.

H04L63/20H04L63/1416

Abstract

Claims (106)

1. A method, comprising:

intercepting, with a forward proxy at a network security system interposed on a network between client devices and hosted services, a request transmitted from a first client device of the client devices to a first hosted service of the hosted services, wherein the hosted services comprise a plurality of generative artificial intelligence applications and a plurality of other hosted applications;

determining, by the network security system, the first hosted service is a first generative artificial intelligence application of the plurality of generative artificial intelligence applications and determining the request comprises a generative artificial intelligence prompt for submission to the first generative artificial intelligence application;

based on the determining, classifying the generative artificial intelligence prompt with a machine learning model classifier trained to receive requests and classify the generative artificial intelligence prompts of the requests directed to any of the plurality of generative artificial intelligence applications as one of a benign prompt, an injection attack prompt, and an uploaded files prompt; and

applying, by the network security system, a security policy to the request based on the classification of the generative artificial intelligence prompt.

2. The method of claim 1 , wherein the applying the security policy comprises:

in response to classifying the generative artificial intelligence prompt as the uploaded files prompt:

extracting, by the network security system, one or more files from the request;

scanning, by the network security system, the one or more files for sensitive information;

scanning, by the network security system, the request for sensitive information; and

applying, by the network security system, a second security policy to the request based on the scanning the one or more files, the scanning the request, or a combination.

3. The method of claim 2 , wherein the applying the second security policy to the request comprises:

increasing, by the network security system, a risk score associated with a user account associated with the request based on the classifying the generative artificial intelligence prompt as the uploaded files prompt, a result of the scanning the one or more files, a result of the scanning the request, or a combination.

4. The method of claim 1 , wherein the applying the security policy comprises:

in response to classifying the generative artificial intelligence prompt as the injection attack prompt:

blocking, by the network security system, transmission of the request to the first hosted service.

5. The method of claim 1 , wherein the applying the security policy comprises:

in response to classifying the generative artificial intelligence prompt as the injection attack prompt:

increasing, by the network security system, a risk score associated with a user account associated with the request.

6. The method of claim 1 , further comprising:

in response to classifying the generative artificial intelligence prompt as the benign prompt:

scanning, by the network security system, the request for sensitive information; and

applying, by the network security system, a second security policy to the request based on the scanning.

7. The method of claim 6 , wherein the applying the second security policy to the request comprises:

based on a result of the scanning, modifying, by the network security system, a risk score associated with a user account associated with the request.

8. The method of claim 1 , wherein the determining the first hosted service is the first generative artificial intelligence application comprises:

comparing a Uniform Resource Locator (URL) of the first hosted service with a list of URLs comprising URLs for at least a subset of the plurality of generative artificial intelligence applications.

9. The method of claim 1 , further comprising:

intercepting, at the network security system, a plurality of prior requests prior to intercepting the request, wherein the plurality of prior requests are transmitted from one or more of the client devices to the first hosted service;

for each prior request of the plurality of prior requests:

comparing, by the network security system, a Uniform Resource Locator (URL) of the first hosted service with a list of URLs comprising URLs for at least a subset of the plurality of generative artificial intelligence applications,

in response to not finding the URL of the first hosted service on the list of URLs, classifying, by the network security system, the request as one of suspected request and not suspected request, and

in response to classifying the request as the suspected request, increasing, by the network security system, a score of the first hosted service; and

in response to the score of the first hosted service exceeding a threshold value, adding, by the network security system, the URL of the first hosted service to the list of URLs.

10. The method of claim 1 , wherein the applying the security policy comprises:

modifying, by the network security system, a risk score associated with a user account associated with the request based on the classification of the generative artificial intelligence prompt; and

applying, by the network security system, a second security policy to the request based on the modified risk score.

11. The method of claim 1 , further comprising:

transmitting, by the network security system, the request to the first hosted service;

intercepting, at the network security system, a response from the first hosted service;

classifying the response with a second machine learning model classifier trained to classify responses from any of the plurality of generative artificial intelligence applications as one of a benign response, a leaked system prompt response, a leaked file response, and a leaked training data response; and

applying, by the network security system, a second security policy to the response based on the classification of the response.

12. The method of claim 11 , wherein the applying the second security policy comprises:

in response to classifying the response as the benign response:

scanning, by the network security system, the response for sensitive information; and

applying, by the network security system, a third security policy to the response based on the scanning.

13. The method of claim 12 , further comprising:

based on a result of the scanning the response, modifying, by the network security system, a risk score associated with the first generative artificial intelligence application.

14. The method of claim 11 , wherein the applying the second security policy comprises:

in response to classifying the response as the leaked system prompt response:

blocking transmission of the response to the first client device; and

increasing, by the network security system, a risk score associated with the first generative artificial intelligence application.

15. The method of claim 11 , wherein the applying the second security policy comprises:

in response to classifying the response as the leaked file response:

extracting, by the network security system, one or more files from the response;

scanning, by the network security system, the one or more files for sensitive information;

scanning, by the network security system, the response for sensitive information;

applying, by the network security system, a third security policy to the response based on the scanning the one or more files, the scanning the response, or a combination; and

increasing, by the network security system, a risk score associated with the first generative artificial intelligence application based on the classifying the response as the leaked file response, a result of the scanning the one or more files, a result of the scanning the response, or a combination.

16. The method of claim 11 , wherein the applying the second security policy comprises:

in response to classifying the response as the leaked training data response:

scanning, by the network security system, the response for sensitive information;

applying, by the network security system, a third security policy to the response based on the scanning; and

increasing, by the network security system, a risk score associated with the first generative artificial intelligence application based on the classifying the response as the leaked training data response, a result of the scanning, or a combination.

17. The method of claim 11 , wherein the applying the second security policy comprises:

modifying, by the network security system, a risk score associated with the first generative artificial intelligence application based on the classification of the response;

in response to the risk score associated with the first generative artificial intelligence application exceeding a threshold value, adding, by the network security system, a Uniform Resource Locator (URL) of the first generative artificial intelligence application to a blacklist; and

blocking, by the network security system, future requests intended for the first generative artificial intelligence application based on finding the URL in the blacklist.

18. A network security system interposed on a network between client devices and hosted services, the network security system comprising:

a forward proxy configured to:

intercept requests transmitted from the client devices to the hosted services;

a filtering component configured to:

for each request of the requests:

determine whether the hosted service of the hosted services identified by the respective request is one of a plurality of generative artificial intelligence applications,

route the respective request to a classifier based on a determination that the respective request is directed to one of the plurality of generative artificial intelligence applications, and

route the respective request to a security policy enforcement component based on a determination that the respective request is not directed to one of the plurality of generative artificial intelligence applications;

the classifier trained to:

for each request routed to the classifier:

classify a generative artificial intelligence prompt of the respective request as one of a benign prompt, an injection attack prompt, and an uploaded files prompt, and

provide the classification to the security policy enforcement component; and

the security policy enforcement component configured to:

apply a security policy to the classified requests based on the classification.

19. The network security system of claim 18 , wherein the security policy enforcement component is further configured to:

in response to the generative artificial intelligence prompt of the request being classified as the uploaded files prompt:

extract one or more files from the request,

scan the one or more files for sensitive information,

scan the request for sensitive information,

apply a second security policy to the request based on the scan of the one or more files, the scan of the request, or a combination, and

increase a risk score associated with a user account associated with the request;

in response to the generative artificial intelligence prompt of the request being classified as the injection attack prompt:

block transmission of the request to the hosted service of the hosted services identified by the respective request, and

increase the risk score associated with the user account associated with the request; and

in response to the generative artificial intelligence prompt of the request response being classified as the benign prompt:

scan the request for sensitive information,

apply a third security policy to the request based on the scan of the request, and

modify the risk score associated with the user account associated with the request based on the scan of the request.

20. The network security system of claim 18 , wherein:

the security policy enforcement component is further configured to:

transmit the request to the hosted service of the hosted services identified by the respective request based on a result of applying the security policy;

the forward proxy is further configured to:

intercept responses to the classified requests; and

the network security system further comprising:

a response classifier trained to:

classify the responses as one of a benign response, a leaked system prompt response, a leaked file response, and a leaked training data response; and

a response security enforcement component configured to:

apply a second security policy to the response based on the classification of the response.

Assignments (1)

References Cited (140)

Cited By (2)

US 12,688,280 US 12,726,509