IP Library › Granted Patent US 12689653

Granted Patent B2

US 12689653 · App. 18/790,780 · Granted Jul 21, 2026

Systems and methods for defending against prompt leakage attacks

Inventors: Divyansh Agarwal (San Francisco, CA); Alexander R. Fabbri (New York, NY); Philippe Laban (New York, NY); Shafiq Rayhan Joty (San Jose, CA); Chien-Sheng Wu (Mountain View, CA)

Assignee: Salesforce, Inc.

H04L63/1466 G06F16/3332

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 12689653
App. No.: 18/790,780
Granted: Jul 21, 2026
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

Embodiments described herein provide a method of sanitizing a user input. A system receives the user input, and may retrieve one or more documents from a database based on the user input. The system then generates, via a first neural network based language model, a sanitized version of the user input in response to a determination to sanitize based on at least one of the user input or the one or more documents. The system then generates, via a second neural network based language model, an output text based on a prompt, the one or more documents, and the sanitized version of the user input.

Claims (56)

1 . A method of protecting data privacy of a neural network system, the method comprising:

receiving, via a data interface, a user input;

retrieving one or more pre-stored contextual information artifacts from a database based on the user input;

determining, via a detection model, a probability value representing a likelihood that the user input contains a leak request by determining that the user input includes a semantic reiteration of a prior user input and a statement that a prior response was incorrect;

generating, via a second neural network based model, an output using the sanitized user input and the one or more retrieved contextual information artifacts, wherein the generating the output includes: transmitting, to a remote server with the second neural network based model, the one or more retrieved contextual information artifacts, and the sanitized user input; and receiving, from the remote server, the output.

2 . The method of claim 1 , wherein the threshold is adjustable automatically based on at least a user information or a prior determination by the detection model.

3 . The method of claim 1 , further comprising:

storing a list of sensitive document categories,

wherein the determining the sensitivity of the one or more retrieved contextual information artifacts is based on the one or more retrieved contextual information artifacts belonging to a category in the list of sensitive document categories.

4 . The method of claim 3 , wherein the list of sensitive document categories includes at least one of:

medical information;

intellectual property;

legal information;

financial information; or

personal identification information.

5 . The method of claim 1 , further comprising:

determining, via a third neural network based model, a subject matter of the user input,

wherein the likelihood that the user input contains a leak request is based on the subject matter.

6 . The method of claim 1 , further comprising:

storing a list of sensitive subject matter categories,

determining a subject matter of the user input,

wherein the likelihood that the user input contains a leak request is based on the subject matter of the user input belonging to a category in the list of sensitive subject matter categories.

7 . The method of claim 1 , wherein the determining includes:

determining one or more predetermined keywords are present in at least one of the user input or the one or more retrieved contextual information artifacts.

8 . The method of claim 1 , wherein the converting the user input to a sanitized user input includes:

transmitting, to a remote server with the first neural network based model, the user input; and

receiving, from the remote server, the sanitized user input.

9 . A system for protecting data privacy of a neural network system, the system comprising:

a memory that stores a plurality of processor executable instructions;

a communication interface that receives a training dataset including a plurality of input samples; and

one or more hardware processors that read and execute the plurality of processor-executable instructions from the memory to perform operations comprising:

receiving, via a data interface, a user input;

retrieving one or more pre-stored contextual information artifacts from a database based on the user input;

determining, via a detection model a probability value representing a likelihood that the user input contains a leak request by determining that the user input includes a semantic reiteration of a prior user input and a statement that a prior response was incorrect;

converting, via a first neural network based model in response to the probability value meeting or exceeding a threshold, the user input to a sanitized user input removing words relating to the leak request, wherein the converting the user input to a sanitized user input includes: transmitting, to a remote server with the first neural network based model, the user input; and receiving, from the remote server, the sanitized user input; and

generating, via a second neural network based model, an output based on using the sanitized user input and the one or more retrieved contextual information artifacts.

10 . The system of claim 9 , wherein the threshold is adjustable automatically based on at least a user information or a prior determination by the detection model.

11 . The system of claim 9 , the one or more hardware processors perform operations further comprising: