IP Library › Granted Patent US 12695713
Granted Patent B2
US 12695713 · App. 18/388,153 · Granted Jul 28, 2026

Chatbot response system

Inventors: Aliaksei Mikhailiuk (London, GB); Sergey Smetanin (London, GB); Pavel Savchenkov (London, GB); Hee Hun Kim (Los Angeles, CA); Tianxiang Gao (Bothell, WA); Neha Yadav (Seattle, WA); Bingqian Lu (Ontario, CA); Emily Ann Claudet (London, GB)
Assignee: Snap Inc.
H04L51/02G06F40/30G06V30/153H04L51/063
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12695713
App. No.
18/388,153
Granted
Jul 28, 2026
Kind
B2
Abstract

A computer-implemented method and system for responding to user posts containing images with relevant image responses during conversation between a user and a chatbot. The system receives an image post from the user and generates a description of the image using an image-to-text model. User intent is determined based on the image and description. If responding with an image is appropriate based on the user intent, the system generates a prompt using the image description and passes it to a text generation model to create an image description and caption. The image description and caption are used to synthesize a new image. The resulting image and caption are packaged into a post that is provided as a response to the user. The system uses machine learning pipelines and models to analyze images, detect inappropriate content, classify user intent, generate text, and synthesize images.

Claims (58)

1 . A computer-implemented method comprising:

receiving, by one or more processors, from a user using a user device, a user interaction system post comprising an image;

generating, by the one or more processors, a description of the user interaction system post using an image to text model;

determining by the one or more processors, using the user interaction system post, that the user interaction system post includes inappropriate content;

in response to determining the user interaction system post includes inappropriate content, modifying, by the one or more processors, the description of the user interaction system post to remove the inappropriate content;

determining, by the one or more processors, a user intent based on the user interaction system post and the description of the user interaction system post as modified;

determining, by the one or more processors, to respond with a chatbot interaction system post using the user intent;

in response to determining to respond with a chatbot interaction system post, generating, by the one or more processors, a prompt using the description of the user interaction system post as modified;

generating, by the one or more processors, an image description and a caption using the prompt;

generating, by the one or more processors, the chatbot interaction system post using the image description and the caption; and

providing, by the one or more processors, the chatbot interaction system post to the user via the user device.

2 . The computer-implemented method of claim 1 , further comprising: generating, by the one or more processors, the prompt by appending one or more style instructions to the user interaction system post description.

3 . The computer-implemented method of claim 2 , wherein the one or more style instructions comprise instructions to generate the chatbot interaction system post image in an interaction system post style.

4 . The computer-implemented method of claim 1 , further comprising:

generating, by the one or more processors, an inappropriate content score for the user interaction system post using the user interaction system post and a sanitation model.

5 . The computer-implemented method of claim 4 , further comprising:

in response to detecting, by the one or more processors, that the inappropriate content score exceeds a threshold inappropriate content score value, providing, by the one or more processors, a text response indicating the user interaction system post cannot be processed.

6 . The computer-implemented method of claim 1 , further comprising:

randomly determining, by the one or more processors, whether to respond with a text response or the chatbot interaction system post.

7 . A machine, comprising:

one or more processors; and

one or more memories storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising:

receiving from a user using a user device, a user interaction system post comprising an image;

generating a description of the user interaction system post using an image to text model;

determining, using the user interaction system post, that the user interaction system post includes inappropriate content;

in response to determining the user interaction system post includes inappropriate content modifying the description of the user interaction system post to remove the inappropriate content;

determining a user intent based on the user interaction system post and the description of the user interaction system post as modified;

determining to respond with a chatbot interaction system post using the user intent;

in response to determining to respond with a chatbot interaction system post, generating a prompt using the description of the user interaction system post as modified;

generating an image description and a caption using the prompt,

generating the chatbot interaction system post using the image description and the caption; and

providing the chatbot interaction system post to the user via the user device.

8 . The machine of claim 7 , wherein the operations further comprise: generating the prompt by appending one or more style instructions to the user interaction system post description.

9 . The machine of claim 8 , wherein the one or more style instructions comprise instructions to generate the chatbot interaction system post image in an interaction system post style.

10 . The machine of claim 7 , wherein the operations further comprise:

generating an inappropriate content score for the user interaction system post using the user interaction system post and a sanitation model.

11 . The machine of claim 10 , wherein the operations further comprise:

in response to detecting that the inappropriate content score exceeds a threshold inappropriate content score value, providing a text response indicating the user interaction system post cannot be processed.

12 . The machine of claim 7 , wherein the operations further comprise:

randomly determining whether to respond with a text response or the chatbot interaction system post.

13 . A machine-storage medium storing instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

receiving from a user using a user device, a user interaction system post comprising an image;

generating a description of the user interaction system post using an image to text model;

determining, using the user interaction system post, that the user interaction system post includes inappropriate content;

in response to determining the user interaction system post includes inappropriate content, modifying the description of the user interaction system post to remove the inappropriate content;

determining a user intent based on the user interaction system post and the description of the user interaction system post as modified;

determining to respond with a chatbot interaction system post using the user intent;

in response to determining to respond with a chatbot interaction system post, generating a prompt using the description of the user interaction system post as modified;

generating an image description and a caption using the prompt;

generating the chatbot interaction system post using the image description and the caption; and

providing the chatbot interaction system post to the user via the user device.

14 . The machine-storage medium of claim 13 , wherein the operations further comprise:

generating the prompt by appending one or more style instructions to the user interaction system post description.

15 . The machine-storage medium of claim 14 , wherein the one or more style instructions comprise instructions to generate the chatbot interaction system post image in an interaction system post style.

16 . The machine-storage medium of claim 13 , wherein the operations further comprise:

generating an inappropriate content score for the user interaction system post using the user interaction system post and a sanitation model.

17 . The machine-storage medium of claim 16 , wherein the operations further comprise:

in response to detecting that the inappropriate content score exceeds a threshold inappropriate content score value, providing a text response indicating the user interaction system post cannot be processed.