Chatbot response system
A computer-implemented method and system for responding to user posts containing images with relevant image responses during conversation between a user and a chatbot. The system receives an image post from the user and generates a description of the image using an image-to-text model. User intent is determined based on the image and description. If responding with an image is appropriate based on the user intent, the system generates a prompt using the image description and passes it to a text generation model to create an image description and caption. The image description and caption are used to synthesize a new image. The resulting image and caption are packaged into a post that is provided as a response to the user. The system uses machine learning pipelines and models to analyze images, detect inappropriate content, classify user intent, generate text, and synthesize images.
1 . A computer-implemented method comprising:
receiving, by one or more processors, from a user using a user device, a user interaction system post comprising an image;
generating, by the one or more processors, a description of the user interaction system post using an image to text model;
determining by the one or more processors, using the user interaction system post, that the user interaction system post includes inappropriate content;
in response to determining the user interaction system post includes inappropriate content, modifying, by the one or more processors, the description of the user interaction system post to remove the inappropriate content;
determining, by the one or more processors, a user intent based on the user interaction system post and the description of the user interaction system post as modified;
determining, by the one or more processors, to respond with a chatbot interaction system post using the user intent;
in response to determining to respond with a chatbot interaction system post, generating, by the one or more processors, a prompt using the description of the user interaction system post as modified;
generating, by the one or more processors, an image description and a caption using the prompt;
generating, by the one or more processors, the chatbot interaction system post using the image description and the caption; and
providing, by the one or more processors, the chatbot interaction system post to the user via the user device.
2 . The computer-implemented method of claim 1 , further comprising: generating, by the one or more processors, the prompt by appending one or more style instructions to the user interaction system post description.
3 . The computer-implemented method of claim 2 , wherein the one or more style instructions comprise instructions to generate the chatbot interaction system post image in an interaction system post style.
4 . The computer-implemented method of claim 1 , further comprising:
generating, by the one or more processors, an inappropriate content score for the user interaction system post using the user interaction system post and a sanitation model.
5 . The computer-implemented method of claim 4 , further comprising:
in response to detecting, by the one or more processors, that the inappropriate content score exceeds a threshold inappropriate content score value, providing, by the one or more processors, a text response indicating the user interaction system post cannot be processed.
6 . The computer-implemented method of claim 1 , further comprising:
randomly determining, by the one or more processors, whether to respond with a text response or the chatbot interaction system post.
7 . A machine, comprising:
one or more processors; and
one or more memories storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising:
receiving from a user using a user device, a user interaction system post comprising an image;
generating a description of the user interaction system post using an image to text model;
determining, using the user interaction system post, that the user interaction system post includes inappropriate content;
in response to determining the user interaction system post includes inappropriate content modifying the description of the user interaction system post to remove the inappropriate content;
determining a user intent based on the user interaction system post and the description of the user interaction system post as modified;
determining to respond with a chatbot interaction system post using the user intent;
in response to determining to respond with a chatbot interaction system post, generating a prompt using the description of the user interaction system post as modified;
generating an image description and a caption using the prompt,
generating the chatbot interaction system post using the image description and the caption; and
providing the chatbot interaction system post to the user via the user device.
8 . The machine of claim 7 , wherein the operations further comprise: generating the prompt by appending one or more style instructions to the user interaction system post description.
9 . The machine of claim 8 , wherein the one or more style instructions comprise instructions to generate the chatbot interaction system post image in an interaction system post style.
10 . The machine of claim 7 , wherein the operations further comprise:
generating an inappropriate content score for the user interaction system post using the user interaction system post and a sanitation model.
11 . The machine of claim 10 , wherein the operations further comprise:
in response to detecting that the inappropriate content score exceeds a threshold inappropriate content score value, providing a text response indicating the user interaction system post cannot be processed.
12 . The machine of claim 7 , wherein the operations further comprise:
randomly determining whether to respond with a text response or the chatbot interaction system post.
13 . A machine-storage medium storing instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:
receiving from a user using a user device, a user interaction system post comprising an image;
generating a description of the user interaction system post using an image to text model;
determining, using the user interaction system post, that the user interaction system post includes inappropriate content;
in response to determining the user interaction system post includes inappropriate content, modifying the description of the user interaction system post to remove the inappropriate content;
determining a user intent based on the user interaction system post and the description of the user interaction system post as modified;
determining to respond with a chatbot interaction system post using the user intent;
in response to determining to respond with a chatbot interaction system post, generating a prompt using the description of the user interaction system post as modified;
generating an image description and a caption using the prompt;
generating the chatbot interaction system post using the image description and the caption; and
providing the chatbot interaction system post to the user via the user device.
14 . The machine-storage medium of claim 13 , wherein the operations further comprise:
generating the prompt by appending one or more style instructions to the user interaction system post description.
15 . The machine-storage medium of claim 14 , wherein the one or more style instructions comprise instructions to generate the chatbot interaction system post image in an interaction system post style.
16 . The machine-storage medium of claim 13 , wherein the operations further comprise:
generating an inappropriate content score for the user interaction system post using the user interaction system post and a sanitation model.
17 . The machine-storage medium of claim 16 , wherein the operations further comprise:
in response to detecting that the inappropriate content score exceeds a threshold inappropriate content score value, providing a text response indicating the user interaction system post cannot be processed.