User feedback for specific portions of responses generated using a large language model (LLM)
Implementations relate to providing a user feedback mechanism that enables a user to provide feedback towards one or more specific portions of a response. The response can be generated based on processing of a user query using a generative model such as a large language model (LLM). The one or more specific portions can be a textual portion that includes textual content, and/or a media content portion that include media content such as one or more images, one or more videos, one or more audio pieces, etc. The feedback towards one or more of the specific portions of the response can be utilized in training or fine-tuning the generative model (or an additional generative model) via approaches such as supervised training or reinforced learning.
1 . A method implemented using one or more processors, the method comprising:
receiving a user query, the user query being received via a user interface of a client device;
generating, using a generative model, a generative response that is responsive to the user query and that includes at least media content to be rendered and one or more selectable elements to be rendered with respect to media content,
wherein each of the one or more selectable elements are selectable and correspond to a respective type of user feedback that evaluates the media content with respect to the user query, and
wherein the one or more selectable elements includes a given selectable element that corresponds to negative user feedback;
causing the media content to be rendered at the user interface;
causing the one or more selectable elements to be rendered at the user interface and with respect to the media content;
in response to receiving user input that selects the given selectable element, of the one or more selectable elements, that corresponds to the negative user feedback:
determining, based on the received user input that selects the given selectable element, user feedback that evaluates the media content with respect to the user query, wherein determining the user feedback that evaluates the media content with respect to the user query comprises:
causing a feedback window to be rendered at the user interface,
wherein the feedback window is rendered in response to receiving the user input that selects the given selectable element, and
wherein the feedback window includes one or more selectable options each displaying a corresponding description that classifies the negative user feedback;
receiving additional user input that selects one of the one or more selectable options; and
determining a classification of the negative user feedback based on the additional user input; and
training or fine-tuning the generative model based on at least the user feedback that evaluates the media content with respect to the user query, wherein training or fine-tuning the generative model based on the user query and the user feedback that evaluates the media content with respect to the user query comprises:
in response to determining that the classification of the negative user feedback is a first classification:
reinforcing the generative model to not output a tag or prompt utilized to obtain the media content in response to the user query, and
in response to determining that the classification of the negative user feedback is a second classification different from the first classification:
rewriting an input prompt that is processed as input using the generative model; and
training or fine-tuning the generative model using the rewritten input prompt.
2 . The method of claim 1 , wherein the media content is an image or a video.
3 . The method of claim 1 , wherein the media content is rendered as part of a multi-modal response that is generated as the generative response and responsive to the user query, and wherein the multi-modal response further includes textual content.
4 . The method of claim 1 , wherein the one or more selectable elements includes an additional given selectable element that corresponds to positive user feedback.
5 . The method of claim 4 , further comprising:
in response to receiving user input that selects the given additional selectable element, of the one or more selectable elements, that corresponds to the positive user feedback:
generating a training instance to: include an input prompt that includes the user query as a training instance input, and include at least a response having the tag or the prompt, that was utilized to obtain the media content, as training instance output.
6 . The method of claim 5 , wherein training or fine-tuning the generative model based on the user query and the user feedback that evaluates the media content with respect to the user query further comprises:
processing the training instance input corresponding to the input prompt that includes the user query, using the generative model, to generate a predicted model output that includes at least a predicted tag or predicted prompt,
comparing the predicted model output with the training instance output; and
training or fine-tuning the generative model based on comparing the predicted model output with the training instance output.
7 . The method of claim 1 , wherein the first classification indicates that the media content is unnecessary or not a good quality, and wherein the second classification indicates that the media content is unrelated to the user query.
8 . The method of claim 1 , wherein the feedback window further includes an input field to receive customized user feedback.
9 . The method of claim 1 , wherein the media content is non-generative media content that is obtained based on the tag generated using the generative model.
10 . The method of claim 9 , further comprising:
generating, based on the tag, a query for the media content; and
obtaining, based on submitting the query for the media content to a media content search system, the media content.
11 . The method of claim 1 , wherein the media content is generative media content that is obtained based on the prompt, wherein the prompt is generated using the generative model.
12 . The method of claim 11 , further comprising:
submitting, to an additional generative model that is in addition to the generative model, the prompt; and
obtaining, based on submitting the prompt to the additional generative model, the media content.
13 . A system including one or more processors and memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations of:
receiving a user query, the user query being received via a user interface of a client device;
generating, using a generative model, a generative response that is responsive to the user query and that includes at least media content to be rendered and one or more selectable elements to be rendered with respect to media content,
wherein each of the one or more selectable elements are selectable and correspond to a respective type of user feedback that evaluates the media content with respect to the user query, and
wherein the one or more selectable elements includes a given selectable element that corresponds to negative user feedback;
causing the media content to be rendered at the user interface;
causing the one or more selectable elements to be rendered at the user interface and with respect to the media content;
in response to receiving user input that selects the given selectable element, of the one or more selectable elements, that corresponds to the negative user feedback:
determining, based on the received user input that selects the given selectable element, user feedback that evaluates the media content with respect to the user query, wherein determining the user feedback that evaluates the media content with respect to the user query comprises:
causing a feedback window to be rendered at the user interface,
wherein the feedback window is rendered in response to receiving the user input that selects the given selectable element, and
wherein the feedback window includes one or more selectable options each displaying a corresponding description that classifies the negative user feedback;
receiving additional user input that selects one of the one or more selectable options; and
determining a classification of the negative user feedback based on the additional user input; and
training or fine-tuning the generative model based on the user query and the user feedback that evaluates the media content with respect to the user query, wherein training or fine-tuning the generative model based on the user query and the user feedback that evaluates the media content with respect to the user query comprises:
in response to determining that the classification of the negative user feedback is a first classification:
reinforcing the generative model to not output a tag or prompt utilized to obtain the media content in response to the user query, and
in response to determining that the classification of the negative user feedback is a second classification different from the first classification:
rewriting an input prompt that is processed as input using the generative model; and
training or fine-tuning the generative model using the rewritten input prompt.
14 . The system of claim 13 , wherein the one or more selectable elements includes an additional given selectable element that corresponds to positive user feedback.
15 . The system of claim 14 , wherein the instructions further cause the one or more processors to perform operations of:
in response to receiving user input that selects the given additional selectable element, of the one or more selectable elements, that corresponds to the positive user feedback:
generating a training instance to: include an input prompt that includes the user query as a training instance input, and include at least a response having the tag or the prompt, that was utilized to obtain the media content, as training instance output.
16 . The system of claim 15 , wherein training or fine-tuning the generative model based on the user query and the user feedback that evaluates the media content with respect to the user query further comprises:
processing the training instance input corresponding to the input prompt that includes the user query, using the generative model, to generate a predicted model output that includes at least a predicted tag or predicted prompt,
comparing the predicted model output with the training instance output; and
training or fine-tuning the generative model based on comparing the predicted model output with the training instance output.
17 . The system of claim 13 , wherein the first classification indicates that the media content is unnecessary or not a good quality, and wherein the second classification indicates that the media content is unrelated to the user query.
18 . The system of claim 13 , wherein the feedback window further includes an input field to receive customized user feedback.
19 . The system of claim 13 , wherein the media content is non-generative media content that is obtained based on the tag generated using the generative model.
20 . A non-transitory storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations of:
receiving a user query, the user query being received via a user interface of a client device;
generating, using a generative model, a generative response that is responsive to the user query and that includes at least media content to be rendered and one or more selectable elements to be rendered with respect to media content,
wherein each of the one or more selectable elements are selectable and correspond to a respective type of user feedback that evaluates the media content with respect to the user query, and
wherein the one or more selectable elements includes a given selectable element that corresponds to negative user feedback:
causing the media content to be rendered at the user interface;
causing the one or more selectable elements to be rendered at the user interface and with respect to the media content;
in response to receiving user input that selects the given selectable element, of the one or more selectable elements, that corresponds to the negative user feedback:
determining, based on the received user input that selects the given selectable element, user feedback that evaluates the media content with respect to the user query, wherein determining the user feedback that evaluates the media content with respect to the user query comprises:
causing a feedback window to be rendered at the user interface,
wherein the feedback window is rendered in response to receiving the user input that selects the given selectable element, and
wherein the feedback window includes one or more selectable options each displaying a corresponding description that classifies the negative user feedback;
receiving additional user input that selects one of the one or more selectable options; and
determining a classification of the negative user feedback based on the additional user input; and
training or fine-tuning the generative model based on at least the user feedback that evaluates the media content with respect to the user query, wherein training or fine-tuning the generative model based on the user query and the user feedback that evaluates the media content with respect to the user query comprises:
in response to determining that the classification of the negative user feedback is a first classification:
reinforcing the generative model to not output a tag or prompt utilized to obtain the media content in response to the user query, and
in response to determining that the classification of the negative user feedback is a second classification different from the first classification:
rewriting an input prompt that is processed as input using the generative model; and
training or fine-tuning the generative model using the rewritten input prompt.