One-shot visual language reasoning over graphical depictions of data
Provided is a one-shot solution to visual language reasoning. Example systems described herein decompose the challenge of visual language reasoning into two steps: translation of a graphical depiction of data (e.g., a plot or chart) into text; followed by reasoning over the translated text. In particular, example systems described herein can include a machine-learned visual-to-language conversion model that translates a graphical depiction of a dataset to a set of text descriptive of the dataset. The output of visual-to-language conversion model can then be directly used to prompt a language model, (e.g., a pretrained large language model (LLM)), exploiting the few-shot reasoning capabilities of the language model.
1 . A computer-implemented method to process graphical depictions of data, the method comprising:
obtaining, by a computing system comprising one or more computing devices, an input comprising a graphical depiction of a dataset;
processing, by the computing system, the graphical depiction of the dataset with a machine-learned visual-to-language conversion model to generate, as an output of the machine-learned visual-to-language conversion model a set of text descriptive of the dataset, the machine-learned visual-to-language conversion model trained using a loss function that measures a relative mapping similarity, the relative mapping similarity comprising a similarity between predicted headers and training headers;
processing, by the computing system, the set of text descriptive of the dataset with a machine-learned language model to generate, as an output of the machine-learned language model, a textual output; and
providing, by the computing system, the textual output as an output.
2 . The computer-implemented method of claim 1 , wherein the graphical depiction of the dataset comprises an image that depicts a chart.
3 . The computer-implemented method of claim 1 , wherein the graphical depiction of the dataset comprises an image that depicts a plot.
4 . The computer-implemented method of claim 1 , wherein the set of text descriptive of the dataset comprises a linearized table.
5 . The computer-implemented method of claim 1 , wherein:
the input further comprises a natural language query;
processing, by the computing system, the set of text descriptive of the dataset with the machine-learned language model comprises jointly processing, by the computing system, the set of text descriptive of the dataset and the natural language query with the machine-learned language model to generate, as the output of the machine-learned language model, the textual output; and
the textual output comprises a textual response that is responsive to the natural language query.
6 . The computer-implemented method of claim 1 , wherein the machine-learned visual-to-language conversion model has been trained separately from the machine-learned language model.
7 . The computer-implemented method of claim 1 , wherein:
the machine-learned visual-to-language conversion model has been trained using a set of supervised training data comprising a plurality of training pairs, each training pair comprising a training graphical depiction of a dataset and a training textual description of the dataset; and
the machine-learned visual-to-language conversion model has been trained to predict the training textual description of the dataset.
8 . The computer-implemented method of claim 7 , wherein the loss function measures the relative mapping similarity between predicted tuples and training tuples, wherein each of the predicted tuples and training tuples comprises a row header, a column header, and a value.
9 . The computer-implemented method of claim 1 , wherein:
the machine-learned visual-to-language conversion model has been trained using a set of supervised training data comprising a plurality of training pairs, each training pair comprising a training graphical depiction of a dataset and training rendering code, wherein the training rendering code comprises rendering code to render the training graphical depiction of a dataset; and
the machine-learned visual-to-language conversion model has been trained to predict the training rendering code.
10 . The computer-implemented method of claim 1 , wherein the machine-learned visual-to-language conversion model has been trained using a math dataset comprising textual math problem inputs rendered as images.
11 . A computing system configured to process graphical depictions of data, the computing system comprising:
one or more processors; and
one or more non-transitory computer-readable media that collectively store:
a machine-learned visual-to-language conversion model configured to convert graphical depictions of data to textual descriptions;
a machine-learned language model configured to process textual input to generate textual output; and
instructions that, when executed by the computing system, cause the computing system to perform operations, the operations comprising:
obtaining, by the computing system, an input comprising a graphical depiction of a dataset;
processing, by the computing system, the graphical depiction of the dataset with a machine-learned visual-to-language conversion model to generate, as an output of the machine-learned visual-to-language conversion model a set of text descriptive of the dataset, the machine-learned visual-to-language conversion model trained using a loss function that measures a relative mapping similarity between predicted headers and training headers; and
processing, by the computing system, the set of text descriptive of the dataset with a machine-learned language model to generate, as an output of the machine-learned language model, a textual output.
12 . The computing system of claim 11 , wherein the graphical depiction of the dataset comprises an image that depicts a chart.
13 . The computing system of claim 11 , wherein the graphical depiction of the dataset comprises an image that depicts a plot.
14 . The computing system of claim 11 , wherein the set of text descriptive of the dataset comprises a linearized table.
15 . The computing system of claim 11 , wherein:
the input further comprises a natural language query;
processing, by the computing system, the set of text descriptive of the dataset with the machine-learned language model comprises jointly processing, by the computing system, the set of text descriptive of the dataset and the natural language query with the machine-learned language model to generate, as the output of the machine-learned language model, the textual output; and
the textual output comprises a textual response that is responsive to the natural language query.
16 . One or more non-transitory computer-readable media that collectively store instructions that, when executed by a computing system, cause the computing system to perform operations, the operations comprising:
obtaining a visual-to-language conversion model;
pre-training the visual-to-language conversion model using one or more pre-training tasks wherein at least one pre-training task of the one or more pre-training tasks comprises a math reasoning task;
after said pre-training, fine-tuning the visual-to-language conversion model on a fine-tuning task, wherein the fine-tuning task comprises converting a graphical depiction of a dataset to a textual description of the dataset; and
after said fine-tuning, deploying the visual-to-language conversion model in combination with a machine-learned language model to perform processing of graphical depictions of data wherein processing the graphical depictions of data comprise rendering a text-based numerical reasoning input as an image and decoding an answer.
17 . The one or more non-transitory computer-readable media of claim 16 , wherein another pre-training task of the one or more pre-training tasks comprises a chart de-rendering task.
18 . The one or more non-transitory computer-readable media of claim 17 , wherein the chart de-rendering task comprises converting a graphical depiction of a dataset to rendering code that, when executed, causes rendering of the graphical depiction of the dataset.
19 . The one or more non-transitory computer-readable media of claim 16 , wherein the math reasoning task comprises processing a textual math reasoning input that has been rendered as an image.