Language models for reading charts
Systems and methods for natural language processing are described. Embodiments of the present disclosure obtain a chart and a query via a user interface. An answer model generates an answer to the query based on the chart, wherein the answer model comprises a machine learning model trained based on chart data for the chart. A description model generates a visual description based on the answer and the chart, wherein the description model comprises a machine learning model trained based on a chart specification for the chart. A response component transmits a response to the query based on the answer and the visual description.
1 . A method comprising:
obtaining chart data, a chart specification, and a query, wherein the chart data comprises a domain attribute of a chart and a data value corresponding to the domain attribute represented in a structured text format, and wherein the chart specification comprises structured text indicating that a visual element of the chart represents the domain attribute of the chart;
generating, by an answer model, an answer to the query based on the chart data, wherein the answer model comprises a machine learning model trained based on training data including the domain attribute of the chart;
generating, by a description model, a visual description based on the answer and the chart specification by providing an output from the answer model as an input to the description model, wherein the visual description comprises a text description of the visual element representing the domain attribute and corresponding to the answer, and wherein the description model comprises a machine learning model trained based on training data including training chart specification data; and
transmitting, by a response component, a response to the query based on the answer and the visual description, wherein the response answers the query and describes the visual element representing the domain attribute corresponding to the answer.
2 . The method of claim 1 , further comprising:
predicting a plurality of probability values for candidate answers based on the query; and
selecting the answer based on the plurality of probability values.
3 . The method of claim 1 , wherein:
the query comprises a natural language query and the response comprises a natural language response.
4 . The method of claim 1 , wherein:
the machine learning model is trained to perform a question answering task and a visual explanation generation task.
5 . The method of claim 1 , wherein:
the chart data comprises domain facts related to the chart.
6 . The method of claim 1 , wherein:
the chart specification indicates a correspondence between elements of the chart and visual properties of the chart.
7 . A method comprising:
obtaining training data including chart data, a chart specification, a query about a chart and a ground-truth visual description of the chart corresponding to the query, wherein the chart data comprises a domain attribute of a chart and a data value corresponding to the domain attribute represented in a structured text format, and wherein the chart specification comprises structured text indicating that a visual element of the chart represents the domain attribute of the chart;
generating, by a description model, a predicted visual description of the chart based on the chart specification by providing an output from an answer model as an input to the description model, wherein the predicted visual description comprises a text description of the visual element representing the domain attribute and corresponding to an answer generated by the answer model; and
training the description model based on the predicted visual description and the ground-truth visual description.
8 . The method of claim 7 , further comprising:
generating the ground-truth visual description based on the chart specification for the chart.
9 . The method of claim 8 , further comprising:
generating a template description based on the chart specification, wherein the ground-truth visual description is generated based on the template description using a rephrasing model.
10 . The method of claim 8 , further comprising:
generating first description model training data based on individual rows of the chart specification and second description model training data based on relationships between the chart specification and chart data for the chart; and
training the description model in a first phase based on the first description model training data and in a second phase based on the second description model training data.
11 . The method of claim 7 , further comprising:
generating the query based on the chart specification and the chart data for the chart.
12 . The method of claim 7 , further comprising:
generating a template query based on an individual row of the chart data for the chart and a query template, wherein the query comprises a rephrased version of the template query.
13 . The method of claim 7 , further comprising:
identifying the answer to the query, wherein the predicted visual description is generated based on the answer.
14 . The method of claim 7 , further comprising:
generating first answer model training data based on individual rows of the chart data for the chart and second answer model training data based on relationships between the individual rows of the chart data; and
training the answer model to predict an answer to the query, wherein the answer model is trained in a first phase based on the first answer model training data and in a second phase based on the second answer model training data.
15 . The method of claim 14 , further comprising:
generating a template query based on an individual row of the chart data and a query template; and
generating a rephrased query based on the template query using a rephrasing model, wherein the first answer model training data includes the rephrased query.
16 . An apparatus comprising:
a processor;
a memory comprising instructions executable by the processor;
a user interface configured to obtain chart data, a chart specification, and a query, wherein the chart data comprises a domain attribute of a chart and a data value corresponding to the domain attribute represented in a structured text format, and wherein the chart specification comprises structured text indicating that a visual element of the chart represents the domain attribute of the chart;
an answer model configured to generate an answer to the query based on the chart data, wherein the answer model comprises a machine learning model trained based on training data including the domain attribute of the chart;
a description model configured to generate a visual description based on the answer and the chart specification by providing an output from the answer model as an input to the description model, wherein the visual description comprises a text description of the visual element representing the domain attribute and corresponding to the answer, and wherein the description model comprises a machine learning model trained based on training data including training chart specification data; and
a response component configured to transmit a response to the query based on the answer and the visual description, wherein the response answers the query and describes the visual element representing the domain attribute corresponding to the answer.
17 . The apparatus of claim 16 , further comprising:
a template component configured to generate template sentences based on the chart data and the chart specification, wherein the answer model or the description model is trained based on the template sentences.
18 . The apparatus of claim 17 , further comprising:
a rephrasing model configured to generate additional training sentences based on the template sentences.
19 . The apparatus of claim 16 , further comprising:
an image encoder configured to encode the chart to obtain a chart representation, wherein the description model takes the chart representation as input.
20 . The apparatus of claim 16 , further comprising:
a training component configured to train the answer model and the description model.