Granted Patent B2

US 12,536,382 · App. 18/381,162 · Granted Jan 27, 2026

Inventors: Melanie K. Tory (Palo Alto, CA); Vidya R. Setlur (Portola Valley, CA)

Assignee: Tableau Software, LLC

G06F40/35G06F16/3329G06F40/00G06F40/20G06F40/289G06F40/30G06F40/40G06F3/0484

Abstract

Claims (76)

1 . A method of using natural language for visual analysis of a dataset, comprising:

at a computer device having one or more processors and memory storing one or more programs configured for execution by the one or more processors:

displaying an initial data visualization based on a first dataset retrieved from a database using a first set of one or more queries;

determining an initial visualization state that includes a plurality of elements corresponding to (i) data attributes of the initial data visualization and (ii) visual encodings that map the data attributes to visual attributes in the initial data visualization;

receiving a user input to specify a natural language command related to the initial data visualization;

determining a transitional goal, to transform the initial data visualization, based on one or more cue phrases extracted from the natural language command;

deriving an updated visualization state based on the transitional goal by applying one or more transitional rules to the plurality of elements of the initial visualization state, wherein the one or more transitional rules prioritize a request directed to a first element, of the plurality of elements, that is explicitly specified in the natural language command over an inference that is derived from the one or more cue phrases, wherein:

the request directed to the first element includes a visual encoding request for a first data attribute; and

the method includes in accordance with a determination that the visual encoding request results in an empty set for the updated visualization state, disregarding the visual encoding request and applying one or more default heuristic rules;

computing a set of one or more analytical functions associated with the updated visualization state, thereby creating a set of one or more functional phrases;

updating the initial data visualization based on the set of one or more functional phrases to generate an updated data visualization; and

displaying the updated data visualization.

2 . The method of claim 1 , wherein:

the plurality of elements includes transformations to generate derived attributes from the data attributes; and

deriving the updated visualization state includes performing a transformation operation that includes at least one of:

binning a quantitative variable;

changing a default ordering of the data attributes on an x-axis and/or a y-axis of the initial data visualization;

adding a redundant color encoding; and

generating a custom encoding.

3 . The method of claim 1 , wherein the transitional goal is selected from the group consisting of: elaboration, retrying, adjustment, undoing, or starting anew.

4 . The method of claim 1 , wherein the one or more transitional rules maintain conversational coherence between the initial visualization state and the updated visualization state by maintaining coherence in the data attributes and the visual encodings between the initial data visualization and the updated data visualization.

5 . The method of claim 1 , further comprising:

determining whether the one or more cue phrases contain a name attribute, a value, a term of measure, and/or a term of aggregation; and

in accordance with a determination that the one or more cue phrases contain a name attribute or a value, but neither a term of measure nor a term of aggregation, determining the transitional goal to be calculating a count of records corresponding to the name attribute or the value.

6 . The method of claim 1 , further comprising:

determining whether the data attributes include a categorical attribute having a number of categories less than a threshold value; and

in accordance with the determination that the data attributes include categorical attribute having a number of distinct categories less than the threshold value, determining the transitional goal to be comparing the categories.

7 . The method of claim 1 , further comprising:

determining if the one or more cue phrases contains a term corresponding to a chart type or a named visual variable;

in accordance with a determination that the one or more cue phrases contains a term corresponding to a chart type, determining the transitional goal to be selecting the chart type; and

in accordance with a determination that the one or more cue phrases contains a term corresponding to a named visual variable, determining the transitional goal to be encoding for the named visual variable according to the one or more cue phrases.

8 . The method of claim 1 , wherein the one or more transitional rules comprise a CONTINUE rule that maintains all the elements of the initial visualization state in the updated visualization state and adding one or more new elements in the updated visualization state based on the one or more cue phrases.

9 . The method of claim 1 , wherein the one or more transitional rules comprise a RETAIN rule that retains all the elements of the initial visualization state in the updated visualization state without adding any element based on the one or more cue phrases.

10 . The method of claim 1 , wherein the one or more transitional rules comprise a SHIFT rule that maintains all the elements of the initial visualization state in the updated visualization state and replacing one or more elements in the updated visualization state based on the one or more cue phrases.

11 . The method of claim 1 , wherein the one or more transitional rules comprise a RESET rule that clears all the elements of the initial visualization state to create an empty set.

12 . The method of claim 3 , further comprising:

determining if the one or more cue phrases contains terms that signify elaboration; and

in accordance with a determination that the one or more cue phrases contains terms that signify elaboration, determining the transitional goal to elaborate.

13 . The method of claim 3 , further comprising:

determining if the one or more cue phrases contains terms that signify adjust/pivot; and

in accordance with a determination that the one or more cue phrases contains terms that signify adjust/pivot, determining the transitional goal to adjust/pivot.

14 . The method of claim 3 , further comprising:

determining if the one or more cue phrases contains terms that signify undoing, or a repetition of an utterance that generated a desired visualization state; and

in accordance with a determination that the one or more cue phrases contains terms that signify undoing, determining the transitional goal to undo.

15 . The method of claim 3 , further comprising:

determining if the one or more cue phrases contains terms that signify starting anew, or corresponds to an explicit reset; and

in accordance with a determination that the one or more cue phrases contains terms that signify starting anew, determining the transitional goal to start anew.

16 . The method of claim 3 , further comprising:

determining if the one or more cue phrases contains terms that signify retry; and

in accordance with a determination that the one or more cue phrases contains terms that signify retry, determining the transitional goal to retry.

17 . The method of claim 1 , wherein (i) elaboration implies adding new information in the updated data visualization, (ii) adjustment implies adapting one or more aspects of the initial data visualization in the updated data visualization, (iii) retrying implies re-attempting a previous step that failed, and (iv) undoing implies reverting the initial data visualization to a previous state.

18 . An electronic device, comprising:

one or more processors;

memory; and

one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the one or more programs including instructions for:

displaying an initial data visualization based on a first dataset retrieved from a database using a first set of one or more queries;

determining an initial visualization state that includes a plurality of elements corresponding to (i) data attributes of the initial data visualization and (ii) visual encodings that map the data attributes to visual attributes in the initial data visualization;

receiving a user input to specify a natural language command related to the initial data visualization;

determining a transitional goal, to transform the initial data visualization, based on one or more cue phrases extracted from the natural language command;

deriving an updated visualization state based on the transitional goal by applying one or more transitional rules to the plurality of elements of the initial visualization state, wherein the one or more transitional rules prioritize a request directed to a first element, of the plurality of elements, that is explicitly specified in the natural language command over an inference that is derived from the one or more cue phrases, wherein:

the request directed to the first element includes a visual encoding request for a first data attribute; and

the one or more programs include instructions for, in accordance with a determination that the visual encoding request results in an empty set for the updated visualization state, disregarding the visual encoding request and applying one or more default heuristic rules;

computing a set of one or more analytical functions associated with the updated visualization state, thereby creating a set of one or more functional phrases;

updating the initial data visualization based on the set of one or more functional phrases to generate an updated data visualization; and

displaying the updated data visualization.

19 . A non-transitory computer readable storage medium storing one or more programs configured for execution by an electronic device, the one or more programs comprising instructions for:

displaying an initial data visualization based on a first dataset retrieved from a database using a first set of one or more queries;

determining an initial visualization state that includes a plurality of elements corresponding to (i) data attributes of the initial data visualization and (ii) visual encodings that map the data attributes to visual attributes in the initial data visualizations;

receiving a user input to specify a natural language command related to the initial data visualization;

determining a transitional goal, to transform the initial data visualization, based on one or more cue phrases extracted from the natural language command;

deriving an updated visualization state based on the transitional goal by applying one or more transitional rules to the plurality of elements of the initial visualization state, wherein the one or more transitional rules prioritize a request directed to a first element, of the plurality of elements, that is explicitly specified in the natural language command over an inference that is derived from the one or more cue phrases, wherein:

the request directed to the first element includes a visual encoding request for a first data attribute; and

the one or more programs comprise instructions for, in accordance with a determination that the visual encoding request results in an empty set for the updated visualization state, disregarding the visual encoding request and applying one or more default heuristic rules;

computing a set of one or more analytical functions associated with the updated visualization state, thereby creating a set of one or more functional phrases;

updating the initial data visualization based on the set of one or more functional phrases to generate an updated data visualization; and

displaying the updated data visualization.

Assignments (4)

Continuity (4)

References Cited (287)