IP Library Granted Patent US 11,455,339
Granted Patent B1
US 11,455,339 · App. 16/601,437 · Granted Sep 27, 2022

Incremental updates to natural language expressions in a data visualization user interface

Inventors: Jeffrey Ericson (Menlo Park, CA); Vidya Raghavan Setlur (Portola Valley, CA); Alex Djalali (Los Gatos, CA); Sanaz Golbabaei (Palo Alto, CA); Theodore Kornish (Palo Alto, CA); Xin Zhao (Mountain View, CA); An Shi (Vancouver, CA); Christoph Sachse (Stanford, CA); Eliana Leite Goldner (Vancouver, CA); Suyang Duan (Vancouver, CA)
Assignee: TABLEAU SOFTWARE, LLC
G06F16/90332G06F16/904G06F16/9038
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,455,339
App. No.
16/601,437
Granted
Sep 27, 2022
Kind
B1
Abstract

A method is performed at a computing device that includes a display, one or more processors, and memory. The device displays a data visualization based on a dataset retrieved from a database. It also displays one or more phrases in a first region. The phrases define the data visualization. The device receives user input in a second region to specify a natural language command related to the displayed data visualization. In response to the user input, the device extracts independent analytic phrases from the command, determines one or more proposed actions based on the extracted phrases, and displays the proposed actions. The device receives user selection of a first proposed action. In response, the device generates an updated data visualization. It displays the updated data visualization and displays one or more updated phrases in the first region. The updated phrases define the updated data visualization.

Claims (80)

1. A method of using natural language for updating visual analysis of datasets, comprising:

at a computing device having a display, one or more processors, and memory storing one or more programs configured for execution by the one or more processors:

displaying a data visualization based on data fields of a dataset retrieved from a database;

displaying, concurrently with the data visualization, a plurality of phrases in a first region in proximity to the data visualization, wherein the plurality of phrases collectively define the data visualization and each of the displayed phrases includes a respective one or more first field names of the data fields in the dataset;

receiving user input in a natural language command input region to specify a natural language command related to the displayed data visualization, wherein the natural language command input region is distinct from the first region;

in response to the user input:

determining that the natural language command includes a metacommand to: (1) add a data field to the data visualization, or (2) remove a data field from the data visualization, or (3) replace a data field in the data visualization with another data field;

determining one or more proposed actions in accordance with the determined metacommand;

for each of the one or more proposed actions, generating one or more respective phrases corresponding to the respective proposed action;

ranking the one or more proposed actions based on a saliency or weight of a missing term that is inferred from the natural language command according to one or more inferencing rules based on syntactic constraints imposed by a context-free grammar and semantic constraints imposed by attributes of the data fields of the dataset; and

displaying the ranked one or more proposed actions in a dropdown menu adjacent to the natural language command input region;

receiving user selection of a first proposed action of the ranked one or more proposed actions; and

in response to the user selection of the first proposed action:

generating an updated data visualization in accordance with the first proposed action;

determining one or more updated phrases corresponding to the first proposed action, wherein the one or more updated phrases collectively define the updated data visualization and include one or more second field names of data fields in the dataset;

displaying the updated data visualization; and

displaying the one or more updated phrases concurrently with the updated data visualization, in the first region.

2. The method of claim 1 , wherein the first field names and the second field names include a common field name.

3. The method of claim 1 , wherein the one or more first field names are displayed in a visually distinctive manner from other words included in the one or more phrases.

4. The method of claim 1 , wherein the natural language command includes a metacommand to remove a data field from the data visualization; and

the one or more proposed actions include a proposed action to remove a first phrase, of the plurality of phrases, containing a field name of the data field.

5. The method of claim 4 , further comprising displaying the proposed action to remove the first phrase in a strikethrough format.

6. The method of claim 4 , wherein the first proposed action includes removing the first phrase; and

displaying the one or more updated phrases includes removing the first phrase from display in the first region.

7. The method of claim 1 , wherein the plurality of phrases includes a first phrase having a first analytical concept; and

the one or more proposed actions include a second analytical concept distinct from the first analytical concept.

8. The method of claim 7 , wherein each of the first and second analytical concepts is selected from the group consisting of: a field/measure, an aggregation type, a level of detail of an aggregation, a filter, a sort, and a calculation over a time range.

9. The method of claim 1 , wherein the natural language command includes a metacommand to remove a filter expression containing a specified data field, and the method further comprises:

identifying a first phrase, from the plurality of phrases, that is a filter expression containing the specified data field; and

the one or more proposed actions include a proposed action to remove the first phrase, while retaining one or more of the plurality of phrases, containing the specified data field, that are not filter expressions.

10. The method of claim 1 , wherein the natural language command includes a command to replace a first aggregation operator for a first data field with a second different aggregation operator; and

displaying the one or more proposed actions includes displaying a proposed action that includes a first text string and a second text string, wherein the first text string includes the first data field and the first aggregation operator, the second text string includes the first data field and the second aggregation operator, and the first text string is visually distinguished from the second text string.

11. The method of claim 1 , wherein the natural language command further specifies replacement of one field name with a distinct field name; and

displaying one or more proposed actions include displaying a proposed action that includes the one field name and the distinct field name.

12. The method of claim 1 , wherein displaying the one or more proposed actions includes:

displaying a second proposed action that includes an add command; and

displaying a third proposed action that includes a replace command.

13. The method of claim 1 , wherein generating the updated data visualization includes requerying the database using a set of one or more queries based on the updated phrases, thereby retrieving an updated dataset.

14. The method of claim 1 , wherein the data visualization has a first data visualization type and the updated data visualization has a second data visualization type distinct from the first data visualization type.

15. The method of claim 1 , wherein the data visualization has a first data visualization type; and

displaying an updated data visualization includes displaying a plurality of second data visualizations each having the first data visualization type.

16. The method of claim 1 , further comprising ranking the one or more proposed actions based on a frequency of usage of a field name based on previous user history.

17. A computing device, comprising:

one or more processors;

memory coupled to the one or more processors;

a display; and

one or more programs stored in the memory and configured for execution by the one or more processors, the one or more programs comprising instructions for:

displaying a data visualization based on data fields of a dataset retrieved from a database;

displaying, concurrently with the data visualization, a plurality of phrases in a first region in proximity to the data visualization, wherein the plurality of phrases collectively define the data visualization and each of the displayed phrases includes a respective one or more first field names of the data fields in the dataset;

receiving user input in a natural language command input region to specify a natural language command related to the displayed data visualization, wherein the natural language command input region is distinct from the first region;

in response to the user input:

determining that the natural language command includes a metacommand to: (1) add a data field to the data visualization, or (2) remove a data field from the data visualization, or (3) replace a data field in the data visualization with another data field;

determining one or more proposed actions in accordance with the determined metacommand;

for each of the one or more proposed actions, generating one or more respective phrases corresponding to the respective proposed action;

ranking the one or more proposed actions based on a saliency or weight of a missing term that is inferred from the natural language command according to one or more inferencing rules based on syntactic constraints imposed by a context-free grammar and semantic constraints imposed by attributes of the data fields of the dataset, and

displaying the ranked one or more proposed actions in a dropdown menu adjacent to the natural language command input region;

receiving user selection of a first proposed action of the ranked one or more proposed actions; and

in response to the user selection of the first proposed action:

generating an updated data visualization in accordance with the first proposed action;

determining one or more updated phrases corresponding to the first proposed action, wherein the one or more updated phrases collectively define the updated data visualization and include one or more second field names of data fields in the dataset;

displaying the updated data visualization; and

displaying the one or more updated phrases concurrently with the updated data visualization, in the first region.

18. A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing device, cause the device to perform the method of:

displaying a data visualization based on data fields of a dataset retrieved from a database;

displaying, concurrently with the data visualization, a plurality of phrases in a first region in proximity to the data visualization, wherein the plurality of phrases collectively define the data visualization and each of the displayed phrases includes a respective one or more first field names of the data fields in the dataset;

receiving user input in a natural language command input region to specify a natural language command related to the displayed data visualization, wherein the natural language command input region is distinct from the first region;

in response to the user input:

determining that the natural language command includes a metacommand to: (1) add a data field to the data visualization, or (2) remove a data field from the data visualization, or (3) replace a data field in the data visualization with another data field;

determining one or more proposed actions in accordance with the determined metacommand;

for each of the one or more proposed actions, generating one or more respective phrases corresponding to the respective proposed action;

ranking the one or more proposed actions based on a saliency or weight of a missing term that is inferred from the natural language command according to one or more inferencing rules based on syntactic constraints imposed by a context-free grammar and semantic constraints imposed by attributes of the data fields of the dataset, and

displaying the one or more proposed actions in a dropdown menu adjacent to the natural language command input region;

receiving user selection of a first proposed action of the one or more proposed actions; and

in response to the user selection of the first proposed action:

generating an updated data visualization in accordance with the first proposed action;

determining one or more updated phrases corresponding to the first proposed action, wherein the one or more updated phrases collectively define the updated data visualization and include one or more second field names of data fields in the dataset;

displaying the updated data visualization; and

displaying the one or more updated phrases concurrently with the updated data visualization, in the first region.

19. The method of claim 1 , wherein the plurality of phrases is displayed as a plurality of distinct entities in the first region.

20. The method of claim 19 , wherein each entity of the plurality of entities is separated from respective adjacent entities in the first region by a comma and/or a conjunction.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 2, 2022
From: ERICSON, JEFFREY; SETLUR, VIDYA RAGHAVAN; DJALALI, ALEX; GOLBABAEI, SANAZ; KORNISH, THEODORE; ZHAO, XIN; SACHSE, CHRISTOPH; GOLDNER, ELIANA LEITE; DUAN, SUYANG
To: TABLEAU SOFTWARE, INC.
Reel/Frame 060086/0522 →
EMPLOYMENT AGREEMENT Recorded Jun 2, 2022
From: SHI, AN
To: TABLEAU SOFTWARE, INC.
Reel/Frame 060255/0069 →
CHANGE OF NAME Recorded Jun 1, 2022
From: TABLEAU SOFTWARE, INC.
To: TABLEAU SOFTWARE, LLC
Reel/Frame 060390/0418 →
Continuity (1)
Provisional Application 62897187 · Sep 6, 2019