IP Library Granted Patent US 12,153,618
Granted Patent B2
US 12,153,618 · App. 17/561,375 · Granted Nov 26, 2024

Applied artificial intelligence technology for automatically generating narratives from visualization data

Inventors: Daniel Joseph Platt (Chicago, IL); Mauro Eduardo Ignacio Mujica-Parodi, III (Chicago, IL); Lawrence A. Birnbaum (Evanston, IL); Alexander Rudolf Sippel (Chicago, IL); Jonathan Alden Drake (Chicago, IL); Peter Horace Sherman (Chicago, IL)
Assignee: Salesforce, Inc.
G06F16/51G06F16/248G06F16/9024G06T11/206
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,153,618
App. No.
17/561,375
Filed
Dec 23, 2021
Granted
Nov 26, 2024
Kind
B2
Examiner
VU, BAI DUC
Art Unit
2162
USPC
707/722
Abstract

Disclosed herein are example embodiments that describe how a narrative generation techniques can be used in connection with data visualization tools to automatically generate narratives that explain the information conveyed by a visualization of a data set. In example embodiments, new data structures and artificial intelligence (AI) logic can be used by narrative generation software to map different types of visualizations to different types of story configurations that will drive how narrative text is generated by the narrative generation software.

Claims (37)

1. A method comprising:

processing, by a processor, visualization parameter data to determine how to translate the visualization parameter data into a natural language narrative that explains the visualization parameter data, wherein the visualization parameter data corresponds to a visualization about a data set, wherein the visualization parameter data comprises visualization metadata that indicates a visualization type for the visualization from among a plurality of different visualization types, wherein the processing step comprises:

mapping the processed visualization parameter data including the visualization metadata that indicates the visualization type to a story configuration for a story type based on a data structure that associates data corresponding to a plurality of different visualization types with data corresponding to a plurality of different story configurations for different story types; and

the processor generating the natural language narrative about the data set based on the mapped story configuration.

2. The method of claim 1 further comprising:

displaying the natural language narrative in coordination with a display of the visualization.

3. The method of claim 1 , wherein the visualization parameter data further comprises visualization data and a plurality of visualization parameters, the visualization data comprising a plurality of data elements from the data set for display by the visualization, the visualization parameters comprising the visualization metadata that indicates the visualization type and additional visualization metadata about the visualization,

wherein the mapped story configuration comprises a parameterizable story configuration with respect to a plurality of story parameters, the method further comprising:

parameterizing the mapped story configuration by (1) mapping the story parameters to a plurality of the visualization parameters and (2) supplying the mapped story configuration with data elements from the data set that correspond to the visualization parameters mapped to the story parameters; and

generating the natural language narrative about the data set based on the mapped and parameterized story configuration.

4. The method of claim 3 , wherein the mapped story configuration comprises a plurality of narrative analytics, the method further comprising executing the narrative analytics using a plurality of the data elements from the data set that correspond to the visualization parameters mapped to the story parameters to determine data content for expression in the natural language narrative.

5. The method of claim 4 , the method further comprising determining a characterization of the data set based on execution of the narrative analytics, and wherein the natural language narrative expresses the determined characterization using natural language.

6. The method of claim 1 , wherein the different visualization types comprise a bar chart and a line chart, wherein the data structure associates (1) bar charts with a first story type and (2) line charts with a second story type that is different than the first story type.

7. The method of claim 6 , wherein the first story type defines a story that compares a plurality of elements of the bar chart, and wherein the second story type defines a story that describes a trend in a plurality of data values of the line chart.

8. The method of claim 1 , wherein the different visualization types comprise a first type of bar chart and a second type of bar chart, wherein the data structure associates (1) the first type of bar charts with a first story type and (2) the second type of bar charts with a second story type that is different than the first story type.

9. The method of claim 1 , wherein the different visualization types comprise a line chart with a single line and a line chart with multiple lines, wherein the data structure associates line charts with a single line with a first story type and line charts with multiple lines with a second story type that is different than the first story type.

10. The method of claim 9 , wherein the first story type defines a story that describes a trend in a plurality of data values of the line chart, and wherein the second story type defines a story that characterizes a correlation between (1) a plurality of data values that correspond to a first line of the multiple lines and (2) a plurality of data values that correspond to a second line of the multiple lines.

11. The method of claim 1 , wherein the processor is part of a narrative generation platform, wherein the narrative generation platform is configured to (1) receive the visualization parameter data from a visualization platform and (2) provide the natural language narrative to the visualization platform for display in coordination with the visualization.

12. One or more non-transitory computer readable media having instructions stored thereon for performing a method, the method comprising:

processing, by a processor, visualization parameter data to determine how to translate the visualization parameter data into a natural language narrative that explains the visualization parameter data, wherein the visualization parameter data corresponds to a visualization about a data set, wherein the visualization parameter data comprises visualization metadata that indicates a visualization type for the visualization from among a plurality of different visualization types, wherein the processing step comprises:

mapping the processed visualization parameter data including the visualization metadata that indicates the visualization type to a story configuration for a story type based on a data structure that associates data corresponding to a plurality of different visualization types with data corresponding to a plurality of different story configurations for different story types; and

the processor generating the natural language narrative about the data set based on the mapped story configuration.

13. The one or more non-transitory computer readable media of claim 12 , wherein the visualization parameter data further comprises visualization data and a plurality of visualization parameters, the visualization data comprising a plurality of data elements from the data set for display by the visualization, the visualization parameters comprising the visualization metadata that indicates the visualization type and additional visualization metadata about the visualization, wherein the mapped story configuration comprises a parameterizable story configuration with respect to a plurality of story parameters, the method further comprising:

parameterizing the mapped story configuration by (1) mapping the story parameters to a plurality of the visualization parameters and (2) supplying the mapped story configuration with data elements from the data set that correspond to the visualization parameters mapped to the story parameters; and

generating the natural language narrative about the data set based on the mapped and parameterized story configuration.

14. The one or more non-transitory computer readable media of claim 13 , wherein the mapped story configuration comprises a plurality of narrative analytics, the method further comprising executing the narrative analytics using a plurality of the data elements from the data set that correspond to the visualization parameters mapped to the story parameters to determine data content for expression in the natural language narrative.

15. The one or more non-transitory computer readable media of claim 14 , the method further comprising determining a characterization of the data set based on execution of the narrative analytics, and wherein the natural language narrative expresses the determined characterization using natural language.

16. The one or more non-transitory computer readable media of claim 12 , wherein the processor is part of a narrative generation platform, wherein the narrative generation platform is configured to (1) receive the visualization parameter data from a visualization platform and (2) provide the natural language narrative to the visualization platform for display in coordination with the visualization.

17. The one or more non-transitory computer readable media of claim 12 , wherein the mapped story configuration influences how the processor applies natural language generation (NLG) to select and organize information from the data set to be conveyed in the generated natural language narrative.

18. A system comprising:

memory storing visualization parameter data; and

a processor configured to process the visualization parameter data to determine how to translate the visualization parameter data into a natural language narrative that explains the visualization parameter data, wherein the visualization parameter data corresponds to a visualization about a data set, wherein the visualization parameter data comprises visualization metadata that indicates a visualization type for the visualization from among a plurality of different visualization types, wherein the processing step comprises:

mapping the processed visualization parameter data including the visualization metadata that indicates the visualization type to a story configuration for a story type based on a data structure that associates data corresponding to a plurality of different visualization types with data corresponding to a plurality of different story configurations for different story types; and

the processor generating the natural language narrative about the data set based on the mapped story configuration.

19. The system of claim 18 wherein the mapped story configuration comprises a plurality of narrative analytics, wherein the processor is further operable to execute the narrative analytics using a plurality of the data elements from the data set that correspond to the visualization parameters mapped to the story parameters to determine data content for expression in the natural language narrative.

20. The system of claim 18 , wherein the different visualization types comprise a bar chart and a line chart, wherein the data structure associates (1) bar charts with a first story type and (2) line charts with a second story type that is different than the first story type.

21. The system of claim 18 , wherein the processor is part of a narrative generation platform, wherein the narrative generation platform is configured to (1) receive the visualization parameter data from a visualization platform and (2) provide the natural language narrative to the visualization platform for display in coordination with the visualization.

Assignments (4)
CHANGE OF NAME Recorded Mar 25, 2024
From: NARRATIVE SCIENCE INC.
To: NARRATIVE SCIENCE LLC
Reel/Frame 066884/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 25, 2024
From: NARRATIVE SCIENCE LLC
To: SALESFORCE, INC.
Reel/Frame 067218/0449 →
CHANGE OF NAME Recorded Oct 13, 2023
From: NARRATIVE SCIENCE INC.
To: NARRATIVE SCIENCE LLC
Reel/Frame 065237/0929 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 23, 2021
From: PLATT, DANIEL JOSEPH; MUJICA-PARODI, MAURO EDUARDO IGNACIO, III; BIRNBAUM, LAWRENCE A.; SIPPEL, ALEXANDER RUDOLF; DRAKE, JONATHAN ALDEN; SHERMAN, PETER HORACE
To: NARRATIVE SCIENCE INC.
Reel/Frame 058474/0284 →
Continuity (7)
Continuation 15414089 · Jan 24, 2017
Continuation 15414027 · Jan 24, 2017
Continuation 15253385 · Aug 31, 2016
Continuation 15253385 · Aug 31, 2016
Continuation 15253385 · Aug 31, 2016
Provisional Application 62249813 · Nov 2, 2015
Related Publication 20220114206A1 · Apr 14, 2022