IP Library › Granted Patent US 12664159

Granted Patent B2

US 12664159 · App. 18/616,801 · Granted Jun 23, 2026

Techniques for transforming natural language conversation into a visualization representation

Inventors: Cong Duy Vu Hoang (Melbourne, AU); Gioacchino Tangari (Sydney, AU); Stephen Andrew McRitchie (Palo Alto, CA); Nitika Mathur (Melbourne, AU); Aashna Devang Kanuga (Foster City, CA); Steve Wai-Chun Siu (Melbourne, AU); Dalu Guo (Sydney, AU); Chang Xu (Sydney, AU); Mark Edward Johnson (Sydney, AU); Christopher Mark Broadbent (Wellington, FL); Thanh Long Duong (Melbourne, AU); Srinivasa Phani Kumar Gadde (Fremont, CA); Vishal Vishnoi (Redwood City, CA); Chandan Basavaraju (Bangalore, IN); Kenneth Khiaw Hong Eng (Newark, CA)

Assignee: Oracle International Corporation

G06F16/24522 G06F16/24573 G06F16/287

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 12664159
App. No.: 18/616,801
Granted: Jun 23, 2026
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

Techniques are disclosed herein for transforming natural language conversations into a visual output. In one aspect, a computer-implement method includes generating an input string by concatenating a natural language utterance with a schema representation comprising a set of entities for visualization actions, generating, by a first encoder of a machine learning model, one or more embeddings of the input string, encoding, by a second encoder of the machine learning model, relations between elements in the schema representation and words in the natural language utterance based on the one or more embeddings, generating, by a grammar-based decoder of the machine learning model and based on the encoded relations and the one or more embeddings, an intermediate logical form that represents at least the query, the one or more visualization actions, or the combination thereof, and generating, based on the intermediate logical form, a command for a computing system.

Claims (64)

1 . A computer-implemented method comprising:

generating an input string by concatenating (a) a natural language utterance comprising a query, one or more visualization actions, or a combination thereof with (b) a schema representation comprising a set of entities for visualization actions;

generating, by a first encoder of a semparizer machine learning model and based on the input string, one or more embeddings of the input string;

encoding, by a second encoder of the semparizer machine learning model, relations between elements in the schema representation and words in the natural language utterance based on the one or more embeddings;

generating, by a grammar-based decoder of the semparizer machine learning model and based on the encoded relations and the one or more embeddings, an intermediate logical form that represents at least the query, the one or more visualization actions, or the combination thereof; and

generating, based on the intermediate logical form, a command for a computing system, wherein the command comprises the query, the one or more visualization actions, or the combination thereof in a system programming language.

2 . The computer-implemented method of claim 1 , further comprising:

executing the command on the computing system, wherein the executing causes the computing system to obtain results for the query, generate, update or modify one or more visualizations based on the one or more visualization actions, or a combination thereof; and

communicating the results, the one or more visualizations, or the combination thereof to a user that provided the natural language utterance,

wherein the natural language utterance comprises the one or more visualization actions, and the one or more visualization actions are represented as one or more entities from the set of entities within the schema representation.

3 . The computer-implemented method of claim 1 , further comprising:

prior to generating the one or more embeddings of the natural language utterance and the schema representation, providing the input string to a filter machine learning model;

generating, by the filter machine learning model and based on the input string, a score indicating whether the natural language utterance is translatable to a command for the computing system and should be routed to the semparizer machine learning model;

comparing the score to a threshold value; and

responsive to determining that the score exceeds the threshold value, providing the input string to the semparizer machine learning model.

4 . The computer-implemented method of claim 1 , further comprising:

converting the intermediate logical form into an enhanced intermediate logical form that comprises the query, the one or more visualization actions, or the combination thereof in a version of a meaning representation language (MRL) that is translatable to the system programming language; and

generating, based on the enhanced intermediate logical form, the command for the computing system, wherein the command comprises the query, the one or more visualization actions, or the combination thereof in the system programming language.

5 . The computer-implemented method of claim 1 , further comprising providing, to the second encoder of the semparizer machine learning model, schema-linking relations that link entities in the schema representation and visualization action words in the natural language utterance, wherein the embeddings are further generated based on the schema-linking relations, and wherein the schema-linking relations comprise metadata specifying synonyms for visualization action words.

6 . The computer-implemented method of claim 1 , wherein the schema representation further comprises a schema specification for a given schema and a link attribute that connects one or more entities from the set of entities for visualization actions to one or more fact entities associated with the given schema.

7 . The computer-implemented method of claim 1 , further comprising providing, to the grammar-based decoder of the semparizer machine learning model, relational algebra grammar that represents the intermediate logical form as a tree, wherein the intermediate logical form is generated by the grammar-based decoder based on the encoded relations, the one or more embeddings, and the relational algebra grammar.

8 . One or more non-transitory computer-readable media storing instructions which, when executed by one or more processors, cause a system to perform cause a system to perform operations comprising:

generating, by a first encoder of a semparizer machine learning model and based on the input string, one or more embeddings of the input string;

9 . The one or more non-transitory computer-readable media of claim 8 , further comprising:

communicating the results, the one or more visualizations, or the combination thereof to a user that provided the natural language utterance,

10 . The one or more non-transitory computer-readable media of claim 8 , further comprising:

prior to generating the one or more embeddings of the natural language utterance and the schema representation, providing the input string to a filter machine learning model;

comparing the score to a threshold value; and

responsive to determining that the score exceeds the threshold value, providing the input string to the semparizer machine learning model.

11 . The one or more non-transitory computer-readable media of claim 8 , further comprising:

12 . The one or more non-transitory computer-readable media of claim 8 , further comprising providing, to the second encoder of the semparizer machine learning model, schema-linking relations that link entities in the schema representation and visualization action words in the natural language utterance, wherein the embeddings are further generated based on the schema-linking relations, and wherein the schema-linking relations comprise metadata specifying synonyms for visualization action words.

13 . The one or more non-transitory computer-readable media of claim 8 , wherein the schema representation further comprises a schema specification for a given schema and a link attribute that connects one or more entities from the set of entities for visualization actions to one or more fact entities associated with the given schema.

14 . The one or more non-transitory computer-readable media of claim 8 , further comprising providing, to the grammar-based decoder of the semparizer machine learning model, relational algebra grammar that represents the intermediate logical form as a tree, wherein the intermediate logical form is generated by the grammar-based decoder based on the encoded relations, the one or more embeddings, and the relational algebra grammar.

15 . A system comprising:

one or more processors; and

one or more computer-readable media storing instructions which, when executed by the one or more processors, cause the system to perform a method comprising:

generating, by a first encoder of a semparizer machine learning model and based on the input string, one or more embeddings of the input string;

16 . The system of claim 15 , further comprising:

communicating the results, the one or more visualizations, or the combination thereof to a user that provided the natural language utterance,

17 . The system of claim 15 , further comprising:

prior to generating the one or more embeddings of the natural language utterance and the schema representation, providing the input string to a filter machine learning model;

comparing the score to a threshold value; and

responsive to determining that the score exceeds the threshold value, providing the input string to the semparizer machine learning model.

18 . The system of claim 15 , further comprising:

19 . The system of claim 15 , further comprising providing, to the second encoder of the semparizer machine learning model, schema-linking relations that link entities in the schema representation and visualization action words in the natural language utterance, wherein the embeddings are further generated based on the schema-linking relations, and wherein the schema-linking relations comprise metadata specifying synonyms for visualization action words.

20 . The system of claim 15 , wherein the schema representation further comprises a schema specification for a given schema and a link attribute that connects one or more entities from the set of entities for visualization actions to one or more fact entities associated with the given schema.