Granted Patent B2

US 12,579,721 · App. 18/081,076 · Granted Mar 17, 2026

Inventors: Robinson Piramuthu (Oakland, CA); Sanqiang Zhao (Santa Clara, CA); Yadunandana Rao (Sunnyvale, CA); Zhiyuan Fang (San Jose, CA)

Assignee: Amazon Technologies, Inc.

G06T13/00G06F40/166G06F40/279G06F40/40G06T11/00G10L13/02G10L15/18G10L15/22

Abstract

Claims (117)

1 . A computer-implemented method comprising:

receiving first input audio data corresponding to a first spoken natural language input requesting a narrative be output, the first spoken natural language input indicating a first narrative parameter;

generating parameter data including the first narrative parameter included in the first spoken natural language input;

generating outline data based on the parameter data, wherein the outline data includes the first narrative parameter and a second narrative parameter not included in the parameter data;

processing, using a first trained machine learning (ML) component, the outline data to generate natural language data corresponding to the narrative requested in the first spoken natural language input, wherein the natural language data includes more words than the outline data, and the natural language data comprises:

a first portion corresponding to a first scene of the narrative, and

a second portion corresponding to a second scene of the narrative;

processing, using a second trained ML component, the first portion of the natural language data to determine a first entity represented in the first portion of the natural language data;

determining, using the second trained ML component, first image data corresponding to the first entity;

processing, using a third trained ML component, the first portion of the natural language data to determine first background image data corresponding to the first scene of the narrative;

processing, using a fourth trained ML component, the first portion of the natural language data and the first entity to determine an attribute corresponding to the first entity, wherein the attribute represents how the first entity is to be presented;

processing, using a fifth trained ML component, the first image data, the first background image data, and the attribute to generate first scene data, wherein the first scene data indicates how the first image data is to be rendered with the first background image data based on the attribute;

based on the first scene data, generating first output image data including the first image data and the first background image data, the first output image data corresponding to the first scene of the narrative;

processing, using the second trained ML component, the second portion of the natural language data to determine the first entity is represented in the second portion of the natural language data;

determining the first image data is to be used to render the first entity in the second scene of the narrative based on the first image data being used to represent the first entity in the first scene data;

generating second output image data including the first image data, the second output image data corresponding to the second scene of the narrative;

causing presentation of the first output image data; and

causing presentation of the second output image data.

2 . The computer-implemented method of claim 1 , further comprising:

storing a representation of the first entity, the first image data, the first background image data, and the attribute in association with a content request identifier corresponding to the first spoken natural language input;

retrieving the representation of the first entity;

based at least in part on determining that the first entity is represented in the second portion, retrieving the first image data;

processing, using the second trained ML component, the second portion of the natural language data to determine a second entity represented in the second portion of the natural language data;

determining, using the second trained ML component, second image data corresponding to the second entity;

processing, using the third trained ML component, the second portion of the natural language data to determine second background image data for the second scene of the narrative; and

processing, using the fifth trained ML component, the first image data, the second image data, and the second background image data to generate second scene data, wherein the second scene data indicates how the first image data and the second image data are to be rendered with the second background image data.

3 . The computer-implemented method of claim 1 , further comprising:

processing, using a sixth ML component, the natural language data to determine a third portion of the natural language data that corresponds to the first narrative parameter; and

replacing the third portion of the natural language data with the first narrative parameter.

4 . The computer-implemented method of claim 1 , further comprising:

processing, using a sixth ML component, the natural language data and the first entity to determine a portion of the first background image data where the first image data is to be located when it is rendered with the first background image data,

wherein the first scene data indicates the portion of the first background image data where the first image data is to be located when it is rendered with the first background image data, and

wherein the second output image data is generated to include the first image data located with respect to the first background image data as indicated in the first scene data.

5 . A computer-implemented method comprising:

receiving first input data requesting content be output, the first input data indicating a first parameter;

generating outline data based on the first parameter, wherein the outline data includes the first parameter and a second parameter not included in the first input data;

based on the outline data, generating natural language data corresponding to the content requested in the first input data, wherein the natural language data includes more words than the outline data;

processing, using a trained machine learning (ML) component, a first portion of the natural language data to determine a first entity included in the first portion of the natural language data;

determining, using the trained ML component, first image data corresponding to the first entity;

determining first background image data representing the natural language data;

generating first scene data indicating how the first image data is to be rendered with the first background image data;

based on the first scene data, generating first output image data including the first image data and the first background image data, the first output image data representing at least the first portion of the natural language data;

determining, by the trained ML component, the first entity is included in a second portion of the natural language data;

based on the first entity being included in the second portion of the natural language data and the first image data being used to represent the first entity in the first scene data, generating second output image data including the first image data, the second output image data representing at least the second portion of the natural language data;

causing presentation of the first output image data; and

causing presentation of the second output image data.

6 . The computer-implemented method of claim 5 , further comprising:

determining, in the first portion of the natural language data, an attribute corresponding to the first entity, wherein the attribute represents how the first entity is to be presented; and

generating the first scene data to indicate how the first image data is to be rendered using the attribute.

7 . The computer-implemented method of claim 5 , further comprising:

determining a spatial relationship between the first entity and a second entity included in the first portion of the natural language data, wherein the spatial relationship represents how the first entity is to be rendered with the second entity.

8 . The computer-implemented method of claim 5 , further comprising:

determining a second entity included in the second portion of the natural language data;

determining second image data corresponding to the second entity;

determining second background image data representing the second portion of the natural language data; and

generating second scene data indicating how the first image data and the second image data are to be rendered with the second background image data.

9 . The computer-implemented method of claim 5 , further comprising:

determining a third portion of the natural language data corresponding to a second entity;

determining the second entity corresponds to the first entity; and

based on the second entity corresponding to the first entity, replacing the second entity with the first entity in the third portion of the natural language data.

10 . The computer-implemented method of claim 5 , further comprising:

performing text-to-speech (TTS) processing using the natural language data to generate first output audio data comprising:

a first portion corresponding to the first portion of the natural language data, and

a second portion corresponding to the second portion of the natural language data;

causing the first output audio data to be presented;

while the first portion of the first output audio data is being presented, causing the first output image data to be presented; and

after the first portion of the first output audio data is presented, and while the second portion of the first output audio data is being presented, causing the second output image data to be presented.

11 . The computer-implemented method of claim 5 , further comprising:

generating third output image data corresponding to the first output image data, wherein the first entity is represented differently in the first output image data than in the third output image data; and

generating output video data using the first output image data and the third output image data.

12 . The computer-implemented method of claim 5 , wherein:

the first input data requests a narrative be output,

the first output image data corresponds to a first portion of the narrative, and

the second output image data corresponds to a second portion of the narrative.

13 . A computing system comprising:

at least one processor; and

at least one memory comprising instructions that, when executed by the at least one processor, cause the computing system to:

receive first input data requesting content be output, the first input data indicating a first parameter;

generate outline data based on the first parameter, wherein the outline data includes the first parameter and a second parameter not included in the first input data;

based on the outline data, generate natural language data corresponding to the content requested in the first input data, wherein the natural language data includes more words than the outline data;

process, using a trained machine learning (ML) component, a first portion of the natural language data to determine a first entity included in the first portion of the natural language data;

determine, using the trained ML component, first image data corresponding to the first entity;

determine first background image data representing the natural language data;

generate first scene data indicating how the first image data is to be rendered with the first background image data;

based on the first scene data, generate first output image data including the first image data and the first background image data, the first output image data representing at least the first portion of the natural language data;

determine, by the trained ML component, the first entity is included in a second portion of the natural language data;

based on the first entity being included in the second portion of the natural language data and the first image data being used to represent the first entity in the first scene data, generate second output image data including the first image data, the second output image data representing at least the second portion of the natural language data;

cause presentation of the first output image data; and

cause presentation of the second output image data.

14 . The computing system of claim 13 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to:

determine, in the first portion of the natural language data, an attribute corresponding to the first entity, wherein the attribute represents how the first entity is to be presented; and

generate the first scene data to indicate how the first image data is to be rendered using the attribute.

15 . The computing system of claim 13 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to:

determine a spatial relationship between the first entity and a second entity included in the first portion of the natural language data, wherein the spatial relationship represents how the first entity is to be rendered with the second entity.

16 . The computing system of claim 13 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to:

determine a second entity included in the second portion of the natural language data;

determine second image data corresponding to the second entity;

determine second background image data representing the second portion of the natural language data; and

generate second scene data indicating how the first image data and the second image data are to be rendered with the second background image data.

17 . The computing system of claim 13 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to:

determine a third portion of the natural language data corresponding to a second entity;

determine the second entity corresponds to the first entity; and

based on the second entity corresponding to the first entity, replace the second entity with the first entity in the third portion of the natural language data.

18 . The computing system of claim 13 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to:

perform text-to-speech (TTS) processing using the natural language data to generate first output audio data comprising:

a first portion corresponding to the first portion of the natural language data, and

a second portion corresponding to the second portion of the natural language data;

cause the first output audio data to be presented;

while the first portion of the first output audio data is being presented, cause the first output image data to be presented; and

after the first portion of the first output audio data is presented, and while the second portion of the first output audio data is being presented, cause the second output image data to be presented.

19 . The computing system of claim 13 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to:

generate third output image data corresponding to the first output image data, wherein the first entity is represented differently in the first output image data than in the third output image data; and

generate output video data using the first output image data and the third output image data.

20 . The computing system of claim 13 , wherein:

the first input data requests a narrative be output,

the first output image data corresponds to a first portion of the narrative, and

the second output image data corresponds to a second portion of the narrative.

Assignments (1)

Continuity (2)

References Cited (37)