IP Library › Granted Patent US 12,118,981
Granted Patent B2
US 12,118,981 · App. 17/475,897 · Granted Oct 15, 2024

Determining multilingual content in responses to a query

Inventors: Wangqing Yuan (Wilmington, MA); Bryan Christopher Horling (Belmont, CA); David Kogan (Natick, MA)
Assignee: GOOGLE LLC
G10L13/086G10L15/22G10L2015/223G10L2015/225
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,118,981
App. No.
17/475,897
Filed
Sep 15, 2021
Granted
Oct 15, 2024
Kind
B2
Art Unit
2658
USPC
704/258
Abstract

Implementations relate to determining multilingual content to render at an interface in response to a user submitted query. Those implementations further relate to determining a first language response and a second language response to a query that is submitted to an automated assistant. Some of those implementations relate to determining multilingual content that includes a response to the query in both the first and second languages. Other implementations relate to determining multilingual content that includes a query suggestion in the first language and a query suggestion in a second language. Some of those implementations relate to pre-fetching results for the query suggestions prior to rendering the multilingual content.

Claims (87)

1. A method implemented by one or more processors, the method comprising:

receiving audio data that captures a spoken query of a user that is in a first language, wherein the spoken query is provided via an automated assistant interface of a client device, and wherein the first language is specified as a primary language for the user;

generating, based on processing the audio data, a first language response to the spoken query, wherein the first language response is in the first language;

generating a second language response, to the spoken query, that is in the second language, wherein the second language is specified as a secondary language of interest to the user;

determining, based on verification data provided with or derived from the audio data, to render multilingual content in response to the spoken query, the multilingual content including the first language response and the second language response;

in response to determining to render the multilingual content:

causing the multilingual content to be rendered at the assistant interface of the client device and in response to the spoken query;

determining a query suggestion, wherein determining the query suggestion is based on the spoken query, the first language response, and/or the second language response;

in response to determining to render the multilingual content:

causing a first language version of the query suggestion and a second language version of the query suggestion to be rendered at the assistant interface of the client device in response to the spoken query;

prior to receiving any selection of the first language version of the query suggestion or the second language version of the query suggestion:

generating a first response to the query suggestion in the first language;

generating a second response to the query suggestion in the second language; and

causing the first response and the second response to be cached;

receiving a selection of the first language version of the query suggestion; and

in response to receiving the selection of first language version of the query suggestion:

causing the cached first response to the query suggestion to be audibly rendered; and

causing the cached second response to the query suggestion to be audibly rendered subsequent to causing the cached first response to the query suggestion to be audibly rendered.

2. The method of claim 1 , further comprising:

causing one or more actions to be performed in response to receiving the audio data.

3. The method of claim 1 , wherein causing the multilingual content to be rendered at the assistant interface of the client device comprises:

causing the first language response to be audibly rendered as first synthesized speech output and then causing the second language response to be audibly rendered as second synthesized speech output.

4. The method of claim 1 , further comprising:

generating a second language query by translating first language recognized text, of the spoken query, to the second language; and

in response to determining to render the multilingual content:

causing the second language query to be rendered at the assistant interface of the client device and in response to the spoken query.

5. The method of claim 4 , further comprising:

causing the second language query to be visually rendered with a selectable audible rendering interface element, wherein the audible rendering interface element, when selected, causes the second language query to be audibly rendered as synthesized speech output.

6. The method of claim 1 , wherein generating the second language response is performed in response to determining to render the multilingual content.

7. The method of claim 1 , further comprising:

receiving a selection of the second language version of the query suggestion; and

in response to receiving the selection of second language version of the query suggestion:

causing the cached second response to the query suggestion to be audibly rendered; and

causing the cached first response to the query suggestion to be audibly rendered subsequent to causing the cached second response to the query suggestion to be audibly rendered.

8. The method of claim 1 , further comprising:

determining a user proficiency measure that is specific to the user and that is specific to the second language, wherein determining to render the multilingual content is further based on the user proficiency measure.

9. The method of claim 8 , further comprising:

determining a complexity measure of the second language response, wherein determining to render the multilingual content based on the user proficiency measure comprises:

determining to render the multilingual content based on comparing the user proficiency measure to the complexity measure of the second language response.

10. The method of claim 9 , wherein determining the complexity measure comprises:

determining, based on the terms of the second language response, a comprehension level for the second language response, wherein the comprehension level is indicative of level of skill in the second language that is sufficient to comprehend the second language response.

11. The method of claim 8 , wherein determining to render the multilingual content based on the user proficiency measure comprises determining that the user proficiency measure satisfies a threshold.

12. The method of claim 1 , further comprising:

determining a user interest measure indicative of user interest in being provided with content in the second language, wherein determining to render the multilingual content is further based on the user interest measure.

13. The method of claim 1 , wherein the verification data includes an identifier of the user that provided the audio data, wherein the user has previously indicated an interest in being provided multilingual content.

14. A method implemented by one or more processors, the method comprising:

receiving audio data that captures a spoken query of a user that is in a first language, wherein the spoken query is provided via an automated assistant interface of a client device, and wherein the first language is specified as a primary language for the user;

generating, based on processing the audio data, a first language response to the spoken query, wherein the first language response is in the first language;

generating a second language response, to the spoken query, that is in the second language, wherein the second language is specified as a secondary language of interest to the user;

determining, based on verification data provided with or derived from the audio data, to render multilingual content in response to the spoken query, the multilingual content including the first language response and the second language response;

in response to determining to render the multilingual content:

causing the multilingual content to be rendered at the assistant interface of the client device and in response to the spoken query;

determining a query suggestion, wherein determining the query suggestion is based on the spoken query, the first language response, and/or the second language response;

in response to determining to render the multilingual content:

causing a first language version of the query suggestion and a second language version of the query suggestion to be rendered at the assistant interface of the client device in response to the spoken query;

prior to receiving any selection of the first language version of the query suggestion or the second language version of the query suggestion:

generating a first response to the query suggestion in the first language;

generating a second response to the query suggestion in the second language; and

causing the first response and the second response to be cached;

receiving a selection of the second language version of the query suggestion; and

in response to receiving the selection of second language version:

causing the cached second response to the query suggestion to be audibly rendered; and

causing the cached first response to the query suggestion to be audibly rendered subsequent to causing the cached second response to the query suggestion to be audibly rendered.

15. The method of claim 14 , further comprising:

causing one or more actions to be performed in response to receiving the audio data.

16. The method of claim 14 , wherein causing the multilingual content to be rendered at the assistant interface of the client device comprises:

causing the first language response to be audibly rendered as first synthesized speech output and then causing the second language response to be audibly rendered as second synthesized speech output.

17. A method implemented by one or more processors, the method comprising:

receiving audio data that captures a spoken query of a user that is in a first language, wherein the spoken query is provided via an automated assistant interface of a client device, and wherein the first language is specified as a primary language for the user;

generating, based on processing the audio data, a first language response to the spoken query, wherein the first language response is in the first language;

generating a second language response, to the spoken query, that is in the second language, wherein the second language is specified as a secondary language of interest to the user;

determining, based on verification data provided with or derived from the audio data, to render multilingual content in response to the spoken query, the multilingual content including the first language response and the second language response;

in response to determining to render the multilingual content:

causing the multilingual content to be rendered at the assistant interface of the client device and in response to the spoken query;

determining a query suggestion, wherein determining the query suggestion is based on the spoken query, the first language response, and/or the second language response;

prior to receiving any selection of a first language version of the query suggestion or a second language version of the query suggestion:

generating a first response to the query suggestion in the first language;

generating a second response to the query suggestion in the second language; and

causing the first response and the second response to be cached; and

in response to determining to render the multilingual content:

causing the first language version of the query suggestion and the second language version of the query suggestion to be rendered at the assistant interface of the client device in response to the spoken query;

wherein selection of the first language version of the query suggestion causes the cached first response to the query suggestion to be audibly rendered, and causes the cached second response to the query suggestion to be audibly rendered subsequent to causing the cached first response to the query suggestion to be audibly rendered, and

wherein selection of the cached second version of the query suggestion causes the cached second response to the query suggestion to be audibly rendered, and causes the cached first response to the query suggestion to be audibly rendered subsequent to causing the cached second response to the query suggestion to be audibly rendered.

18. The method of claim 17 , further comprising:

causing one or more actions to be performed in response to receiving the audio data.

19. The method of claim 17 , wherein causing the multilingual content to be rendered at the assistant interface of the client device comprises:

causing the first language response to be audibly rendered as first synthesized speech output and then causing the second language response to be audibly rendered as second synthesized speech output.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 6, 2021
From: YUAN, WANGQING; HORLING, BRYAN CHRISTOPHER; KOGAN, DAVID
To: GOOGLE LLC
Reel/Frame 057719/0091 →
Continuity (1)
Related Publication 20230084294A1 · Mar 16, 2023
Cited By (1)
US 12,711,329