IP Library Granted Patent US 11,087,760
Granted Patent B2
US 11,087,760 · App. 16/696,622 · Granted Aug 10, 2021

Multimodal transmission of packetized data

Inventors: Gaurav Bhaya (Sunnyvale, CA); Robert Stets (Mountain View, CA)
Assignee: Google, LLC
G10L15/22G06F3/165G10L15/14G10L15/1822G10L15/26G10L15/30H04L47/25G10L2015/088
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,087,760
App. No.
16/696,622
Granted
Aug 10, 2021
Kind
B2
Abstract

A system of multi-modal transmission of packetized data in a voice activated data packet based computer network environment is provided. A natural language processor component can parse an input audio signal to identify a request and a trigger keyword. Based on the input audio signal, a direct action application programming interface can generate a first action data structure, and a content selector component can select a content item. An interface management component can identify first and second candidate interfaces, and respective resource utilization values. The interface management component can select, based on the resource utilization values, the first candidate interface to present the content item. The interface management component can provide the first action data structure to the client computing device for rendering as audio output, and can transmit the content item converted for a first modality to deliver the content item for rendering from the selected interface.

Claims (65)

1. A system to transmit data in a voice-based computing environment, comprising:

a data processing system comprising one or more processors and memory to:

receive, via an interface of the data processing system, data packets comprising an input audio signal detected by a sensor of a client computing device;

parse the input audio signal to identify a request;

generate, based on the request, a first action data structure;

select a content item responsive to the request;

identify a plurality of interfaces of the client computing device;

determine a characteristic of each of the plurality of interfaces;

select, based on the characteristic of each of the plurality of interfaces, a first interface of the plurality of interfaces having a first characteristic; and

provide the first action data structure and the content item to the client computing device for presentation as audio output via the first interface of the client computing device.

2. The system of claim 1 , comprising:

the data processing system to provide the content item in a modality compatible with the first interface.

3. The system of claim 1 , comprising the data processing system to:

determine a capability of the first interface; and

convert the content item to a modality compatible with the capability of the first interface.

4. The system of claim 1 , wherein the first interface comprises an audio interface, comprising:

the data processing system to provide the content item for presentation via the audio interface.

5. The system of claim 1 , comprising the data processing system to:

select a second content item based on the first characteristic of the first interface; and

provide the second content item to the client computing device for presentation via the first interface.

6. The system of claim 1 , comprising the data processing system to:

parse the input audio signal to identify a keyword corresponding to the request; and

select the content item based at least on the keyword.

7. The system of claim 1 , comprising the data processing system to:

select a second content item; and

provide the second content item to the client computing device for presentation via a second interface of the client computing device that has a different characteristic than the first characteristic.

8. The system of claim 1 , comprising the data processing system to:

select a second content item comprising visual output;

select a second interface comprising a display device based on the second content item comprising visual output; and

provide the second content item to the client computing device for presentation via the second interface of the client computing device.

9. The system of claim 1 , wherein the characteristic of each of the plurality of interfaces comprises a resource utilization value, comprising:

the data processing system to select the first interface based on the resource utilization value to reduce resource utilization associated with presentation of the content item.

10. The system of claim 9 , wherein the resource utilization value comprises at least one of a battery status, a processor utilization, a memory utilization, or a network bandwidth utilization.

11. The system of claim 1 , comprising:

the data processing system to deliver the content item to the client computing device subsequent to transmission of the first action data structure to the client computing device.

12. The system of claim 1 , wherein the plurality of interfaces include at least one of a display screen, an audio interface, a vibration interface, an email interface, a push notification interface, a mobile computing device interface, a portable computing device application, a content slot on an online document, a chat application, mobile computing device application, a laptop, a watch, a virtual reality headset, and a speaker.

13. A method of transmitting data in a voice-based computing environment, comprising:

receiving, by a data processing system comprising one or more processors and memory, data packets comprising an input audio signal detected by a sensor of a client computing device;

parsing, by the data processing system, the input audio signal to identify a request;

generating, by the data processing system based on the request, a first action data structure;

selecting, by the data processing system, a content item responsive to the request;

identifying, by the data processing system, a plurality of interfaces of the client computing device;

determining, by the data processing system, a characteristic of each of the plurality of interfaces;

selecting, by the data processing system based on the characteristic of each of the plurality of interfaces, a first interface of the plurality of interfaces having a first characteristic; and

providing, by the data processing system, the first action data structure and the content item to the client computing device for presentation as audio output via the first interface of the client computing device.

14. The method of claim 13 , comprising:

providing the content item in a modality compatible with the first interface.

15. The method of claim 13 , comprising:

determining a capability of the first interface; and

converting the content item to a modality compatible with the capability of the first interface.

16. The method of claim 13 , wherein the first interface comprises an audio interface, comprising:

providing the content item for presentation via the audio interface.

17. The method of claim 13 , comprising:

selecting a second content item based on the first characteristic of the first interface; and

providing the second content item to the client computing device for presentation via the first interface.

18. The method of claim 13 , comprising:

parsing the input audio signal to identify a keyword corresponding to the request; and

selecting the content item based at least on the keyword.

19. The method of claim 13 , comprising:

selecting a second content item; and

providing the second content item to the client computing device for presentation via a second interface of the client computing device that has a different characteristic than the first characteristic.

20. The method of claim 13 , comprising:

selecting a second content item comprising visual output;

selecting a second interface comprising a display device based on the second content item comprising visual output; and

providing the second content item to the client computing device for presentation via the second interface of the client computing device.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 27, 2019
From: BHAYA, GAURAV; STETS, ROBERT JAMES, JR
To: GOOGLE INC.
Reel/Frame 051128/0330 →
CHANGE OF NAME Recorded Nov 27, 2019
From: GOOGLE INC.
To: GOOGLE LLC
Reel/Frame 051851/0256 →
Continuity (3)
Continuation 16039202 · Jul 18, 2018
Continuation 15395703 · Dec 30, 2016
Related Publication 20200098369A1 · Mar 26, 2020
Cited By (1)
US 12,243,521