IP Library › Granted Patent US 11,514,896
Granted Patent B2
US 11,514,896 · App. 16/622,805 · Granted Nov 29, 2022

Interfacing with applications via dynamically updating natural language processing

Inventors: Quazi Hussain (Mountain View, CA); Adam Coimbra (Mountain View, CA); Ilya Firman (Mountain View, CA)
Assignee: GOOGLE LLC
G10L15/1822G10L15/063G10L15/083G10L2015/0635
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,514,896
App. No.
16/622,805
Granted
Nov 29, 2022
Kind
B2
Abstract

Dynamic interfacing with applications is provided. For example, a system receives a first input audio signal. The system processes, via a natural language processing technique, the first input audio signal to identify an application. The system activates the application for execution on the client computing device. The application declares a function the application is configured to perform. The system modifies the natural language processing technique responsive to the function declared by the application. The system receives a second input audio signal. The system processes, via the modified natural language processing technique, the second input audio signal to detect one or more parameters. The system determines that the one or more parameters are compatible for input into an input field of the application. The system generates an action data structure for the application. The system inputs the action data structure into the application, which executes the action data structure.

Claims (80)

1. A system to interface with applications, comprising:

a digital assistant component executed by a client computing device comprising one or more processors and memory;

a natural language processor component executed by the digital assistant component to:

receive a first input audio signal detected by a microphone of the client computing device; and

process, via a natural language processing technique, the first input audio signal to identify an indication of an application;

an application manager component of the digital assistant component to activate, responsive to the indication, the application for execution on the client computing device and cause the application to declare, to the digital assistant component via a protocol, a function the application is configured to perform;

the natural language processor component to:

subsequent to activation of the application and based on the function declared by the application, modify the natural language processing technique responsive to the function declared by the application, to increase a likelihood of detection of one or more parameters relative to the natural language processing technique not modified based on the function;

receive a second input audio signal detected by the microphone of the client computing device;

process, via the modified natural language processing technique, the second input audio signal to detect the one or more parameters; and

determine, based on a comparison between the one or more parameters and the function declared by the application, that the one or more parameters are compatible for input into an input field of the application activated by the digital assistant component on the client computing device; and

a direct action API executed by the digital assistant component to:

generate, responsive to the determination of compatibility and based on the one or more parameters, an action data structure for the application; and

input the action data structure into the application via the protocol to cause the application to execute the action data structure and update a user interface element of the application.

2. The system of claim 1 , comprising the digital assistant component to:

update a model based on the function; and

provide the updated model to the natural language processing technique.

3. The system of claim 1 , comprising the digital assistant component to:

receive state information declared by the application via the protocol; and

use the state information to perform real-time natural language processing of the second input audio signal to identify the one or more parameters.

4. The system of claim 1 , comprising the digital assistant component to:

receive state information declared by the application via the protocol; and

map, via the modified natural language processing technique, the second input audio signal to the state information to identify a request.

5. The system of claim 1 , comprising the digital assistant component to:

receive, subsequent to input of the action data structure into the application, an indication of a change to a state of the application;

modify, responsive to the indication of the change to the state, the natural language processing technique previously modified responsive to the function declared by the application;

receive a third input audio signal; and

process, via the natural language processing technique modified responsive to the indication of the change to the state, the third input audio signal to identify a second one or more parameters.

6. The system of claim 1 , comprising the digital assistant component to:

receive, via the protocol from the application, an indication of a change to a state of the application, wherein the indication of the change to the state invokes the digital assistant component;

parse a current user interface of the application responsive to the indication of the change to identify one or more input fields provided via the user interface element of the application;

receive a third input audio signal; and

process the third input audio signal using the natural language processing technique based on the one or more input fields to identify a second one or more parameters of the third input audio signal for input into the application responsive to the change to the state of the application.

7. The system of claim 1 , comprising the digital assistant component to:

perform a lookup in an index of the client computing device to determine the application is installed on the client computing device; and

activate, responsive to the determination the application is installed on the client computing device, the application.

8. The system of claim 1 , comprising the digital assistant component to:

receive an indication that the application is deactivated; and

return the modified natural language processing technique to a default natural language processing technique.

9. The system of claim 1 , comprising the digital assistant component to:

receive an indication that the application is deactivated; and

remove, from the natural language processing technique, the modification based on the function.

10. A method of interfacing with applications, comprising:

receiving, by a digital assistant component executed by one or more processors of a client computing device, a first input audio signal detected by a microphone of the client computing device;

processing, by the digital assistant component using a natural language processing technique, the first input audio signal to identify an indication of an application;

activating, by the digital assistant component responsive to the indication, the application for execution on the client computing device and cause the application to declare, to the digital assistant component via a protocol, a function the application is configured to perform;

subsequent to activation of the application and based on the function declared by the application, modifying, by the digital assistant component, the natural language processing technique responsive to the function declared by the application, to increase a likelihood of detection of one or more parameters relative to the natural language processing technique not modified based on the function;

receiving, by the digital assistant component, a second input audio signal detected by the microphone of the client computing device;

processing, by the digital assistant component using the modified natural language processing technique, the second input audio signal to detect the one or more parameters;

determining, by the digital assistant component based on a comparison between the one or more parameters and the function declared by the application, that the one or more parameters are compatible for input into an input field of the application activated by the digital assistant component on the client computing device;

generating, by the digital assistant component responsive to the determination of compatibility and based on the one or more parameters, an action data structure for the application; and

inputting, by the digital assistant component, the action data structure into the application via the protocol to cause the application to execute the action data structure and update a user interface element of the application.

11. The method of claim 10 , comprising:

updating a model based on the function; and

providing the updated model to the natural language processing technique.

12. The method of claim 10 , comprising:

receiving state information declared by the application via the protocol; and

using the state information to perform real-time natural language processing of the second input audio signal to identify the one or more parameters.

13. The method of claim 10 , comprising:

receiving state information declared by the application via the protocol; and

mapping, via the modified natural language processing technique, the second input audio signal to the state information to identify a request.

14. The method of claim 10 , comprising:

receiving, subsequent to input of the action data structure into the application, an indication of a change to a state of the application;

modifying, responsive to the indication of the change to the state, the natural language processing technique previously modified responsive to the function declared by the application;

receiving a third input audio signal; and

processing, via the natural language processing technique modified responsive to the indication of the change to the state, the third input audio signal to identify a second one or more parameters.

15. The method of claim 10 , comprising:

receiving, via the protocol from the application, an indication of a change to a state of the application, wherein the indication of the change to the state invokes the digital assistant component;

parsing a current user interface of the application responsive to the indication of the change to identify one or more input fields provided via the user interface element of the application;

receiving a third input audio signal; and

processing the third input audio signal using the natural language processing technique based on the one or more input fields to identify a second one or more parameters of the third input audio signal for input into the application responsive to the change to the state of the application.

16. The method of claim 10 , comprising:

performing a lookup in an index of the client computing device to determine the application is installed on the client computing device; and

activating, responsive to the determination the application is installed on the client computing device, the application.

17. The method of claim 10 , comprising:

receiving an indication that the application is deactivated; and

returning the modified natural language processing technique to a default natural language processing technique.

18. The method of claim 10 , comprising:

receiving an indication that the application is deactivated; and

removing, from the natural language processing technique, the modification based on the function.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 16, 2019
From: HUSSAIN, QUAZI; COIMBRA, ADAM; FIRMAN, ILYA
To: GOOGLE LLC
Reel/Frame 051298/0184 →
Continuity (1)
Related Publication 20210358489A1 · Nov 18, 2021
Cited By (1)
US 12,548,568