IP Library Granted Patent US 11,461,221
Granted Patent B2
US 11,461,221 · App. 16/482,187 · Granted Oct 4, 2022

Action validation for digital assistant-based applications

Inventors: Dharma Muppalla (Lawrenceville, NJ); Nikhil Rao (Santa Clara, CA)
Assignee: GOOGLE LLC
G06F11/366G06F3/167G06F11/3664G06F11/3692G06F16/243G06F16/24565
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,461,221
App. No.
16/482,187
Granted
Oct 4, 2022
Kind
B2
Abstract

Validating actions in a digital assistant-based application is provided. The system identifies an application with a conversational interface. The system selects an action from an action repository and generates, via a natural language processor, a trigger phrase for input into the application. The system executes the application to process the trigger phrase to identify an action of the application. The system identifies a parameter used by the application to execute the action, and generates, based on the parameter and via execution of the conversational interface of the application, a first query responsive to the trigger phrase. The system generates a first response to the first query for input into the application. The system determines, based on execution of the application to process the first response, a state of the application. The system evaluates the state to determine an error code and provide a notification based on the error code.

Claims (86)

1. A method of validating actions in a digital assistant-based application, comprising:

identifying, by a data processing system comprising at least one processor, an application provided by an application developer device of an application developer, the application having a conversational interface;

selecting, by the data processing system based on the application, an action from an action repository of the data processing system;

generating, via a natural language processor of the data processing system and based on the action, a trigger phrase for input into the application, the trigger phrase representing a simulated voice-based input detected by a sensor, and the trigger phrase being generated based on logs that include historical trigger phrases detected by sensors of computing devices to initiate execution of actions for the application or other applications having a same type or domain as the application;

executing, by the data processing system, the application to process the trigger phrase via the conversational interface;

identifying, by the data processing system, an action of the application responsive to the trigger phrase;

identifying, by the data processing system, a parameter used by the application to execute the action of the application;

generating, by the data processing system, based on the parameter and via execution of the conversational interface of the application, a first query responsive to the trigger phrase;

generating, by the natural language processor of the data processing system, a first response to the first query for input into the application;

determining, by the data processing system based on execution of the application to process the first response, a state of the application;

evaluating, by the data processing system, the state of the application to determine an error code; and

providing, by the data processing system to the application developer device, a notification based on the error code.

2. The method of claim 1 , comprising:

determining the parameter used by the application comprises an electronic account identifier;

generating the first query comprising a request for the electronic account identifier; and

generating, via the natural language processor, the first response comprising a simulation value for the electronic account identifier.

3. The method of claim 1 , comprising:

preventing the application from executing the action of the application responsive to a value provided with the first response.

4. The method of claim 1 , comprising:

determining the state of the application is a termination state;

evaluating the state of the application to determine a resource utilized by the application is active; and

generating the error code indicating that the application failed to deactivate the resource responsive to the termination state.

5. A system to validate actions in a digital assistant-based application, comprising:

a data processing system comprising one or more processors to:

identify an application provided by an application developer device of an application developer, the application having a conversational interface;

select, for the application, an action from an action repository of the data processing system;

generate, based on the action and via a natural language processor of the data processing system, a trigger phrase for input into the application, the trigger phrase representing a simulated voice-based input detected by a sensor, and the trigger phrase being generated based on logs that include historical trigger phrases detected by sensors of computing devices to initiate execution of actions for the application or other applications having a same type or domain as the application;

execute the application to process the trigger phrase via the conversational interface, and identify an action of the application responsive to the trigger phrase;

identify a parameter used by the application to execute the action of the application;

generate, based on the parameter and via execution of the conversational interface of the application, a first query responsive to the trigger phrase;

generate, via the natural language processor, a first response to the first query for input into the application;

determine, based on execution of the application to process the first response, a state of the application;

evaluate the state of the application to determine an error code; and

provide, to the application developer device, a notification based on the error code.

6. The system of claim 5 , wherein the trigger phrase is a sentence that is constructed by the natural language processor to initiate execution of the action.

7. The system of claim 5 , comprising the data processing system to:

determine the parameter used by the application comprises an electronic account identifier;

generate the first query comprising a request for the electronic account identifier; and

generate, via the natural language processor, the first response comprising a simulation value for the electronic account identifier.

8. The system of claim 5 , comprising the data processing system to:

prevent the application from executing the action of the application responsive to a value provided with the first response.

9. The system of claim 5 , comprising the data processing system to:

determine the state of the application is a termination state;

evaluate the state of the application to determine a resource utilized by the application is active; and

generate the error code indicating that the application failed to deactivate the resource responsive to the termination state.

10. The system of claim 5 , comprising the data processing system to:

determine the state of the application is a termination state;

detect that a client device sensor utilized by the application is active; and

generate the error code indicating that the application failed to deactivate the client device sensor responsive to the termination state.

11. The system of claim 5 , comprising the data processing system to:

generate the first response based on a model trained with historical logs comprising responses detected by microphones of computing devices.

12. The system of claim 5 , comprising the data processing system to:

select, based on a metric associated with a model trained with historical logs, a metric of the model or a predetermined response repository for generation of the first query; and

generate, based on the selection, the first response via the metric of the model or the predetermined response repository.

13. The system of claim 5 , comprising the data processing system to:

establish, responsive to the first response, a telecommunication channel between a virtual client device and the application; and

provide, via the virtual client device and through the telecommunication channel, a second response to the application.

14. The system of claim 5 , comprising the data processing system to:

establish, responsive to the first response, a telecommunication channel between a device associated with the application developer and a virtual client device; and

provide, via the virtual client device and through the telecommunication channel, a second response to the device associated with the application developer.

15. The system of claim 5 , comprising the data processing system to:

generate a second response based on a semantic analysis of the first response, the first query, and the trigger phrase; and

provide the second response for processing by the application executed by the data processing system.

16. The system of claim 5 , comprising the data processing system to:

identify a virtual client account; and

provide information associated with the virtual client account to the application to process the action of the application responsive to the first response.

17. The system of claim 5 , comprising the data processing system to:

identify a virtual client account;

provide information associated with the virtual client account to the application to process the action responsive to the first response; and

block, responsive to detection of the virtual client account, execution of the action of the application by the application.

18. The system of claim 5 , wherein the error code indicates that the application attempted to execute the action of the application with a missing value or partial data for the parameter used by the application to execute the action of the application.

19. A system to validate actions in a digital assistant-based application, comprising:

a data processing system comprising one or more processors to:

identify an application provided by an application developer device of an application developer, the application having a conversational interface;

select, for the application, an action from an action repository of the data processing system;

generate, based on the action and via a natural language processor of the data processing system, a trigger phrase for input into the application, the trigger phrase representing a simulated voice-based input detected by a sensor;

execute the application to process the trigger phrase via the conversational interface, and identify an action of the application responsive to the trigger phrase;

identify a parameter used by the application to execute the action of the application;

generate, based on the parameter and via execution of the conversational interface of the application, a first query responsive to the trigger phrase;

generate, via the natural language processor, a first response to the first query for input into the application;

establish, responsive to the first response, a telecommunication channel between a device associated with the application developer and a virtual client device;

determine, based on execution of the application to process the first response, a state of the application, including:

detecting termination of the telecommunication channel; and

identifying, responsive to termination of the telecommunication channel, a state of a microphone of the virtual client device;

evaluate the state of the application to determine an error code, including setting the error code based on the state of the microphone of the virtual client device being active; and

provide, to the application developer device, a notification based on the error code.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 18, 2019
From: MUPPALLA, DHARMA; RAO, NIKHIL
To: GOOGLE LLC
Reel/Frame 050411/0094 →
Continuity (1)
Related Publication 20210294724A1 · Sep 23, 2021
Cited By (1)
US 12,400,655