IP Library Granted Patent US 10,964,322
Granted Patent B2
US 10,964,322 · App. 16/255,481 · Granted Mar 30, 2021

Voice interaction tool for voice-assisted application prototypes

Inventors: Mark C. Webster (New York, NY); Susse Soenderby Jensen (Brooklyn, NY); Scott Thomas Werner (Brooklyn, NY); Daniel Cameron Cundiff (Brooklyn, NY); Blake Allen Clayton Sawyer (Astoria, NY)
Assignee: Adobe Inc.
G10L15/22G10L13/00G10L15/08G10L15/30G10L2015/223
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,964,322
App. No.
16/255,481
Granted
Mar 30, 2021
Kind
B2
Abstract

A voice interaction tool for voice-assisted application prototypes is described. A visual page of an application prototype is displayed in a design interface. The design interface is controlled to provide an interaction interface to receive a trigger and an associated action for the visual page of the application prototype. The trigger may correspond to one of a voice command, a user gesture, or a time delay, and the action may correspond to one of a speech response, a page transition to an additional visual page of the application prototype, or playback of a media file. User input is received to provide the trigger and the action, and associated interaction data is generated to include the trigger, the action, and the visual page of the application prototype. The associated interaction data is stored to enable testing of the trigger and the action during a testing phase of the application prototype.

Claims (77)

1. In a digital medium environment to aid design of an application prototype with voice functionality, a method implemented by at least one computing device, the method comprising:

displaying, on a display device of the at least one computing device, at least one visual page of the application prototype with voice functionality in a design interface;

controlling, by the at least one computing device, the design interface to provide an interaction interface to receive a trigger and an associated action for the at least one visual page of the application prototype with voice functionality;

receiving, via the design interface, user input to provide the trigger and the action and to select a trigger type of the trigger and an action type of the action from a selectable list of trigger types and action types presented within the interaction interface, the selectable list including trigger types comprising at least a voice command and a user gesture and action types comprising at least a speech response and a page transition; and

generating, by the at least one computing device, associated interaction data comprising the trigger, the action, and the visual page of the application prototype.

2. The method as described in claim 1 , wherein the interaction interface is displayed in response to receiving linking input to link the visual page to the additional visual page.

3. The method as described in claim 1 , wherein the selectable list of action types further comprises playback of a media file.

4. The method as described in claim 1 , further comprising generating a design specification comprising instructions to develop the application based on the application prototype, the design specification including visual pages of the application prototype, triggers provided via the design interface, and associated actions provided via the design interface.

5. The method as described in claim 1 , further comprising generating an executable application based on the application prototype designed using the design application.

6. The method as described in claim 1 , wherein the trigger type comprises the voice command and the action type comprises the speech response.

7. A method as described in claim 1 , further comprising storing the associated interaction data to enable testing of the application prototype during a testing phase.

8. A method as described in claim 7 , wherein the selected action type comprises the speech response, and wherein the method further comprises:

initiating, by the at least one computing device, the testing phase for the application prototype;

in the testing phase, displaying the at least one visual page of the application prototype;

detecting, by the at least one computing device, an additional trigger;

comparing the additional trigger to stored triggers of the associated interaction data to identify a matching trigger;

determining that the matching trigger is associated with the speech response in the associated interaction data;

providing, in real time, a text string of the speech response to a text-to-voice service;

receiving, from the text-to-voice service, an audio file of the speech response; and

outputting the speech response by playing back the audio file.

9. A method as described in claim 7 , wherein the selected trigger type comprises the voice command.

10. A method as described in claim 9 , wherein the receiving the user input to provide the trigger further comprises:

capturing, by at least one microphone of the at least one computing device, the voice command;

providing the captured voice command to a voice-to-text service;

obtaining a text string of the voice command from the voice-to-text service; and

associating the text string of the voice command with the action.

11. A method as described in claim 9 , further comprising:

initiating, by the at least one computing device, the testing phase for the application prototype;

in the testing phase, displaying the at least one visual page of the application prototype;

capturing, by the at least one microphone of the at least one computing device, an additional voice command;

providing, in real time, the additional voice command to a voice-to-text service;

receiving an additional text string of the additional voice command from the voice-to-text service;

comparing the additional text string of the additional voice command to stored text strings of voice commands of the associated interaction data to identify a matching voice command; and

performing the action associated with the matching voice command in the associated interaction data.

12. A system comprising:

at least one microphone;

at least one processor; and

a memory having stored thereon computer-readable instructions that are executable by the at least one processor to perform operations including:

controlling a design interface to provide an interaction interface to receive a voice command and an associated action for at least one visual page of an application prototype with voice functionality;

receiving, via the design interface, user input to provide the voice command and the associated action and to select an action type of the action from a selectable list of action types presented within the interaction interface, the selectable list of action types comprising at least a speech response and a page transition;

initiating a testing phase for the application prototype with voice functionality;

in the testing phase, displaying the at least one visual page of the application prototype;

capturing, by the at least one microphone, a voice command;

providing, in real time, the captured voice command to a voice-to-text service;

receiving, from the voice-to-text service, a text string of the captured voice command;

comparing the text string of the captured voice command to stored text strings of voice commands associated with the application prototype to identify the voice command as matching the captured voice command; and

performing the action associated with the voice command during the testing phase of the application prototype.

13. A system as described in claim 12 , wherein performing the action comprises:

identifying a page transition associated with the matching voice command, the page transition identifying an additional visual page of the application prototype; and

displaying the different visual page of the application prototype.

14. A system as described in claim 12 , wherein performing the action comprises:

identifying a media file associated with the matching voice command; and

initiating playback of the media file.

15. A system as described in claim 12 , wherein performing the action comprises outputting a speech response associated with the voice command.

16. A system as described in claim 15 , wherein performing the action comprises:

identifying a speech response associated with the matching voice command;

providing a text string of the speech response to a text-to-speech service;

receiving, from the text-to-speech service, an audio file of the speech response; and

outputting the speech response by playing back the audio file.

17. In a digital medium environment to enable testing of an application prototype with voice functionality, a method implemented by at least one computing device, the method comprising:

controlling, by the at least one computing device, a design interface to provide an interaction interface to receive a trigger and an associated action for at least one visual page of the application prototype with voice functionality;

receiving, via the design interface, user input to provide the trigger and the action and to select a trigger type of the trigger and an action type of the action from a selectable list of trigger types and action types presented within the interaction interface, the selectable list including trigger types comprising at least a voice command and a user gesture and action types comprising at least a speech response and a page transition;

initiating, by the at least one computing device, a testing phase for the application prototype with voice functionality;

in the testing phase, displaying the at least one visual page of the application prototype;

detecting, by the at least one computing device, user input providing a trigger;

comparing the trigger to stored triggers associated with the application prototype to identify the trigger as matching the detected trigger;

identifying a speech response associated with the trigger matching trigger;

providing, in real time, a text string of the speech response to a text-to-voice service;

receiving, from the text-to-voice service, an audio file of the speech response; and

outputting, by the at least one computing device, the speech response by playing back the audio file.

18. A method as described in claim 17 , wherein the trigger comprises a voice command that is captured by at least one microphone of the at least one computing device, and wherein the method further comprises:

providing, in real time, the voice command to a voice-to-text service;

receiving, from the voice-to-text service, a text string of the voice command;

comparing the text string of the voice command to stored text strings of voice commands associated with the application prototype to identify a matching voice command; and

wherein the identifying the speech response comprises identifying the speech response associated with the matching voice command.

19. A method as described in claim 17 , wherein the trigger comprises a user gesture.

20. A method as described in claim 17 , wherein the trigger comprises a time delay, and wherein the outputting the speech response by playing back the audio file occurs after the time delay expires.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 3, 2019
From: WEBSTER, MARK C.; JENSEN, SUSSE SOENDERBY; WERNER, SCOTT THOMAS; CUNDIFF, DANIEL CAMERON; SAWYER, BLAKE ALLEN CLAYTON
To: ADOBE INC.
Reel/Frame 048783/0393 →
Continuity (1)
Related Publication 20200234697A1 · Jul 23, 2020