IP Library Granted Patent US 10,656,808
Granted Patent B2
US 10,656,808 · App. 13/683,341 · Granted May 19, 2020

Natural language and user interface controls

Inventors: Gregg D. Wilensky (Pacific Palisades, CA); Walter W. Chang (San Jose, CA); Lubomira A. Dontcheva (San Francisco, CA); Gierad P. Laput (Canton, MI); Aseem O. Agarwala (Seattle, WA)
Assignee: Adobe Inc.
G06F3/04845G06F3/048G06F3/0484G06F3/04842G06F3/167G06F2203/0381
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,656,808
App. No.
13/683,341
Granted
May 19, 2020
Kind
B2
Abstract

Natural language and user interface control techniques are described. In one or more implementations, a natural language input is received that is indicative of an operation to be performed by one or more modules of a computing device. Responsive to determining that the operation is associated with a degree to which the operation is performable, a user interface control is output that is manipulable by a user to control the degree to which the operation is to be performed.

Claims (60)

1. A method comprising:

displaying, by a display device, in a user interface, an image;

capturing, by an audio capture device of a computing device, audio speech data describing a natural language input;

converting, by a speech-to-text engine, the audio speech data into text;

identifying, by a natural language processing module implemented by a processing system and computer-readable storage medium of the computing device, in the converted text of the audio speech data describing the natural language input:

an object included in the image;

an operation to be performed on the object in the image by one or more modules of the computing device; and

a degree to which the operation is to be performed on the object in the image;

recognizing, by an object identification module implemented by the processing system and computer-readable storage medium of the computing device implementing an object recognition algorithm, the object in the image indicated in the natural language input, wherein the object recognized by the computing device is a person, and the object recognition algorithm includes a facial recognition algorithm;

drawing, by the object identification module implemented by the processing system and computer-readable storage medium of the computing device, a user manipulable boundary in the user interface around the recognized object;

performing the identified operation to the identified degree on the recognized object, the operation being associated with a plurality of degrees;

displaying a user interface control in the user interface including an indication of the identified degree to which the operation is performed, the user interface control manipulable by a user via the user interface to further adjust the degree to which the operation is performed, the displaying the user interface control occurring simultaneously with performing the identified operation to the identified degree on the recognized object;

receiving, via the displayed user interface control, user input to adjust the identified degree to an adjusted degree;

adjusting the identified degree to the adjusted degree; and

adjusting the indication of the user interface control to indicate the adjusted degree.

2. A method as described in claim 1 , wherein the user interface control includes a marker as the indication of the identified degree.

3. A method as described in claim 1 , wherein the user interface control includes a plurality of additional indications corresponding to the plurality of the degrees to which the operation is performable, the plurality of additional indications displayed concurrently with the indication of the identified degree.

4. A method as described in claim 1 , wherein the user interface control is configured as a slider control or a radial dial.

5. A method as described in claim 1 , wherein the user interface control is manipulable by a user using a speech input.

6. A method as described in claim 1 , further comprising ceasing the outputting of the user interface control after a threshold amount of time has passed during which interaction with the user interface control has not occurred.

7. A method as described in claim 1 , further comprising locating the user interface control from a plurality of user interface controls that is associated with the operation.

8. A method as described in claim 1 , further comprising receiving one or more inputs generated through interaction with the user interface control and adjusting the identified degree based on the one or more inputs.

9. A method as described in claim 1 , further comprising setting the degree to which the operation is to be performed to a default level responsive to a determination that the natural language user input does not specify the degree.

10. One or more non-transitory computer-readable storage media comprising instructions stored thereon that, responsive to execution by a computing device, causes the computing device to perform operations comprising:

capturing, by an audio capture device of the computing device, audio speech data describing a natural language input;

converting, by a speech-to-text engine, the audio speech data into text;

identifying, by a natural language processing module implemented by a processing system and computer-readable storage medium of the computing device, in the converted text of the audio speech data:

an operation specified by the natural language input,

a numerical degree to which to perform the operation, and

an object in an image;

recognizing, by an object identification module implemented by the processing system and computer-readable storage medium of the computing device implementing an object recognition algorithm, the object in the image, wherein the object recognized by the computing device is a person, and the object recognition algorithm includes a facial recognition algorithm;

drawing, by the object identification module implemented by the processing system and computer-readable storage medium of the computing device, a user manipulable boundary in a user interface around the recognized object;

performing the operation on the object to the identified numerical degree;

finding, by a user interface control module, which of a plurality of user interface controls are associated with the operation;

displaying by a display device, the found user interface control, the found user interface control displaying an indication of the identified degree to which the operation is performed;

receiving, via the found user interface control, user input to adjust the identified degree to an adjusted degree;

adjusting the identified degree to the adjusted degree; and

adjusting the indication of the user interface control to indicate the adjusted degree.

11. One or more non-transitory computer-readable storage media as described in claim 10 , wherein the found user interface control is a slider.

12. One or more non-transitory computer-readable storage media as described in claim 10 , wherein the found user interface control is a radial dial.

13. One or more non-transitory computer-readable storage media as described in claim 10 , wherein the found user interface control is displayed separate from a portion of the user interface on which the operation is to be performed.

14. A system comprising:

a display device to display an image in a user interface;

an audio capture device to capture audio speech data describing a natural language input;

at least a memory and a processor configured to:

convert the audio speech data into text;

identify, in the converted text of the audio speech data describing the natural language input, an object included in the image, an operation to be performed on the object in the image, and a degree to which the operation is to be performed on the object in the image;

recognize the object in the image indicated in the natural language input, wherein the object recognized by the computing device is a person that is recognized using a facial recognition algorithm;

draw a user manipulable boundary in the user interface around the recognized object;

perform the identified operation to the identified degree on the recognized object, the operation being associated with a plurality of degrees;

cause display of a user interface control in the user interface including an indication of the identified degree to which the operation is performed, the user interface control manipulable by a user via the user interface to further adjust the degree to which the operation is performed, the display of the user interface control occurring simultaneously with performing the identified operation to the identified degree on the recognized object;

receive, via the displayed user interface control, user input to adjust the identified degree to an adjusted degree;

adjust the identified degree to the adjusted degree; and

adjust the indication of the user interface control to indicate the adjusted degree.

15. A system as described in claim 14 , wherein the user interface control includes a marker as the indication of the identified degree.

16. A system as described in claim 14 , wherein the user interface control includes a plurality of additional indications corresponding to the plurality of the degrees to which the operation is performable, the plurality of additional indications displayed concurrently with the indication of the identified degree.

17. A system as described in claim 14 , wherein the user interface control is configured as a slider control or a radial dial.

18. A system as described in claim 14 , wherein the user interface control is manipulable by a user using a speech input.

19. A system as described in claim 14 , wherein the memory and the processor are further configured to cease displaying the user interface control after a threshold amount of time has passed during which interaction with the user interface control has not occurred.

20. A system as described in claim 14 , wherein the memory and the processor are further configured to locate the user interface control from a plurality of user interface controls that is associated with the operation.

Assignments (2)
CHANGE OF NAME Recorded Jan 18, 2019
From: ADOBE SYSTEMS INCORPORATED
To: ADOBE INC.
Reel/Frame 048097/0414 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 21, 2013
From: WILENSKY, GREGG D.; CHANG, WALTER W.; DONTCHEVA, LUBOMIRA A.; LAPUT, GIERAD P.; AGARWALA, ASEEM O.
To: ADOBE SYSTEMS INCORPORATED
Reel/Frame 029849/0207 →