IP Library Granted Patent US 10,741,175
Granted Patent B2
US 10,741,175 · App. 15/365,243 · Granted Aug 11, 2020

Systems and methods for natural language understanding using sensor input

Inventors: Ming Qian (Cary, NC); Song Wang (Cary, NC); John Weldon Nicholson (Cary, NC); Jatinder Kumar (Cary, NC)
Assignee: Lenovo (Singapore) Pte. Ltd.
G10L15/22G06F3/017G06F3/0304G06K9/00302G06K9/00335G01S19/14G10L2015/227G10L2015/228
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,741,175
App. No.
15/365,243
Granted
Aug 11, 2020
Kind
B2
Abstract

In one aspect, a device includes a processor, a microphone accessible to the processor, at least a first sensor that is accessible to the processor, and storage accessible to the processor. The storage bears instructions executable by the processor to receive first input from the microphone that is generated based on audible input from a user. The instructions are also executable by the processor to receive second input from the first sensor, perform natural language understanding based on the first input, augment the natural language understanding based on the second input, and provide an output based on the augmentation.

Claims (53)

1. A device, comprising:

at least one processor;

a microphone accessible to the at least one processor;

at least a first sensor other than the microphone and that is accessible to the at least one processor; and

storage accessible to the at least one processor and bearing instructions executable by the at least one processor to:

receive first input from the microphone, the first input generated based on audible input from a user;

receive second input from the first sensor;

perform natural language understanding based on the first input;

augment the natural language understanding based on the second input;

provide an output based on the augmentation; and

present a user interface (UI) on an electronic display accessible to the at least one processor, wherein the UI comprises an option that is selectable to enable the device to perform respective augmentations to respective natural language understandings as respective natural language inputs are received, and wherein the UI comprises an option that is selectable to enable the device to perform respective augmentations to respective natural language understandings based on respective ambiguities in respective natural language understandings.

2. The device of claim 1 , wherein the first sensor is a camera, and wherein the instructions are executable by the at least one processor to:

perform gesture recognition based on input from the camera; and

augment the natural language understanding based on the gesture recognition.

3. The device of claim 1 , wherein the first sensor is a camera, and wherein the instructions are executable by the at least one processor to:

perform object recognition based on input from the camera; and

augment the natural language understanding based on the object recognition.

4. The device of claim 1 , wherein the first sensor is a camera, and wherein the instructions are executable by the at least one processor to:

perform facial expression recognition based on input from the camera; and

augment the natural language understanding based on the facial expression recognition.

5. The device of claim 1 , wherein the first sensor is a global positioning system (GPS) transceiver, and wherein the instructions are executable by the at least one processor to:

identify a current location of the device based on input from the GPS transceiver; and

augment the natural language understanding based on the current location.

6. The device of claim 1 , wherein the first sensor is an accelerometer, and wherein the instructions are executable by the at least one processor to:

identify movement based on input from the accelerometer; and

augment the natural language understanding based on the movement.

7. The device of claim 1 , wherein the output comprises one or more of: a warning for a person not to do something, and an instruction for the user to take an action that is identified based on the second input.

8. The device of claim 1 , wherein the instructions are executable by the at least one processor to:

augment the natural language understanding based on the second input and based on at least one application that is currently executing at the device.

9. The device of claim 1 , comprising the electronic display.

10. The device of claim 1 , wherein the options are different from each other.

11. The device of claim 1 , wherein the output comprises presentation of text on the electronic display, the text corresponding to the augmented natural language understanding.

12. The device of claim 11 , wherein the text comprises first text corresponding to one or more words determined from the natural language understanding itself, and wherein the text comprises second text corresponding to one or more words determined to be augmentations to the natural language understanding.

13. The device of claim 12 , wherein the first text is visually distinguished from the second text as presented on the electronic display.

14. The device of claim 13 , wherein the second text is presented one or more of: within parentheses to visually distinguish the second text from the first text, in quotation marks to visually distinguish the first text from the second text.

15. A method, comprising:

receiving, at a device, natural language input from a user;

performing natural language understanding based on the natural language input to generate a natural language output;

receiving input from a sensor;

altering the natural language output based on the input from the sensor;

performing at least one action at the device based on the altered natural language output; and

presenting a user interface (UI) on an electronic display, wherein the UI comprises an option that is selectable to enable the device to perform respective alterations to respective natural language outputs as respective natural language inputs are received, and wherein the UI comprises an option that is selectable to enable the device to perform respective alterations to respective natural language outputs based on respective ambiguities in respective natural language understandings.

16. The method of claim 15 , wherein the UI comprises an option that is selectable to enable the device to perform respective alterations to respective natural language understandings based on current time of day.

17. The method of claim 15 , wherein the UI comprises an option that is selectable to enable the device to perform respective alterations to respective natural language understandings based on camera input.

18. The method of claim 15 , wherein the UI comprises an option that is selectable to enable the device to perform respective alterations to respective natural language understandings based on current location of the device.

19. The method of claim 15 , wherein the UI comprises an option that is selectable a single time to enable plural respective future alterations of respective natural language understandings based on respective sensor inputs.

20. A computer readable storage medium (CRSM) that is not a transitory signal, the computer readable storage medium comprising instructions executable by at least one processor to:

receive first input from a microphone accessible to the at least one processor, the first input being generated based on audible input from a user;

receive second input from a camera;

perform natural language understanding based on the first input;

augment the natural language understanding based on the second input from the camera to generate a natural language output;

execute at least one task based on the natural language output; and

present a user interface (UI) on a display accessible to the at least one processor, wherein the UI comprises an option that is selectable to perform augmentation of natural language understanding proactively as natural language input is received, and wherein the UI comprises an option that is selectable to perform augmentation of natural language understanding based on an ambiguity in natural language understanding.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 10, 2025
From: LENOVO PC INTERNATIONAL LIMITED
To: LENOVO SWITZERLAND INTERNATIONAL GMBH
Reel/Frame 069870/0670 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 14, 2022
From: LENOVO (SINGAPORE) PTE LTD
To: LENOVO PC INTERNATIONAL LIMITED
Reel/Frame 060651/0634 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 30, 2016
From: QIAN, MING; WANG, SONG; NICHOLSON, JOHN WELDON; KUMAR, JATINDER
To: LENOVO (SINGAPORE) PTE. LTD.
Reel/Frame 040469/0742 →