IP Library › Granted Patent US 12,399,560
Granted Patent B2
US 12,399,560 · App. 18/793,290 · Granted Aug 26, 2025

Natural human-computer interaction for virtual personal assistant systems

Inventor: William C. Deleeuw (Beaverton, OR)
Assignee: INTEL CORPORATION
G06F3/013G06T13/80G06V10/143G06V40/19G10L15/02G10L15/22G10L2015/025G10L2015/227G10L2015/228G10L15/30G10L21/003
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,399,560
App. No.
18/793,290
Granted
Aug 26, 2025
Kind
B2
Abstract

Technologies for natural language interactions with virtual personal assistant systems include a computing device configured to capture audio input, distort the audio input to produce a number of distorted audio variations, and perform speech recognition on the audio input and the distorted audio variants. The computing device selects a result from a large number of potential speech recognition results based on contextual information. The computing device may measure a user's engagement level by using an eye tracking sensor to determine whether the user is visually focused on an avatar rendered by the virtual personal assistant. The avatar may be rendered in a disengaged state, a ready state, or an engaged state based on the user engagement level. The avatar may be rendered as semitransparent in the disengaged state, and the transparency may be reduced in the ready state or the engaged state. Other embodiments are described and claimed.

Claims (37)

1. An electronic user device comprising:

a screen;

machine-readable instructions; and

at least one processor circuit to be programmed by the machine-readable instructions to at least:

cause the screen to present an avatar in a first position, the avatar having eyes, a gaze of the eyes of the avatar in a first direction relative to a user;

detect activation of an application installed on the electronic user device; and

cause the screen to present the avatar in a second position relative to an image associated with the application, the gaze of the eyes of the avatar in a second direction relative to the user, the second direction different than the first direction.

2. The electronic user device of claim 1 , wherein a face of the avatar includes human facial features.

3. The electronic user device of claim 1 , wherein one or more of the at least one processor circuit is to cause the screen to present the avatar as facing toward the user in the first position.

4. The electronic user device of claim 1 , wherein the gaze of the eyes of the avatar in the second direction is directed away from the user.

5. The electronic user device of claim 1 , wherein one or more of the at least one processor circuit is to cause the screen to present the avatar spaced apart from the image.

6. The electronic user device of claim 1 , wherein one or more of the at least one processor circuit is to cause the screen to present content with the avatar when the avatar is in the first position.

7. The electronic user device of claim 6 , wherein one or more of the at least one processor circuit is to cause the screen to present the avatar in closer proximity to the content when the avatar is in the first position than to the image when the avatar is in the second position.

8. The electronic user device of claim 1 , further including a microphone.

9. At least one non-transitory machine-readable storage medium comprising machine-readable instructions to cause at least one processor circuit of an electronic user device to at least:

cause a screen of the electronic user device to present an avatar, the avatar in a first position, the avatar having eyes, a gaze of the eyes of the avatar in a first direction relative to a user;

detect a user input associated with an application installed on the electronic user device; and

responsive to the user input, cause the screen to present the avatar in a second position relative to content associated with the application, the gaze of the eyes of the avatar in a second direction relative to the user, the second direction different than the first direction.

10. The at least one non-transitory machine-readable storage medium of claim 9 , wherein the avatar has a face including human features.

11. The at least one non-transitory machine-readable storage medium of claim 9 , wherein the machine-readable instructions are to cause one or more of the at least one processor circuit to cause the screen to present other content with the avatar when the avatar is in the first position.

12. The at least one non-transitory machine-readable storage medium of claim 11 , wherein a face of the avatar is opaque.

13. The at least one non-transitory machine-readable storage medium of claim 9 , wherein the gaze of the eyes of the avatar in the second direction is directed away from the user.

14. The at least one non-transitory machine-readable storage medium of claim 9 , wherein one or more of the at least one processor circuit is to cause the screen to present the avatar as facing out of the screen in the first position.

15. An electronic user device comprising:

a screen;

machine-readable instructions;

at least one processor circuit to be programmed by the machine-readable instructions to:

cause the screen to present an avatar in a first position relative to content associated with a first application installed on the electronic user device;

detect a user input associated with a second application installed on the electronic user device; and

responsive to the user input, cause the screen to present the avatar in a second position relative to content associated with the second application, the first position of the avatar in closer proximity to the content associated with the first application than the proximity of the avatar to the content associated with the second application in the second position.

16. The electronic user device of claim 15 , wherein the avatar has eyes and one or more of the at least one processor circuit is to:

cause the screen to present a gaze of the eyes of the avatar in a first direction relative to a user when the avatar is in the first position; and

cause the screen to present the gaze of the eyes of the avatar in a second direction relative to the user when the avatar is in the second position, the second direction different than the first position.

17. The electronic user device of claim 16 , wherein one or more of the at least one processor circuit is to cause the screen to present the gaze of the eyes of the avatar toward the user in the first direction.

18. The electronic user device of claim 15 , further including a sensor to track an eye of a user.

19. The electronic user device of claim 15 , wherein the avatar has a human face appearance.

20. The electronic user device of claim 19 , wherein the face of the avatar is opaque.

Continuity (7)
Continuation 18186680 · Mar 20, 2023
Continuation 17532816 · Nov 22, 2021
Continuation 16902967 · Jun 16, 2020
Continuation 16257347 · Jan 25, 2019
Continuation 15452131 · Mar 7, 2017
Division 14129435
Related Publication 20240411365A1 · Dec 12, 2024
References Cited (149)
US 5799276A · Komissarchik · 1998 [cited by applicant]
US 5809471A · Brodsky · 1998 [cited by applicant]
US 6317711B1 · Muroi · 2001 [cited by applicant]
US 6360202B1 · Bhadkamkar · 2002 [cited by applicant]
US 6793673B2 · Kowalsky · 2004 [cited by applicant]
US 6824562B2 · Mathis · 2004 [cited by applicant]
US 6908478B2 · Alferness · 2005 [cited by applicant]
US 6949122B2 · Adams · 2005 [cited by applicant]
US 6960229B2 · Mathis · 2005 [cited by applicant]
US 6964683B2 · Kowalsky · 2005 [cited by applicant]
US 6976995B2 · Mathis · 2005 [cited by applicant]
US 7179282B2 · Alferness · 2007 [cited by applicant]
US 7309354B2 · Mathis · 2007 [cited by applicant]
US 7311729B2 · Mathis · 2007 [cited by applicant]
US 7316708B2 · Gordon · 2008 [cited by applicant]
US 7351260B2 · Nieminen · 2008 [cited by applicant]
US 7503931B2 · Kowalsky · 2009 [cited by applicant]
US 7608102B2 · Adams · 2009 [cited by applicant]
US 7635387B2 · Reuter · 2009 [cited by applicant]
US 7674287B2 · Alferness · 2010 [cited by applicant]
US 7828841B2 · Mathis · 2010 [cited by applicant]
US 7828842B2 · Nieminen · 2010 [cited by applicant]
US 7837729B2 · Gordon · 2010 [cited by applicant]
US 7857846B2 · Alferness · 2010 [cited by applicant]
US 7966177B2 · Geiger · 2011 [cited by applicant]
US 8062358B2 · Mathis · 2011 [cited by applicant]
US 8075608B2 · Gordon · 2011 [cited by applicant]
US 8172898B2 · Alferness · 2012 [cited by applicant]
US 8182529B2 · Gordon · 2012 [cited by applicant]
US 8223156B2 · Boss · 2012 [cited by applicant]
US 8439971B2 · Reuter · 2013 [cited by applicant]
US 8990079B1 · Newman · 2015 [cited by applicant]
US 9607612B2 · Deleeuw · 2017 [cited by applicant]
US 10022011B2 · Norton · 2018 [cited by applicant]
US 10062004B2 · Sugahara · 2018 [cited by applicant]
US 10198068B2 · Holz · 2019 [cited by applicant]
US 10198069B2 · Deleeuw · 2019 [cited by applicant]
US 10684683B2 · Deleeuw · 2020 [cited by applicant]
US 11181980B2 · Deleeuw · 2021 [cited by applicant]
US 11609631B2 · Deleeuw · 2023 [cited by applicant]
US 12099651B2 · Deleeuw · 2024 [cited by applicant]
US 20030083538A1 · Adams · 2003 [cited by applicant]
US 20030105520A1 · Alferness · 2003 [cited by applicant]
US 20030144697A1 · Mathis · 2003 [cited by applicant]
US 20030149566A1 · Levin · 2003 [cited by applicant]
US 20030212453A1 · Mathis · 2003 [cited by applicant]
US 20030225454A1 · Mathis · 2003 [cited by applicant]
US 20030236569A1 · Mathis · 2003 [cited by applicant]
US 20030236661A1 · Burges · 2003 [cited by applicant]
US 20040010305A1 · Alferness · 2004 [cited by applicant]
US 20040111095A1 · Gordon · 2004 [cited by applicant]
US 20040127980A1 · Kowalsky · 2004 [cited by applicant]
US 20040193260A1 · Alferness · 2004 [cited by applicant]
US 20040243228A1 · Kowalsky · 2004 [cited by applicant]
US 20040249452A1 · Adams · 2004 [cited by applicant]
US 20050004799A1 · Lyudovyk · 2005 [cited by applicant]
US 20050021121A1 · Reuter · 2005 [cited by applicant]
US 20050033582A1 · Gadd · 2005 [cited by applicant]
US 20050096666A1 · Gordon · 2005 [cited by applicant]
US 20050119673A1 · Gordon · 2005 [cited by applicant]
US 20050125229A1 · Kurzweil · 2005 [cited by applicant]
US 20050149180A1 · Mathis · 2005 [cited by applicant]
US 20050187619A1 · Mathis · 2005 [cited by applicant]
US 20050209690A1 · Mathis · 2005 [cited by applicant]
US 20050216077A1 · Mathis · 2005 [cited by applicant]
US 20050272969A1 · Alferness · 2005 [cited by applicant]
US 20060020335A1 · Kowalsky · 2006 [cited by applicant]
US 20060142854A1 · Alferness · 2006 [cited by applicant]
US 20060145944A1 · Tarlton · 2006 [cited by applicant]
US 20060167544A1 · Nieminen · 2006 [cited by applicant]
US 20060173536A1 · Mathis · 2006 [cited by applicant]
US 20070038451A1 · Cogne · 2007 [cited by applicant]
US 20070055293A1 · Alferness · 2007 [cited by applicant]
US 20070066879A1 · Mathis · 2007 [cited by applicant]
US 20070225972A1 · Kim · 2007 [cited by applicant]
US 20080096533A1 · Manfredi · 2008 [cited by applicant]
US 20080097594A1 · Mathis · 2008 [cited by applicant]
US 20080109059A1 · Gordon · 2008 [cited by applicant]
US 20080140191A1 · Mathis · 2008 [cited by applicant]
US 20080319542A1 · Nieminen · 2008 [cited by applicant]
US 20090150156A1 · Kennewick · 2009 [cited by applicant]
US 20090271189A1 · Agapi · 2009 [cited by applicant]
US 20100079467A1 · Boss · 2010 [cited by applicant]
US 20100100175A1 · Reuter · 2010 [cited by applicant]
US 20100156781A1 · Fahn · 2010 [cited by applicant]
US 20100168847A1 · Alferness · 2010 [cited by applicant]
US 20110015928A1 · Odell · 2011 [cited by applicant]
US 20110035000A1 · Nieminen · 2011 [cited by applicant]
US 20110066234A1 · Gordon · 2011 [cited by applicant]
US 20110175810A1 · Markovic · 2011 [cited by applicant]
US 20120154277A1 · Bar-Zeev · 2012 [cited by applicant]
US 20120257035A1 · Larsen · 2012 [cited by applicant]
US 20120295708A1 · Hernandez-Abrego · 2012 [cited by applicant]
US 20130005443A1 · Kosta · 2013 [cited by applicant]
US 20130005458A1 · Kosta · 2013 [cited by applicant]
US 20130005482A1 · Kosta · 2013 [cited by applicant]
US 20130006709A1 · Kosta · 2013 [cited by applicant]
US 20140184550A1 · Hennessey · 2014 [cited by applicant]
US 20140225901A1 · Shuster · 2014 [cited by applicant]
US 20140267311A1 · Evertt · 2014 [cited by applicant]
US 20160063989A1 · Deleeuw · 2016 [cited by applicant]
US 20160210503A1 · Yin · 2016 [cited by applicant]
US 20160260436A1 · Lemay · 2016 [cited by applicant]
US 20170039751A1 · Tong · 2017 [cited by applicant]
US 20170177080A1 · Deleeuw · 2017 [cited by applicant]
US 20190187787A1 · White · 2019 [cited by applicant]
US 20190227623A1 · Deleeuw · 2019 [cited by applicant]
US 20190369748A1 · Hindi · 2019 [cited by applicant]
US 20210072825A1 · Deleeuw · 2021 [cited by applicant]
CN 1134074A · 1996 [cited by applicant]
CN 1164801A · 1997 [cited by applicant]
CN 1343968A · 2002 [cited by applicant]
CN 101292282A · 2008 [cited by applicant]
CN 101939740A · 2011 [cited by applicant]
CN 101952818A · 2011 [cited by applicant]
CN 102129292A · 2011 [cited by applicant]
CN 102591016A · 2012 [cited by applicant]
CN 102749991A · 2012 [cited by applicant]
CN 102814814A · 2012 [cited by applicant]
JP 2012221498A · 2012 [cited by applicant]
“Intelligent personal assistant,” Wikipedia, The Free Encyclopedia, retrieved from: http://en.wikipedia.org/w/index.phptitle=Intelligent.sub.-personal.-sub.--assistant oldid=520490140, edited Oct. 29, 2012, 2 pages. [cited by applicant]
“Automated online assistant,” Wikipedia, The Free Encyclopedia, retrieved from http://en.wikipedia.org/w/index.phptitle=Automated.sub.-onlin-e.sub.-assistant_oldid=521739395, edited Nov. 6, 2012, 3 pages. [cited by applicant]
“Eye tracking,” Wikipedia, The Free Encyclopedia, retrieved from: http://en.wikipedia.org/w/index.phptitle=Eye.sub-tracking_oldid=52-3123367, edited Nov. 15, 2012, 7 pages. [cited by applicant]
“Speech recognition,” Wikipedia, The Free Encyclopedia, retrieved from: http://en.wikipedia.org/w/index.phptitle=Speech.sub.-recognition ol- did=528092164, edited Dec. 15, 2012, 12 pages. [cited by applicant]
International Searching Authority, “International Search Report and Written Opinion,” issued in connection with International Patent Application No. PCT/US2013/041866, dated Feb. 28, 2014, 19 pages. [cited by applicant]
The International Bureau, “International Preliminary Report on Patentability,” issued in connection with International Patent Application No. PCT/US2013/041866, dated Nov. 24, 2015, 12 pages. [cited by applicant]
United States Patent and Trademark Office, “Non-Final Office Action,” issued in connection with U.S. Appl. No. 14/129,435, dated Feb. 16, 2016, 9 pages. [cited by applicant]
United States Patent and Trademark Office, “Final Office Action,” issued in connection with U.S. Appl. No. 14/129,435, dated Jun. 23, 2016, 16 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due”, issued in connection with U.S. Appl. No. 14/129,435, dated Nov. 16, 2016, 13 pages. [cited by applicant]
Chinese Patent Office, English translation of “Search Report,” issued in connection with Chinese Patent Application No. 2013800757640 dated Mar. 14, 2018, 2 pages. [cited by applicant]
Chinese Patent Office, “First Office Action,” issued in connection with Chinese Patent Application No. 201380075764, dated Mar. 22, 2018, 24 pages (includes English translation). [cited by applicant]
United States Patent and Trademark Office, “Non-Final Office Action,” issued in connection with U.S. Appl. No. 15/452,131, dated May 17, 2018, 6 pages. [cited by applicant]
Chinese Patent Office, English translation of “Supplemental Search Report,” issued in connection with Chinese Patent Application No. 2013800757640 dated Aug. 8, 2018, 2 pages (includes English translation). [cited by applicant]
Chinese Patent Office, English translation of “Second Office Action,” issued in connection with Chinese Patent Application No. 201380075764, dated Sep. 5, 2018, 5 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” issued in connection with U.S. Appl. No. 15/452,131, dated Sep. 25, 2018, 7 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” issued in connection with U.S. Appl. No. 15/452,131, dated Oct. 17, 2018, 2 pages. [cited by applicant]
Chinese Patent Office, “Notification to Grant Patent Right for Invention,” issued in connection with Chinese Patent Application No. 2013800757640, dated Mar. 25, 2019, 4 pages (includes English translation). [cited by applicant]
United States Patent and Trademark Office, “Non-Final Office Action,” issued in connection with U.S. Appl. No. 16/257,347, dated May 2, 2019, 6 pages. [cited by applicant]
United States Patent and Trademark Office, “Final Office Action,” issued in connection with U.S. Appl. No. 16/257,347, dated Oct. 29, 2019, 7 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” issued in connection with U.S. Appl. No. 16/257,347, dated Feb. 12, 2020, 7 pages. [cited by applicant]
United States Patent and Trademark Office, “Corrected Notice of Allowabilty,” issued in connection with U.S. Appl. No. 16/257,347, dated Apr. 9, 2020, 2 pages. [cited by applicant]
United States Patent and Trademark Office, “Non-Final Office Action,” issued in connection with U.S. Appl. No. 16/902,967, dated Mar. 22, 2021, 10 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) due,” issued in connection with U.S. Appl. No. 16/902,967, dated Jul. 12, 2021, 7 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance,” issued in U.S. Appl. No. 16/902,967 on Oct. 20, 2021, 2 pages. [cited by applicant]
United States Patent and Trademark Office, “Non-Final Action” issued in U.S. Appl. No. 17/532,816 on Jun. 17, 2022 (8 pages). [cited by applicant]
National Intellectual Property Administration, P.R. China, “Notification to Grant Patent Right for Invention,” issued in connection with Chinese Patent Application No. 201811329915.1, dated Sep. 8, 2022, 7 pages (Englis… [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance” issued in U.S. Appl. No. 17/532,816 on Nov. 16, 2022 (8 pages). [cited by applicant]
United States Patent and Trademark Office, “Non-Final Office Action,” issued in connection with U.S. Appl. No. 18/186,680, dated Feb. 1, 2024, 8 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” issued in connection with U.S. Appl. No. 18/186,680, dated May 24, 2024, 7 pages. [cited by applicant]