IP Library › Granted Patent US 12,243,523
Granted Patent B2
US 12,243,523 · App. 17/944,110 · Granted Mar 4, 2025

Digital assistant for providing handsfree notification management

Inventors: Andrea Valentina Simes (San Francisco, CA); Daniel W. Loo (Seattle, WA); Felicia W. Edwards (Loveland, CA); Harry M. Simmonds (Calgary, CA); Tyler R. Calderone (Boulder, CO); Kevin D. Pitolin (San Jose, CA); Lorena S. Pazmino (San Francisco, CA)
Assignee: Apple Inc.
G10L15/22G06F3/013G10L2015/223
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,243,523
App. No.
17/944,110
Granted
Mar 4, 2025
Kind
B2
Abstract

This relates generally to intelligent automated assistants and, more specifically, to provide a handsfree notification management system. An example method includes displaying one or more notifications by the electronic device. In response to displaying the one or more notifications, the method includes detecting a visual interaction of a user with the one or more notifications, identifying a notification from the one or more notifications based on the visual interaction, receiving a speech input related to the notification from the user, determining one or more actions associated with the notification based on the speech input, and performing the one or more actions.

Claims (139)

1. A method performed on an electronic device, the method comprising:

displaying one or more notifications by the electronic device;

in response to displaying the one or more notifications:

detecting a visual interaction of a user with the one or more notifications;

identifying a notification from the one or more notifications based on the visual interaction;

receiving a speech input related to the notification from the user;

determining one or more actions associated with the notification based on the speech input; and

performing the one or more actions, wherein performing the one or more actions comprises:

determining whether the one or more actions are associated with a notification system or an application associated with the notification; and

in response to determining that the one or more actions are associated with the notification system:

invoking the notification system to perform the one or more actions based on determining that the one or more actions are associated with the notification of the notification system.

2. The method of claim 1 , further comprising:

in response to identifying the notification from the one or more notifications based on the visual interaction:

determining time-interval of the visual interaction with the notification;

determining that the time-interval exceeds at least one predetermined threshold;

identifying one or more actions associated with the notification based on determining that the time-interval exceeds the at least one predetermined threshold; and

performing the one or more actions.

3. The method of claim 1 , wherein each of the one or more notifications is an alert of an event associated with the electronic device.

4. The method of claim 1 , wherein detecting the visual interaction of the user with the one or more notifications is based on a gaze direction of the user's eyes.

5. The method of claim 1 , wherein identifying the notification from the one or more notifications based on the visual interaction comprises:

determining a gaze direction of the user directed at the one or more notifications;

determining a time-interval of the gazing direction of the user; and

selecting the notification from the one or more notifications based on the time-interval.

6. The method of claim 1 , wherein the speech input is a natural language input from the user in form of a spoken command presented at the electronic device.

7. The method of claim 1 , wherein determining one or more actions associated with the notification based on the speech input comprises:

parsing the speech input;

determining that the speech input contains one or more ambiguous references;

resolving the one or more ambiguous references based on a content of the notification; and

identifying one or more actions based on resolving the one or more ambiguous references and the speech input.

8. The method of claim 1 , wherein determining one or more actions associated with the notification based on the speech input comprises:

parsing the speech input; and

identifying the one or more actions based on parsing the speech input.

9. The method of claim 1 , wherein the speech input includes a wake word for enabling a digital assistant associated with the electronic device for identifying the one or more actions.

10. The method of claim 1 , further comprising:

in response to determining that the one or more actions are associated with the notification system:

determining that the one or more actions are associated with manipulating one or more user interface elements of the notification; and

manipulating the one or more user interface elements to perform the one or more actions.

11. The method of claim 1 , wherein performing the one or more actions comprises:

in response to determining that the one or more actions are associated with the application:

invoking the application to perform the one or more actions based on determining that the one or more action is associated with the notification of the notification system.

12. The method of claim 11 , further comprising:

in response to determining that the one or more actions are associated with the application:

identifying an entity associated with the notification; and

invoking the application to perform the one or more actions involving the entity to change a state of the entity associated with the notification of the notification system.

13. The method of claim 12 , wherein the one or more notifications are displayed on a portion of a user interface of the electronic device.

14. An electronic device comprising:

one or more processors;

a memory; and

one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for:

displaying one or more notifications by the electronic device;

in response to displaying the one or more notifications:

detecting a visual interaction of a user with the one or more notifications;

identifying a notification from the one or more notifications based on the visual interaction;

receiving a speech input related to the notification from the user;

determining one or more actions associated with the notification based on the speech input; and

performing the one or more actions, wherein performing the one or more actions comprises:

determining whether the one or more actions are associated with a notification system or an application associated with the notification; and

in response to determining that the one or more actions are associated with the notification system:

 invoking the notification system to perform the one or more actions based on determining that the one or more actions are associated with the notification of the notification system.

15. A non-transitory computer-readable storage medium storing

one or more programs, the one or more programs including instructions, which when executed by one or more processors of an electronic device, cause the electronic device to:

display one or more notifications by the electronic device;

in response to displaying the one or more notifications:

detect a visual interaction of a user with the one or more notifications;

identify a notification from the one or more notifications based on the visual interaction;

receive a speech input related to the notification from the user;

determine one or more actions associated with the notification based on the speech input; and

perform the one or more actions, wherein performing the one or more actions comprises:

determining whether the one or more actions are associated with a notification system or an application associated with the notification; and

in response to determining that the one or more actions are associated with the notification system:

 invoking the notification system to perform the one or more actions based on determining that the one or more actions are associated with the notification of the notification system.

16. The electronic device of claim 14 , wherein the one or more programs further include instructions for:

in response to identifying the notification from the one or more notifications based on the visual interaction:

determining time-interval of the visual interaction with the notification;

determining that the time-interval exceeds at least one predetermined threshold;

identifying one or more actions associated with the notification based on determining that the time-interval exceeds the at least one predetermined threshold; and

performing the one or more actions.

17. The electronic device of claim 14 , wherein each of the one or more notifications is an alert of an event associated with the electronic device.

18. The electronic device of claim 14 , wherein detecting the visual interaction of the user with the one or more notifications is based on a gaze direction of the user's eyes.

19. The electronic device of claim 14 , wherein identifying the notification from the one or more notifications based on the visual interaction comprises:

determining a gaze direction of the user directed at the one or more notifications;

determining a time-interval of the gazing direction of the user; and

selecting the notification from the one or more notifications based on the time-interval.

20. The electronic device of claim 14 , wherein the speech input is a natural language input from the user in form of a spoken command presented at the electronic device.

21. The electronic device of claim 14 , wherein determining one or more actions associated with the notification based on the speech input comprises:

parsing the speech input;

determining that the speech input contains one or more ambiguous references;

resolving the one or more ambiguous references based on a content of the notification; and

identifying one or more actions based on resolving the one or more ambiguous references and the speech input.

22. The electronic device of claim 14 , wherein determining one or more actions associated with the notification based on the speech input comprises:

parsing the speech input; and

identifying the one or more actions based on parsing the speech input.

23. The electronic device of claim 14 , wherein the speech input includes a wake word for enabling a digital assistant associated with the electronic device for identifying the one or more actions.

24. The electronic device of claim 14 , wherein the one or more programs further include instructions for:

in response to determining that the one or more actions are associated with the notification system:

determining that the one or more actions are associated with manipulating one or more user interface elements of the notification; and

manipulating the one or more user interface elements to perform the one or more actions.

25. The electronic device of claim 14 , wherein performing the one or more actions comprises:

in response to determining that the one or more actions are associated with the application:

invoking the application to perform the one or more actions based on determining that the one or more action is associated with the notification of the notification system.

26. The electronic device of claim 25 , wherein the one or more programs further include instructions for:

in response to determining that the one or more actions are associated with the application:

identifying an entity associated with the notification; and

invoking the application to perform the one or more actions involving the entity to change a state of the entity associated with the notification of the notification system.

27. The electronic device of claim 26 , wherein the one or more notifications are displayed on a portion of a user interface of the electronic device.

28. The non-transitory computer-readable storage medium of claim 15 , wherein the one or more programs further include instructions, which when executed by the one or more processors of the electronic device, cause the electronic device to:

in response to identifying the notification from the one or more notifications based on the visual interaction:

determine time-interval of the visual interaction with the notification;

determine that the time-interval exceeds at least one predetermined threshold;

identify one or more actions associated with the notification based on determining that the time-interval exceeds the at least one predetermined threshold; and

perform the one or more actions.

29. The non-transitory computer-readable storage medium of claim 15 , wherein each of the one or more notifications is an alert of an event associated with the electronic device.

30. The non-transitory computer-readable storage medium of claim 15 , wherein detecting the visual interaction of the user with the one or more notifications is based on a gaze direction of the user's eyes.

31. The non-transitory computer-readable storage medium of claim 15 , wherein identifying the notification from the one or more notifications based on the visual interaction comprises:

determining a gaze direction of the user directed at the one or more notifications;

determining a time-interval of the gazing direction of the user; and

selecting the notification from the one or more notifications based on the time-interval.

32. The non-transitory computer-readable storage medium of claim 15 , wherein the speech input is a natural language input from the user in form of a spoken command presented at the electronic device.

33. The non-transitory computer-readable storage medium of claim 15 , wherein determining one or more actions associated with the notification based on the speech input comprises:

parsing the speech input;

determining that the speech input contains one or more ambiguous references;

resolving the one or more ambiguous references based on a content of the notification; and

identifying one or more actions based on resolving the one or more ambiguous references and the speech input.

34. The non-transitory computer-readable storage medium of claim 15 , wherein determining one or more actions associated with the notification based on the speech input comprises:

parsing the speech input; and

identifying the one or more actions based on parsing the speech input.

35. The non-transitory computer-readable storage medium of claim 15 , wherein the speech input includes a wake word for enabling a digital assistant associated with the electronic device for identifying the one or more actions.

36. The non-transitory computer-readable storage medium of claim 15 , wherein the one or more programs further include instructions, which when executed by the one or more processors of the electronic device, cause the electronic device to:

in response to determining that the one or more actions are associated with the notification system:

determine that the one or more actions are associated with manipulating one or more user interface elements of the notification; and

manipulate the one or more user interface elements to perform the one or more actions.

37. The non-transitory computer-readable storage medium of claim 15 , wherein performing the one or more actions comprises:

in response to determining that the one or more actions are associated with the application:

invoking the application to perform the one or more actions based on determining that the one or more action is associated with the notification of the notification system.

38. The non-transitory computer-readable storage medium of claim 37 , wherein the one or more programs further include instructions, which when executed by the one or more processors of the electronic device, cause the electronic device to:

in response to determining that the one or more actions are associated with the application:

identify an entity associated with the notification; and

invoke the application to perform the one or more actions involving the entity to change a state of the entity associated with the notification of the notification system.

39. The non-transitory computer-readable storage medium of claim 38 , wherein the one or more notifications are displayed on a portion of a user interface of the electronic device.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 7, 2022
From: PITOLIN, KEVIN D.; PAZMINO, LORENA S.; SIMES, ANDREA VALENTINA; CALDERONE, TYLER R.; EDWARDS, FELICIA W.; SIMMONDS, HARRY M.; LOO, DANIEL W.
To: APPLE INC.
Reel/Frame 061680/0554 →
Continuity (3)
Provisional Application 63345384 · May 24, 2022
Provisional Application 63248315 · Sep 24, 2021
Related Publication 20230098174A1 · Mar 30, 2023
References Cited (29)
US 9818400B2 · Paulik et al. · 2017 [cited by applicant]
US 9966065B2 · Gruber et al. · 2018 [cited by applicant]
US 10147421B2 · Liddell et al. · 2018 [cited by applicant]
US 10497365B2 · Gruber et al. · 2019 [cited by applicant]
US 10679608B2 · Mixter et al. · 2020 [cited by applicant]
US 11093306B1 · Libin · 2021 [cited by examiner]
US 20120089659A1 · Halevi et al. · 2012 [cited by applicant]
US 20150130716A1 · Sridharan · 2015 [cited by examiner]
US 20150348551A1 · Gruber et al. · 2015 [cited by applicant]
US 20160078860A1 · Paulik et al. · 2016 [cited by applicant]
US 20160350812A1 · Priness et al. · 2016 [cited by applicant]
US 20160371054A1 · Beaumont · 2016 [cited by examiner]
US 20180108346A1 · Paulik et al. · 2018 [cited by applicant]
US 20180350353A1 · Gruber et al. · 2018 [cited by applicant]
US 20200043482A1 · Gruber et al. · 2020 [cited by applicant]
US 20200357406A1 · York et al. · 2020 [cited by applicant]
US 20210241468A1 · Zhang · 2021 [cited by examiner]
US 20220068278A1 · York et al. · 2022 [cited by applicant]
US 20230070209A1 · Srinivasan · 2023 [cited by examiner]
EP 2575128A2 · 2013 [cited by applicant]
EP 2741175A2 · 2014 [cited by applicant]
IN 1537MUM2011A · 2011 [cited by applicant]
JP 201380476A · 2013 [cited by applicant]
WO 2015184186A1 · 2015 [cited by applicant]
Coulouris et al., “Distributed Systems: Concepts and Design (Fifth Edition)”, Addison-Wesley, May 7, 2011, 391 pages. [cited by applicant]
Hutsko et al., “iPhone All-in-One For Dummies”, 3rd Edition, Dec. 31, 2013, 98 pages. [cited by applicant]
Zhong et al., “JustSpeak: Enabling Universal Voice Control on Android”, W4A'14, Proceedings of the 11th Web for All Conference, No. 36, Apr. 7-9, 2014, 8 pages. [cited by applicant]
International Preliminary Report on Patentability received for PCT Patent Application No. PCT/US2022/044183, mailed on Apr. 4, 2024. 9 pages. [cited by applicant]
International Search Report and Written Opinion received for PCT Patent Application No. PCT/US2022/044183, mailed on Jan. 9, 2023, 12 pages. [cited by applicant]
Cited By (1)
US 12,464,228