IP Library Granted Patent US 12,437,747
Granted Patent B2
US 12,437,747 · App. 19/014,030 · Granted Oct 7, 2025

Digital assistant for providing and modifying an output of an electronic document

Inventors: Daniel A. Castellani (San Jose, CA); Didier Guzzoni (Mont-sur-Rolle, CH); Pierre-Louis Jallerat (San Francisco, CA)
Assignee: Apple Inc.
G10L13/10G10L13/0335G10L2013/105
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,437,747
App. No.
19/014,030
Granted
Oct 7, 2025
Kind
B2
Abstract

Systems and processes for providing and modifying an output of an electronic document using a digital assistant of an electronic device are provided. An example method comprises receiving, at the first electronic device, a message, wherein the message includes an electronic document including text; detecting, by a digital assistant, the electronic document in the received message; prompting, by the digital assistant, a user to provide a user input; receiving the user input, wherein the user input is associated with an intent to provide an audio output of the electronic document; and in accordance with a determination that the received user input is associated with an intent to provide the audio output of the electronic document: generating a media item based on the text of the electronic document; and after generating the media item, outputting the media item.

Claims (58)

1. An electronic device, comprising:

a display;

one or more processors;

a memory; and

one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the one or more programs including instructions for:

receiving, at the electronic device, a message, wherein the message includes an electronic document including text;

detecting, based on at least one keyword in the message and by a digital assistant, the electronic document in the received message;

prompting, by the digital assistant, a user to provide a user input, wherein prompting the user to provide the user input comprises displaying the message concurrently with a text prompt associated with a suggestion to provide an audio output of the electronic document;

receiving the user input, wherein the user input is associated with an intent to provide an audio output of the electronic document; and

in accordance with a determination that the received user input is associated with an intent to provide the audio output of the electronic document:

generating a media item based on the text of the electronic document; and

after generating the media item, audibly outputting the media item.

2. The electronic device of claim 1 , wherein the message includes a hyperlink to the electronic document.

3. The electronic device of claim 1 , wherein prompting a user to provide a user input further comprises:

providing an auditory response associated with a suggestion to provide an audio output of the electronic document.

4. The electronic device of claim 1 , wherein the user input is a physical input.

5. The electronic device of claim 1 , wherein the user input is a speech input.

6. The electronic device of claim 1 , wherein the one or more programs further include instructions for:

in accordance with a determination that the received user input is associated with an intent to save the electronic document to a user's reading list:

saving the electronic document to the user's reading list.

7. The electronic device of claim 1 , wherein in accordance with the determination that the received user input is associated with the intent to provide the audio output of the electronic document further comprises:

after generating the media item, displaying the electronic document.

8. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of an electronic device, cause the electronic device to:

receive, at the electronic device, a message, wherein the message includes an electronic document including text;

detect, based on at least one keyword in the message and by a digital assistant, the electronic document in the received message;

prompt, by the digital assistant, a user to provide a user input, wherein prompting the user to provide the user input comprises displaying the message concurrently with a text prompt associated with a suggestion to provide an audio output of the electronic document;

receive the user input, wherein the user input is associated with an intent to provide an audio output of the electronic document; and

in accordance with a determination that the received user input is associated with an intent to provide the audio output of the electronic document:

generate a media item based on the text of the electronic document; and

after generating the media item, audibly outputting the media item.

9. The non-transitory computer-readable storage medium of claim 8 , wherein the message includes a hyperlink to the electronic document.

10. The non-transitory computer-readable storage medium of claim 8 , wherein prompting a user to provide a user input further comprises:

providing an auditory response associated with a suggestion to provide an audio output of the electronic document.

11. The non-transitory computer-readable storage medium of claim 8 , wherein the user input is a physical input.

12. The non-transitory computer-readable storage medium of claim 8 , wherein the user input is a speech input.

13. The non-transitory computer-readable storage medium of claim 8 , wherein the one or more programs further comprise instructions, which when executed by the one or more processors of the electronic device, cause the electronic device to:

in accordance with a determination that the received user input is associated with an intent to save the electronic document to a user's reading list:

save the electronic document to the user's reading list.

14. The non-transitory computer-readable storage medium of claim 8 , wherein in accordance with the determination that the received user input is associated with the intent to provide the audio output of the electronic document further comprises:

after generating the media item, displaying the electronic document.

15. A method, comprising:

receiving, at an electronic device, a message, wherein the message includes an electronic document including text;

detecting, based on at least one keyword in the message and by a digital assistant, the electronic document in the received message;

prompting, by the digital assistant, a user to provide a user input, wherein prompting the user to provide the user input comprises displaying the message concurrently with a text prompt associated with a suggestion to provide an audio output of the electronic document;

receiving the user input, wherein the user input is associated with an intent to provide an audio output of the electronic document; and

in accordance with a determination that the received user input is associated with an intent to provide the audio output of the electronic document:

generating a media item based on the text of the electronic document; and

after generating the media item, audibly outputting the media item.

16. The method of claim 15 , wherein the message includes a hyperlink to the electronic document.

17. The method of claim 15 , wherein prompting a user to provide a user input further comprises:

providing an auditory response associated with a suggestion to provide an audio output of the electronic document.

18. The method of claim 15 , wherein the user input is a physical input.

19. The method of claim 15 , wherein the user input is a speech input.

20. The method of claim 15 , wherein further comprising:

in accordance with a determination that the received user input is associated with an intent to save the electronic document to a user's reading list:

saving the electronic document to the user's reading list.

21. The method of claim 15 , wherein in accordance with the determination that the received user input is associated with the intent to provide the audio output of the electronic document further comprises:

after generating the media item, displaying the electronic document.

Continuity (3)
Continuation 18414321 · Jan 16, 2024
Provisional Application 63459580 · Apr 14, 2023
Related Publication 20250149024A1 · May 8, 2025
References Cited (93)
US 5534888A · Lebby · 1996 [cited by examiner]
US 6033224A · Kurzweil et al. · 2000 [cited by applicant]
US 6250928B1 · Poggio et al. · 2001 [cited by applicant]
US 6421707B1 · Miller · 2002 [cited by examiner]
US 7096183B2 · Junqua · 2006 [cited by applicant]
US 7684991B2 · Stohr et al. · 2010 [cited by applicant]
US 8073695B1 · Hendricks et al. · 2011 [cited by applicant]
US 8452854B2 · Plestid · 2013 [cited by examiner]
US 8498866B2 · Kurzweil · 2013 [cited by examiner]
US 8832584B1 · Killalea et al. · 2014 [cited by applicant]
US 9002703B1 · Crosley · 2015 [cited by applicant]
US 9087508B1 · Dzik · 2015 [cited by applicant]
US 9135333B2 · Cameron · 2015 [cited by examiner]
US 9286287B1 · Tierney · 2016 [cited by applicant]
US 9606986B2 · Bellegarda · 2017 [cited by applicant]
US 9998410B1 · Morris · 2018 [cited by examiner]
US 10147421B2 · Liddell et al. · 2018 [cited by applicant]
US 10303715B2 · Graham · 2019 [cited by examiner]
US 10317992B2 · Prokofieva et al. · 2019 [cited by applicant]
US 10643611B2 · Lindahl · 2020 [cited by applicant]
US 10691473B2 · Karashchuk et al. · 2020 [cited by applicant]
US 10747498B2 · Stasior et al. · 2020 [cited by applicant]
US 10904488B1 · Weisz et al. · 2021 [cited by applicant]
US 10909171B2 · Graham et al. · 2021 [cited by applicant]
US 11087759B2 · Lemay et al. · 2021 [cited by applicant]
US 11152002B2 · Walker et al. · 2021 [cited by applicant]
US 11348582B2 · Lindahl · 2022 [cited by applicant]
US 11388291B2 · Van Os et al. · 2022 [cited by applicant]
US 11704552B2 · Sim et al. · 2023 [cited by applicant]
US 12327557B2 · Hussain · 2025 [cited by examiner]
US 20030212559A1 · Xie · 2003 [cited by applicant]
US 20060008122A1 · Kurzweil et al. · 2006 [cited by applicant]
US 20060112169A1 · Bodlaender · 2006 [cited by examiner]
US 20070173231A1 · Fadell · 2007 [cited by examiner]
US 20080140408A1 · Basir · 2008 [cited by examiner]
US 20090210908A1 · Sammarco · 2009 [cited by examiner]
US 20090259475A1 · Yamagami et al. · 2009 [cited by applicant]
US 20090313348A1 · Plestid · 2009 [cited by examiner]
US 20100097634A1 · Meyers · 2010 [cited by examiner]
US 20100324904A1 · Kurzweil · 2010 [cited by examiner]
US 20110119590A1 · Seshadri · 2011 [cited by examiner]
US 20110153047A1 · Cameron · 2011 [cited by examiner]
US 20110167287A1 · Walsh et al. · 2011 [cited by applicant]
US 20110177481A1 · Haff · 2011 [cited by examiner]
US 20120278082A1 · Borodin et al. · 2012 [cited by applicant]
US 20120311491A1 · Melton · 2012 [cited by examiner]
US 20130159853A1 · Story, Jr. · 2013 [cited by examiner]
US 20130307855A1 · Lamb et al. · 2013 [cited by applicant]
US 20140108010A1 · Maltseff et al. · 2014 [cited by applicant]
US 20140283111A1 · Dolph et al. · 2014 [cited by applicant]
US 20140365505A1 · Clark · 2014 [cited by examiner]
US 20150066506A1 · Romano et al. · 2015 [cited by applicant]
US 20150163610A1 · Sampat et al. · 2015 [cited by applicant]
US 20160027431A1 · Kurzweil et al. · 2016 [cited by applicant]
US 20160043974A1 · Purcell · 2016 [cited by examiner]
US 20160071510A1 · Li et al. · 2016 [cited by applicant]
US 20160171980A1 · Liddell et al. · 2016 [cited by applicant]
US 20180035155A1 · Garner et al. · 2018 [cited by applicant]
US 20180108356A1 · Mizumoto et al. · 2018 [cited by applicant]
US 20180285063A1 · Huynh et al. · 2018 [cited by applicant]
US 20180336894A1 · Graham · 2018 [cited by examiner]
US 20200143330A1 · Perumalla et al. · 2020 [cited by applicant]
US 20200159838A1 · Kikin-gil et al. · 2020 [cited by applicant]
US 20200365134A1 · Tu et al. · 2020 [cited by applicant]
US 20210097134A1 · Livshits et al. · 2021 [cited by applicant]
US 20220093191A1 · Yang · 2022 [cited by applicant]
US 20220291792A1 · Alston · 2022 [cited by applicant]
US 20230352014A1 · Tennant et al. · 2023 [cited by applicant]
US 20240347041A1 · Castellani et al. · 2024 [cited by applicant]
EP 1291847A2 · 2003 [cited by applicant]
JP 2018523102A · 2018 [cited by applicant]
KR 1020130075783A · 2013 [cited by applicant]
WO 2012051052A1 · 2012 [cited by applicant]
WO 2016191737A2 · 2016 [cited by applicant]
Advisory Action received for U.S. Appl. No. 18/414,321, mailed on Jul. 24, 2024, 3 pages. [cited by applicant]
Applicant-Initiated Interview Summary received for U.S. Appl. No. 18/414,321, mailed on Apr. 3, 2024, 4 pages. [cited by applicant]
Applicant-Initiated Interview Summary received for U.S. Appl. No. 18/414,321, mailed on Jun. 25, 2024, 3 pages. [cited by applicant]
Applicant-Initiated Interview Summary received for U.S. Appl. No. 18/414,321, mailed on Oct. 23, 2024, 5 pages. [cited by applicant]
Corrected Notice of Allowance received for U.S. Appl. No. 18/414,321, mailed on Jan. 10, 2025, 3 pages. [cited by applicant]
Ermolina et al., “Voice-Controlled Intelligent Personal Assistants in Health Care: International Delphi Study”, Journal of Medical Internet Research, 2021. Online Available at: Doi: 10.2196/25312., Apr. 9, 2021, 20 page… [cited by applicant]
Final Office Action received for U.S. Appl. No. 18/414,321, mailed on May 14, 2024, 44 pages. [cited by applicant]
Guo et al., “VizLens: A Robust and Interactive Screen Reader for Interfaces in the Real World”, In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16), Tokyo, Japan, online avai… [cited by applicant]
International Search Report and Written Opinion received for PCT Patent Application No. PCT/US2024/024028, mailed on Aug. 8, 2024, 24 pages. [cited by applicant]
Invitation to Pay Additional Fees and Partial International Search Report received for PCT Patent Application No. PCT/US2024/024028, mailed on Jun. 18, 2024, 15 pages. [cited by applicant]
Majerus Wesley. “Cell phone accessibility for your blind child”, Retrieved from the Internet <URL: https://web.archive.org/web/20100210001100/https://nfb.org/images/nfb/publications/fr/fr28/3/fr280314.htm>, 2010, pp. 1-… [cited by applicant]
Matias Yossi, “Easier access to web pages: Ask Google Assistant to read it aloud”, Available online at: https://blog.google/products/assistant/easier-access-web-pages-let-assistant-read-it-aloud/, Mar. 4, 2020, 3 pages. [cited by applicant]
Non-Final Office Action received for U.S. Appl. No. 18/414,321, mailed on Mar. 18, 2024, 44 pages. [cited by applicant]
Non-Final Office Action received for U.S. Appl. No. 18/414,321, mailed on Sep. 6, 2024, 59 pages. [cited by applicant]
Notice of Allowance received for U.S. Appl. No. 18/414,321, mailed on Dec. 4, 2024, 21 pages. [cited by applicant]
“Nuance Dragon Naturally Speaking”, Version 13 End-User Workbook, Nuance Communications, Inc. Online Available at: https://www.nuance.com/content/dam/nuance/en_us/collateral/dragon/guide/gd-dragon-naturally-speaking-13-… [cited by applicant]
Yeh et al., “Dialog modeling in audiobook synthesis”, Retrieved on Sep. 27, 2023, 6 pages. [cited by applicant]
Zhang et al., “Interaction Proxies for Runtime Repair and Enhancement of Mobile Application Accessibility”, In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, Denver, CO, USA… [cited by applicant]
Zhao et al., “SeeingVR: A Set of Tools to Make Virtual Reality More Accessible to People with Low Vision”, In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, Article 111, Gla… [cited by applicant]