IP Library › Granted Patent US 10,720,160
Granted Patent B2
US 10,720,160 · App. 16/504,782 · Granted Jul 21, 2020

Voice interaction at a primary device to access call functionality of a companion device

Inventors: Karl Ferdinand Schramm (Monte Sereno, CA); Justin Binder (Oakland, CA); Benjamin S. Phipps (San Francisco, CA); Po Keng Sung (San Francisco, CA)
Assignee: Apple Inc.
G10L15/22G10L15/1815G10L15/30H04M3/42212G10L2015/088G10L2015/223
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,720,160
App. No.
16/504,782
Filed
Jul 8, 2019
Granted
Jul 21, 2020
Kind
B2
Art Unit
2674
USPC
704/257
Abstract

The present disclosure generally relates to using voice interaction to access call functionality of a companion device. In an example process, a user utterance is received. Based on the user utterance and contextual information, the process causes a server to determine a user intent corresponding to the user utterance. The contextual information is based on a signal received from the companion device. In accordance with the user intent corresponding to an actionable intent of answering the incoming call, a command is received. Based on the command, instructions are provided to the companion device, which cause the companion device to answer the incoming call and provide audio data of the answered incoming call. Audio is outputted according to the audio data of the answered incoming call.

Claims (103)

1. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of an electronic device with a speaker, cause the electronic device to:

receive a user utterance representing a request for call-related information stored on a companion device;

cause a server to determine, based on the user utterance, a user intent of providing call-related information;

receive, from the server, a command representing a task to satisfy the user intent of providing call-related information;

determine, based on the command and contextual information, that the companion device is to execute the command;

in accordance with a determination that the companion device is to execute the command, provide instructions to the companion device to retrieve the call-related information and send the call-related information to the electronic device; and

in accordance with the instructions successfully causing the companion device to retrieve and send the call-related information to the electronic device, provide an audio output at the speaker, wherein the audio output includes the call-related information received from the companion device.

2. The non-transitory computer-readable storage medium of claim 1 , wherein the call-related information comprises information stored on the companion device based on one or more calls received by the companion device prior to the electronic device receiving the user utterance.

3. The non-transitory computer-readable storage medium of claim 1 , wherein the call-related information comprises a voicemail audio file stored on the companion device, and

wherein the voicemail audio file comprises call information for a voicemail and an audio recording of the voicemail.

4. The non-transitory computer-readable storage medium of claim 3 , wherein the audio output at the speaker comprises a first spoken dialogue and the audio recording of the voicemail, and

wherein the first spoken dialogue comprises the call information for the voicemail.

5. The non-transitory computer-readable storage medium of claim 4 , wherein the one or more programs comprise further instructions, which when executed by the one or more processors of the electronic device, cause the electronic device to:

after receiving the call-related information at the electronic device and prior to providing the audio output at the speaker:

send, to the server, a representation of the call information for the voicemail; and

receive, from the server, the first spoken dialogue comprising the call information.

6. The non-transitory computer-readable storage medium of claim 1 , wherein the call-related information comprises caller information stored on the companion device for one or more missed calls.

7. The non-transitory computer-readable storage medium of claim 6 , wherein the audio output at the speaker comprises a second spoken dialogue, and

wherein the second spoken dialogue comprises the caller information corresponding to the one or more missed calls.

8. The non-transitory computer-readable storage medium of claim 1 , wherein the command specifies an actionable intent node corresponding to the user intent, and wherein the instructions include the command.

9. The non-transitory computer-readable storage medium of claim 1 , wherein the contextual information indicates that the companion device has telephone capabilities.

10. The non-transitory computer-readable storage medium of claim 1 , wherein the one or more programs comprise further instructions, which when executed by the one or more processors of the electronic device, cause the electronic device to:

prior to receiving the user utterance, exchange authentication information with the companion device; and

in accordance with a determination, based on the authentication information, that the companion device is a registered device of the electronic device, establish a wireless communication connection with the companion device,

wherein the instructions are provided via the established wireless communication connection, and

wherein the call-related information is received via the established wireless communication connection.

11. The non-transitory computer-readable storage medium of claim 10 , wherein the contextual information indicates that the wireless communication connection is established between the electronic device and the companion device and that the companion device is a registered device of the electronic device.

12. The non-transitory computer-readable storage medium of claim 1 , wherein the one or more programs comprise further instructions, which when executed by the one or more processors of the electronic device, cause the electronic device to:

after providing the instructions to the companion device, receive a confirmation signal from the companion device, wherein the confirmation signal indicates whether the call-related information was successfully retrieved; and

provide a representation of the confirmation signal to the server.

13. The non-transitory computer-readable storage medium of claim 1 , wherein the one or more programs comprise further instructions, which when executed by the one or more processors of the electronic device, cause the electronic device to:

determine whether a first portion of the user utterance contains a predefined spoken trigger; and

in accordance with a determination that the first portion of the user utterance contains a predefined spoken trigger, receive a second portion of the user utterance separate from the first portion, wherein the user intent is determined from the second portion of the user utterance.

14. An electronic device, comprising:

a speaker;

one or more processors; and

memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for:

receiving a user utterance representing a request for call-related information stored on a companion device;

causing a server to determine, based on the user utterance, a user intent of providing call-related information;

receiving, from the server, a command representing a task to satisfy the user intent of providing call-related information;

determining, based on the command and contextual information, that the companion device is to execute the command;

in accordance with a determination that the companion device is to execute the command, providing instructions to the companion device to retrieve the call-related information and send the call-related information to the electronic device; and

in accordance with the instructions successfully causing the companion device to retrieve and send the call-related information to the electronic device, providing an audio output at the speaker, wherein the audio output includes the call-related information received from the companion device.

15. A method for using voice interaction to access call-related information of a companion device, the method comprising:

at an electronic device having a processor, memory, and a speaker:

receiving a user utterance representing a request for call-related information stored on the companion device;

causing a server to determine, based on the user utterance, a user intent of providing call-related information;

receiving, from the server, a command representing a task to satisfy the user intent of providing call-related information;

determining, based on the command and contextual information, that the companion device is to execute the command;

in accordance with a determination that the companion device is to execute the command, providing instructions to the companion device to retrieve the call-related information and send the call-related information to the electronic device; and

in accordance with the instructions successfully causing the companion device to retrieve and send the call-related information to the electronic device, providing an audio output at the speaker, wherein the audio output includes the call-related information received from the companion device.

16. The electronic device of claim 14 , wherein the call-related information comprises information stored on the companion device based on one or more calls received by the companion device prior to the electronic device receiving the user utterance.

17. The electronic device of claim 14 , wherein the call-related information comprises a voicemail audio file stored on the companion device, and

wherein the voicemail audio file comprises call information for a voicemail and an audio recording of the voicemail.

18. The electronic device of claim 17 , wherein the audio output at the speaker comprises a first spoken dialogue and the audio recording of the voicemail, and

wherein the first spoken dialogue comprises the call information for the voicemail.

19. The electronic device of claim 18 , wherein the one or more programs further include instructions for:

after receiving the call-related information at the electronic device and prior to providing the audio output at the speaker:

sending, to the server, a representation of the call information for the voicemail; and

receiving, from the server, the first spoken dialogue comprising the call information.

20. The electronic device of claim 14 , wherein the call-related information comprises caller information stored on the companion device for one or more missed calls.

21. The electronic device of claim 20 , wherein the audio output at the speaker comprises a second spoken dialogue, and

wherein the second spoken dialogue comprises the caller information corresponding to the one or more missed calls.

22. The electronic device of claim 14 , wherein the command specifies an actionable intent node corresponding to the user intent, and wherein the instructions include the command.

23. The electronic device of claim 14 , wherein the contextual information indicates that the companion device has telephone capabilities.

24. The electronic device of claim 14 , wherein the one or more programs further include instructions for:

prior to receiving the user utterance, exchanging authentication information with the companion device; and

in accordance with a determination, based on the authentication information, that the companion device is a registered device of the electronic device, establishing a wireless communication connection with the companion device,

wherein the instructions are provided via the established wireless communication connection, and

wherein the call-related information is received via the established wireless communication connection.

25. The electronic device of claim 24 , wherein the contextual information indicates that the wireless communication connection is established between the electronic device and the companion device and that the companion device is a registered device of the electronic device.

26. The electronic device of claim 14 , wherein the one or more programs further include instructions for:

after providing the instructions to the companion device, receiving a confirmation signal from the companion device, wherein the confirmation signal indicates whether the call-related information was successfully retrieved; and

providing a representation of the confirmation signal to the server.

27. The electronic device of claim 14 , wherein the one or more programs further include instructions for:

determining whether a first portion of the user utterance contains a predefined spoken trigger; and

in accordance with a determination that the first portion of the user utterance contains a predefined spoken trigger, receiving a second portion of the user utterance separate from the first portion, wherein the user intent is determined from the second portion of the user utterance.

28. The method of claim 15 , wherein the call-related information comprises information stored on the companion device based on one or more calls received by the companion device prior to the electronic device receiving the user utterance.

29. The method of claim 15 , wherein the call-related information comprises a voicemail audio file stored on the companion device, and

wherein the voicemail audio file comprises call information for a voicemail and an audio recording of the voicemail.

30. The method of claim 29 , wherein the audio output at the speaker comprises a first spoken dialogue and the audio recording of the voicemail, and

wherein the first spoken dialogue comprises the call information for the voicemail.

31. The method of claim 30 , further comprising:

after receiving the call-related information at the electronic device and prior to providing the audio output at the speaker:

sending, to the server, a representation of the call information for the voicemail; and

receiving, from the server, the first spoken dialogue comprising the call information.

32. The method of claim 15 , wherein the call-related information comprises caller information stored on the companion device for one or more missed calls.

33. The method of claim 32 , wherein the audio output at the speaker comprises a second spoken dialogue, and

wherein the second spoken dialogue comprises the caller information corresponding to the one or more missed calls.

34. The method of claim 15 , wherein the command specifies an actionable intent node corresponding to the user intent, and wherein the instructions include the command.

35. The method of claim 15 , wherein the contextual information indicates that the companion device has telephone capabilities.

36. The method of claim 15 , further comprising:

prior to receiving the user utterance, exchanging authentication information with the companion device; and

in accordance with a determination, based on the authentication information, that the companion device is a registered device of the electronic device, establishing a wireless communication connection with the companion device,

wherein the instructions are provided via the established wireless communication connection, and

wherein the call-related information is received via the established wireless communication connection.

37. The method of claim 36 , wherein the contextual information indicates that the wireless communication connection is established between the electronic device and the companion device and that the companion device is a registered device of the electronic device.

38. The method of claim 15 , further comprising:

after providing the instructions to the companion device, receiving a confirmation signal from the companion device, wherein the confirmation signal indicates whether the call-related information was successfully retrieved; and

providing a representation of the confirmation signal to the server.

39. The method of claim 15 , further comprising:

determining whether a first portion of the user utterance contains a predefined spoken trigger; and

in accordance with a determination that the first portion of the user utterance contains a predefined spoken trigger, receiving a second portion of the user utterance separate from the first portion, wherein the user intent is determined from the second portion of the user utterance.

Continuity (3)
Continuation 16113119 · Aug 27, 2018
Provisional Application 62679177 · Jun 1, 2018
Related Publication 20190371331A1 · Dec 5, 2019