Granted Patent B2

US 10,831,923 · App. 16/003,691 · Granted Nov 10, 2020

Inventors: Milos Dunjic (Oakville, CA); David Samuel Tax (Toronto, CA); Arthur Carroll Chow (Markham, CA); Peter Glen Nairn (Cobourg, CA); Edward James Hood (Toronto, CA); Martin Albert Lozon (London, CA); John Jong-Suk Lee (Toronto, CA); Arun Victor Jagga (Toronto, CA)

Assignee: The Toronto-Dominion Bank

G06F21/6245G10L15/1822G10L15/22H04L63/0861H04L63/104H04L65/1083H04R1/406G10L2015/223

Abstract

Claims (143)

1. An electronic device, comprising:

a processor;

a speaker, a plurality of sensors for sensing an environment of the electronic device, and a memory, each of the speaker, the plurality of sensors and the memory being coupled to the processor, wherein the plurality of sensors comprises one or more microphones;

a privacy module stored in the memory, the privacy module comprising executable instructions that, when executed by the processor, cause the electronic device to:

authenticate a user via one or more first criteria in response to a request to initiate a communication session with a voice assistant;

initiate the communication session with the voice assistant in response to a successful authentication of the user;

periodically during the communication session with the voice assistant:

acquire sensor data from one or more sensors of the plurality of sensors;

process the sensor data to determine a number of persons present in the environment of the electronic device via one or more second criteria;

when one person is present in the environment of the electronic device:

process the sensor data to identify the one person, determine whether the one person is the authenticated user based on whether the one person is identified as the authenticated user, enable communication of private data by the voice assistant when the one person in the environment is determined to be the authenticated user, and disable communication of private data by the voice assistant when the one person in the environment is determined not to be the authenticated user; and

when more than one person is present in the environment of the electronic device:

process the sensor data to determine whether the environment of the electronic device matches one or more predetermined privacy criteria for a multi-person environment, enable communication of private data by the voice assistant when the environment of the electronic device is determined to match the one or more predetermined privacy criteria for the multi-person environment, and disable communication of private data by the voice assistant when the environment of the electronic device is determined not to match the one or more predetermined privacy criteria for the multi-person environment.

2. The electronic device of claim 1 , wherein the sensor data used to determine the number of persons in the environment comprises one or a combination of a facial data, voice data, IR heat sensor data, movement sensor data, or wireless device usage data.

3. The electronic device of claim 2 , wherein the authenticated user is identified using voice recognition, facial recognition or a combination thereof, the sensor data used to identify the one person as the authenticated user comprises voice data, facial data, or a combination thereof.

4. The electronic device of claim 1 , wherein the one or more first criteria used to authenticate the user is different from the one or more second criteria used to determine the number of persons in the environment of the electronic device.

5. The electronic device of claim 1 , wherein the one or more first criteria used to authenticate the user comprises a shared secret, and the one or more second criteria comprises voice recognition, facial recognition, or a combination thereof.

6. The electronic device of claim 5 , wherein the one or more first criteria comprises input via a keyboard, the one or more first criteria comprising the shared secret and biometric factors acquired during the input via the keyboard.

7. The electronic device of claim 6 , wherein the biometric factors comprise a typing cadence, fingerprint recognition, voice recognition, facial recognition, or a combination thereof.

8. The electronic device of claim 1 , wherein the privacy module, to determine the number of persons in the environment, monitors for and detects a wireless communication devices in the environment of the electronic device, each wireless communication device in the environment of the electronic device being counted as a person in the environment of the electronic device.

9. The electronic device of claim 1 , wherein the privacy module, to determine whether the environment of the electronic device matches one or more predetermined privacy criteria for a multi-person environment, is further configured to:

determine, using the sensor data, an audibility probability that private information audibly communicated by the electronic device may be heard by any of the other persons present in the environment of the electronic device;

determine, using the audibility probability, that the environment of the electronic device matches the one or more predetermined privacy criteria for a multi-person environment when the audibility probability is greater than or equal to an audibility probability threshold; and

determine, using the audibility probability, that the environment of the electronic device does not match the one or more predetermined privacy criteria for a multi-person environment when the audibility probability is less than the audibility probability threshold.

10. The electronic device of claim 1 , wherein the one or more predetermined privacy criteria for a multi-person environment comprises each person in the environment being an authorized user.

11. The electronic device of claim 1 , wherein the one or more predetermined privacy criteria for a multi-person environment comprises each person in the environment being an authorized user and each person other than the authenticated user being more than a threshold distance from the authenticated user.

12. The electronic device of claim 1 , wherein the one or more predetermined privacy criteria for a multi-person environment comprises each person other than the authenticated user being more than a threshold distance from the authenticated user.

13. The electronic device of claim 12 , wherein the privacy module comprises executable instructions that, when executed by the processor, cause the electronic device to:

periodically during the communication session with the voice assistant when more than one person is present in the environment of the electronic device:

sense, via the plurality of sensors, the environment of the electronic device to generate sensed data;

generate a three-dimensional model of the environment using the sensed data;

generate an audio profile of the environment based on the three-dimensional model and an audio sample of the environment, the audio profile defines a sound transmission pattern within the environment given a three-dimensional shape as defined by the three-dimensional model of the environment;

determine an audible transmission distance of the voice of the authenticated user based on the audio profile of the environment as the threshold distance, wherein the audible transmission distance determines a distance from the authenticated user within which the voice of the authenticated user is discernable to other persons in the environment;

localize, via the sensed data, all persons in the environment; and

determine, for each person other than the authenticated user, a distance of the person from the authenticated user;

determine whether any person other than the authenticated user is within the audible transmission distance from the authenticated user;

determine that the environment of the electronic device matches the one or more predetermined privacy criteria for a multi-person environment when no one is within the audible transmission distance from the authenticated user; and

determine that the environment of the electronic device does not match the one or more predetermined privacy criteria for a multi-person environment when at least one person is within the audible transmission distance from the authenticated user.

14. The electronic device of claim 13 , wherein the audio profile of the environment is based on the three-dimensional model and an audio sample of the environment.

15. The electronic device of claim 13 , wherein the audible transmission distance of the voice of the authenticated user is based on the audio profile and one or more characteristics of the voice of the authenticated user.

16. The electronic device of claim 13 , wherein the plurality of sensors further comprises one or more cameras and one or more proximity sensors, wherein the privacy module comprises executable instructions that, when executed by the processor, cause the electronic device to:

periodically during the communication session with the voice assistant when more than one person is present in the environment of the electronic device:

acquire, via the one or more cameras, one or more images of the environment;

identify one or more reference points in the one or more images of the environment;

determine, via proximity data sensed by the one or more proximity sensors, a distance to the one or more reference points; and

generate the three-dimensional model of the environment using the one or more images and the distance to the one or more reference points.

17. The electronic device of claim 13 , wherein the privacy module comprises executable instructions that, when executed by the processor, cause the electronic device to:

periodically during the communication session with the voice assistant when more than one person is present in the environment of the electronic device:

generate, via the speaker, a multi-tone signal;

receive, via the microphone, a reflected multi-tone signal; and

generate the three-dimensional model of the environment using the multi-tone signal and the reflected multi-tone signal.

18. The electronic device of claim 1 , wherein the privacy module, to disable communication of private data by the voice assistant when the environment of the electronic device is determined not to match the one or more predetermined privacy criteria for a multi-person environment, is further configured to:

generate, via the speaker of the electronic device, an audible notification that the communication session is not private, the notification comprising a voice prompt whether to continue the communication session via a different channel or continue the communication session from a private location;

receive, via the one or more microphones, a voice input;

parse, via speech recognition, the voice input to extract a command to be performed from a plurality of commands;

transfer the communication session to a second electronic device in response to the voice input containing a corresponding command;

initiate a call back to a designated telephone number in response to the voice input containing a corresponding command; and

temporarily suspend the communication session in response to the voice input containing a corresponding command.

19. The electronic device of claim 18 , wherein the privacy module, to disable communication of private data by the voice assistant when the environment of the electronic device is determined not to match the one or more predetermined privacy criteria for a multi-person environment, is further configured to:

while the communication session is temporarily suspended:

receive, via the one or more microphones, a voice input;

parse, via speech recognition, the voice input to extract a command to be performed from a plurality of commands; and

resume the communication session from the temporary suspension in response to the voice input containing a corresponding command.

20. The electronic device of claim 18 , wherein the privacy module, to disable communication of private data by the voice assistant when the environment of the electronic device is determined not to match the one or more predetermined privacy criteria for a multi-person environment, is further configured to:

while the communication session is temporarily suspended:

monitor for changes in the location of the electronic device to determine whether the authenticated user has moved to a private location; and

automatically resume the communication session from the temporary suspension in response to a determination that the authenticated user has moved to a private location.

21. The electronic device of claim 1 , wherein the privacy module, to disable communication of private data by the voice assistant when the environment of the electronic device is determined not to match the one or more predetermined privacy criteria for a multi-person environment, is configured to:

generate, via the speaker of the electronic device, an audible notification that the communication session is not private, the notification comprising a voice prompt whether to continue communication of private data even though the communication session is not private;

receive, via the one or more microphones, a voice input;

parse, via speech recognition, the voice input to extract a command to be performed from a plurality of commands; and

re-enable the communication of private data in response to the voice input containing a corresponding command.

22. The electronic device of claim 1 , wherein the privacy module, to disable communication of private data by the voice assistant when the environment of the electronic device is determined not to match the one or more predetermined privacy criteria for a multi-person environment, is configured to:

generate, via the speaker of the electronic device, an audible notification that the communication session is not private, the notification comprising a voice prompt whether to continue the communication session with only non-private data;

receive, via the one or more microphones, a voice input;

parse, via speech recognition, the voice input to extract a command to be performed from a plurality of commands;

terminate the communication session in response to the voice input containing a corresponding command; and

continue the communication session in response to the voice input containing a corresponding command.

23. The electronic device of claim 1 , wherein the privacy module comprises executable instructions that, when executed by the processor, cause the electronic device to:

periodically during the communication session with the voice assistant:

when no one is present in the environment of the electronic device, disable communication of private data by the voice assistant when the one person in the environment is not the authenticated user.

24. An electronic device, comprising:

a processor;

a speaker, a plurality of sensors for sensing an environment of the electronic device, and a memory, each of the speaker, plurality of sensors and memory being coupled to the processor, wherein the plurality of sensors comprises one or more microphones;

a privacy module stored in the memory, the privacy module comprising executable instructions that, when executed by the processor, cause the electronic device to:

periodically during a communication session with a voice assistant:

acquire sensor data from one or more sensors in the plurality of sensors;

process the sensor data to determine a number of persons present in the environment of the electronic device;

when one person is present in the environment of the electronic device:

process the sensor data to identify the one person, determine whether the one person is an authorized user, enable communication of private data by the voice assistant when the one person in the environment is determined to be an authorized user, and disable communication of private data by the voice assistant when the one person in the environment is determined not to be an authorized user; and

when more than one person is present in the environment of the electronic device, disable communication of private data by the voice assistant.

25. An electronic device, comprising:

a processor;

a speaker, a plurality of sensors for sensing an environment of the electronic device, a communication subsystem and a memory, each of the speaker, the plurality of sensors and the memory being coupled to the processor, wherein the plurality of sensors comprises one or more microphones;

a privacy module stored in the memory, the privacy module comprising executable instructions that, when executed by the processor, cause the electronic device to:

authenticate a user via one or more first criteria in response to a request to initiate a communication session with a voice assistant;

initiate the communication session with the voice assistant in response to a successful authentication of the user;

periodically during the communication session with the voice assistant:

acquire sensor data from one or more sensors of the plurality of sensors;

send the sensor data to a remote server to process the sensor data;

receive privacy enforcement instructions from the remote server in response to processing the sensor data; and

apply the privacy enforcement instructions, wherein application of the privacy enforcement instructions causes the electronic device to:

enable communication of private data by the voice assistant when one person is present in the environment of the electronic device and the one person in the environment has been determined to be the authenticated user;

disable communication of private data by the voice assistant when one person is present in the environment of the electronic device and the one person in the environment is determined not to be the authenticated user;

enable communication of private data by the voice assistant when more than one person is present in the environment of the electronic device and the environment of the electronic device is determined to match the one or more predetermined privacy criteria for a multi-person environment; and

disable communication of private data by the voice assistant when more than one person is present in the environment of the electronic device and the environment of the electronic device is determined not to match the one or more predetermined privacy criteria for a multi-person environment.

26. The electronic device of claim 25 , wherein the remote server processes the sensor data to identify the one person when only one person is present in the environment of the electronic device, wherein the remote server processes the sensor data to determine whether the environment of the electronic device matches one or more predetermined privacy criteria for a multi-person environment when more than one person is present in the environment of the electronic device.

27. An electronic device, comprising:

a processor;

a speaker, a plurality of sensors for sensing an environment of the electronic device, a communication subsystem and a memory, each of the speaker, the plurality of sensors and the memory being coupled to the processor, wherein the plurality of sensors comprises one or more microphones;

a privacy module stored in the memory, the privacy module comprising executable instructions that, when executed by the processor, cause the electronic device to:

periodically during a communication session with the voice assistant:

acquire sensor data from one or more sensors of the plurality of sensors;

send the sensor data to a remote server to process the sensor data;

receive privacy enforcement instructions from the remote server in response to processing the sensor data; and

apply the privacy enforcement instructions, wherein application of the privacy enforcement instructions causes the electronic device to:

enable communication of private data by the voice assistant when one person is present in the environment of the electronic device and the one person in the environment has been determined to be an authorized user; and

disable communication of private data by the voice assistant when more than one person is present in the environment of the electronic device.

28. A server device, comprising:

a processor;

a communication subsystem and a memory each coupled to the processor;

a privacy module stored in the memory, the privacy module comprising executable instructions that, when executed by the processor, cause the server device to:

receive sensor data acquired by one or more sensors of plurality of sensors of an electronic device;

process the sensor data to determine a number of persons present in the environment of the electronic device via one or more criteria;

process the sensor data to identify the one person and determine whether the one person is the authenticated user when one person is present in the environment of the electronic device;

process the sensor data to determine whether the environment of the electronic device matches one or more predetermined privacy criteria for a multi-person environment when more than one person is present in the environment of the electronic device;

generate privacy enforcement instructions in response to processing the sensor data; and

send the privacy enforcement instructions to the electronic device on which the privacy enforcement instructions are to be applied, wherein application of the privacy enforcement instructions causes the electronic device to:

enable communication of private data by the voice assistant when one person is present in the environment of the electronic device and the one person in the environment has been determined to be the authenticated user;

disable communication of private data by the voice assistant when one person is present in the environment of the electronic device and the one person in the environment is determined not to be the authenticated user;

enable communication of private data by the voice assistant when more than one person is present in the environment of the electronic device and the environment of the electronic device is determined to match the one or more predetermined privacy criteria for a multi-person environment; and

disable communication of private data by the voice assistant when more than one person is present in the environment of the electronic device and the environment of the electronic device is determined not to match the one or more predetermined privacy criteria for a multi-person environment.

29. A server device, comprising:

a processor;

a communication subsystem and a memory each coupled to the processor;

a privacy module stored in the memory, the privacy module comprising executable instructions that, when executed by the processor, cause the server device to:

receive sensor data acquired by one or more sensors of plurality of sensors of an electronic device;

process the sensor data to determine a number of persons present in the environment of the electronic device via one or more second criteria;

process the sensor data to identify the one person and determine whether the one person is an authorized user when one person is present in the environment of the electronic device;

generate privacy enforcement instructions in response to processing the sensor data; and

send the privacy enforcement instructions to the electronic device on which the privacy enforcement instructions are to be applied, wherein application of the privacy enforcement instructions causes the electronic device to:

enable communication of private data by the voice assistant when one person is present in the environment of the electronic device and the one person in the environment has been determined to be an authorized user; and

disable communication of private data by the voice assistant when more than one person is present in the environment of the electronic device.

Assignments (1)

Continuity (1)

Cited By (25)

US 12,499,241 US 12,517,812 US 12,536,264 US 12,541,894 US 12,566,541 US 12,585,435 US 12,591,559 US 12,592,301 US 12,625,680 US 12,641,178 US 12,645,429 US 12,645,689 US 12,645,838 US 12,646,051 US 12,650,836 US 12,657,566 US 12,670,334 US 12,670,640 US 12,682,179 US 12,688,620 US 12,693,842 US 12,699,556 US 12,705,398 US 12,711,683 US 12,725,152