IP Library Granted Patent US 9,053,096
Granted Patent B2
US 9,053,096 · App. 13/340,143 · Granted Jun 9, 2015

Language translation based on speaker-related information

Inventors: Richard T. Lord (Tacoma, WA); Robert W. Lord (Seattle, WA); Nathan P. Myhrvold (Medina, WA); Clarence T. Tegreene (Bellevue, WA); Roderick A. Hyde (Redmond, WA); Lowell L. Wood, Jr. (Bellevue, WA); Muriel Y. Ishikawa (Livermore, CA); Victoria Y. H. Wood (Livermore, CA); Charles Whitmer (North Bend, WA); Paramvir Bahl (Bellevue, WA); Douglas C. Burger (Bellevue, WA); Ranveer Chandra (Kirkland, WA); William H. Gates, III (Medina, WA); Paul Holman (Seattle, WA); Jordin T. Kare (Seattle, WA); Craig J. Mundie (Seattle, WA); Tim Paek (Sammamish, WA); Desney S. Tan (Kirkland, WA); Lin Zhong (Houston, TX); Matthew G. Dyor (Bellevue, WA)
Assignee: Elwha LLC
G06F17/289
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,053,096
App. No.
13/340,143
Granted
Jun 9, 2015
Kind
B2
Abstract

Techniques for ability enhancement are described. Some embodiments provide an ability enhancement facilitator system (“AEFS”) configured to automatically translate utterances from a first to a second language, based on speaker-related information determined from speaker utterances and/or other sources of information. In one embodiment, the AEFS receives data that represents an utterance of a speaker in a first language, the utterance obtained by a hearing device of the user, such as a hearing aid, smart phone, media player/device, or the like. The AEFS then determines speaker-related information associated with the identified speaker, such as by determining demographic information (e.g., gender, language, country/region of origin) and/or identifying information (e.g., name or title) of the speaker. The AEFS translates the utterance in the first language into a message in a second language, based on the determined speaker-related information. The AEFS then presents the message in the second language to the user.

Claims (89)

1. A method for ability enhancement, the method comprising:

receiving data representing a speech signal obtained at a hearing device associated with a user, the speech signal representing an utterance of a speaker in a first language;

determining speaker-related information associated with the speaker, based on the data representing the speech signal;

translating the utterance in the first language into a message in a second language, based on the speaker-related information;

presenting the message in the second language; and

distributing processing tasks among available computing resources including the hearing device, a computing device of the user and/or the speaker, and a remote computing system, by determining where to perform the processing tasks by identifying one of the computing resources that has available capacity, the processing tasks including determining speaker-related information and translating the utterance in the first language into a message in a second language.

2. The method of claim 1 , wherein the determining speaker-related information includes: determining the first language.

3. The method of claim 2 , wherein the determining the first language includes:

concurrently processing the received data with multiple speech recognizers that are each configured to recognize speech in a different corresponding language; and

selecting as the first language the language corresponding to a speech recognizer of the multiple speech recognizer that produces a result that has a higher confidence level than other of the multiple speech recognizers.

4. The method of claim 2 , wherein the determining the first language includes: identifying signal characteristics in the received data that are correlated with the first language.

5. The method of claim 2 , wherein the determining the first language includes:

receiving an indication of a current location of the user;

determining one or more languages that are commonly spoken at the current location; and

selecting one of the one or more languages as the first language.

6. The method of claim 2 , wherein the determining the first language includes:

presenting indications of multiple languages to the user; and

receiving from the user an indication of one of the multiple languages.

7. The method of claim 2 , further comprising: selecting a speech recognizer configured to recognize speech in the first language.

8. The method of claim 1 , wherein the translating the utterance in the first language into a message in a second language includes:

performing speech recognition, based on the speaker-related information, on the data representing the speech signal to convert the utterance in the first language into text representing the utterance in the first language; and

translating, based on the speaker-related information, the text representing the utterance in the first language into text representing the message in the second language.

9. The method of claim 8 , wherein the presenting the message in the second language includes:

performing speech synthesis to convert the text representing the utterance in the second language into audio data representing the message in the second language; and

causing the audio data representing the message in the second language to be played to the user.

10. The method of claim 1 , wherein the translating the utterance in the first language into a message in a second language includes: translating the utterance based on speaker-related information including an identity of the speaker.

11. The method of claim 1 , wherein the translating the utterance in the first language into a message in a second language includes: translating the utterance based on speaker-related information including a language model that is specific to the speaker.

12. The method of claim 11 , wherein the translating the utterance based on speaker-related information including a language model that is specific to the speaker includes: translating the utterance based on a language model that is tailored to a group of people of which the speaker is a member.

13. The method of claim 11 , wherein the translating the utterance based on speaker-related information including a language model that is specific to the speaker includes: generating the language model based on communications generated by the speaker.

14. The method of claim 13 , wherein the generating the language model based on communications generated by the speaker includes: generating the language model based on emails transmitted by the speaker.

15. The method of claim 13 , wherein the generating the language model based on communications generated by the speaker includes: generating the language model based on documents authored by the speaker.

16. The method of claim 13 , wherein the generating the language model based on communications generated by the speaker includes: generating the language model based on social network messages transmitted by the speaker.

17. The method of claim 1 , wherein the translating the utterance in the first language into a message in a second language includes: translating the utterance based on speaker-related information including a speech model that is tailored to the speaker.

18. The method of claim 17 , wherein the translating the utterance based on speaker-related information including a speech model that is tailored to the speaker includes: translating the utterance based on a speech model that is tailored to a group of people of which the speaker is a member.

19. The method of claim 1 , wherein the translating the utterance in the first language into a message in a second language includes: translating the utterance based on speaker-related information including an information item that references the speaker.

20. The method of claim 1 , wherein the translating the utterance in the first language into a message in a second language includes: translating the utterance based on speaker-related information including a document that references the speaker.

21. The method of claim 1 , wherein the translating the utterance in the first language into a message in a second language includes: translating the utterance based on speaker-related information including a message that references the speaker.

22. The method of claim 1 , wherein the translating the utterance in the first language into a message in a second language includes: translating the utterance based on speaker-related information including a calendar event that references the speaker.

23. The method of claim 1 , wherein the translating the utterance in the first language into a message in a second language includes: translating the utterance based on speaker-related information including an indication of gender of the speaker.

24. The method of claim 1 , wherein the translating the utterance in the first language into a message in a second language includes: translating the utterance based on speaker-related information including an organization to which the speaker belongs.

25. The method of claim 1 , wherein the determining speaker-related information includes: performing voice identification based on the received data to identify the speaker.

26. The method of claim 25 , further comprising: determining that the speaker cannot be identified.

27. The method of claim 26 , further comprising: when it is determined that the speaker cannot be identified, storing the received data for system training.

28. The method of claim 26 , further comprising: when it is determined that the speaker cannot be identified, notifying the user.

29. The method of claim 1 , further comprising:

receiving data representing a speech signal that represents an utterance of the user; and

determining the speaker-related information based on the data representing a speech signal that represents an utterance of the user.

30. The method of claim 29 , wherein the determining the speaker-related information based on the data representing a speech signal that represents an utterance of the user includes: determining whether the utterance of the user includes a name of the speaker.

31. The method of claim 1 , wherein the determining speaker-related information includes:

identifying a plurality of candidate speakers; and

presenting indications of the plurality of candidate speakers.

32. The method of claim 31 , further comprising:

receiving from the user a selection of one of the plurality of candidate speakers that is the speaker; and

determining the speaker-related information based on the selection received from the user.

33. The method of claim 31 , further comprising:

receiving from the user an indication that none of the plurality of candidate speakers are the speaker; and

training a speaker identification system based on the received indication.

34. The method of claim 31 , further comprising: training a speaker identification system based on a selection regarding the plurality of candidate speakers received from a user.

35. The method of claim 1 , wherein the presenting the message in the second language includes: transmitting the message in the second language from a first device to a second device.

36. The method of claim 35 , wherein the transmitting the message in the second language from a first device to a second device includes: transmitting the message in the second language from a smart phone or portable media device to the second device.

37. The method of claim 35 , wherein the transmitting the message in the second language from a first device to a second device includes: transmitting the message in the second language from a server system to the second device.

38. The method of claim 37 , wherein the transmitting the message in the second language from a server system includes: transmitting the message in the second language from a server system to a mobile device of the user.

39. The method of claim 1 , further comprising: performing the receiving data representing a speech signal, the determining speaker-related information, the translating the utterance in the first language into a message in a second language, and/or the presenting the message in the second language on a mobile device that is operated by the user.

40. The method of claim 1 , further comprising: performing the receiving data representing a speech signal, the determining speaker-related information, the translating the utterance in the first language into a message in a second language, and/or the presenting the message in the second language on a desktop computer that is operated by the user.

41. The method of claim 1 , further comprising: receiving at least some of speaker-related information from the identified computing resource.

42. The method of claim 1 , further comprising: informing the user of the speaker-related information.

43. The method of claim 42 , further comprising:

receiving feedback from the user regarding correctness of the speaker-related information; and

refining the speaker-related information based on the received feedback.

44. The method of claim 43 , wherein the refining the speaker-related information based on the received feedback includes:

presenting speaker-related information corresponding to each of multiple likely speakers; and

receiving from the user an indication that the speaker is one of the multiple likely speakers.

45. The method of claim 42 , wherein the informing the user of the speaker-related information includes: presenting the speaker-related information on a display of the hearing device.

46. The method of claim 42 , wherein the informing the user of the speaker-related information includes: presenting the speaker-related information on a display of a computing device that is distinct from the hearing device.

47. A non-transitory computer-readable medium including instructions that are configured, when executed, to cause a computing system to perform a method for ability enhancement, the method comprising:

receiving data representing a speech signal obtained at a hearing device associated with a user, the speech signal representing an utterance of a speaker in a first language;

determining speaker-related information associated with the speaker, based on the data representing the speech signal;

translating the utterance in the first language into a message in a second language, based on the speaker-related information;

presenting the message in the second language; and

distributing processing tasks among available computing resources including the hearing device, a computing device of the user and/or the speaker, and a remote computing system, by determining where to perform the processing tasks by identifying one of the computing resources that has available capacity, the processing tasks including determining speaker-related information and translating the utterance in the first language into a message in a second language.

48. A computing system for ability enhancement, the computing system comprising:

a processor;

a memory; and

a module that is stored in the memory and that is configured, when executed by the processor, to perform a method comprising:

receiving data representing a speech signal obtained at a hearing device associated with a user, the speech signal representing an utterance of a speaker in a first language;

determining speaker-related information associated with the speaker, based on the data representing the speech signal;

translating the utterance in the first language into a message in a second language, based on the speaker-related information;

presenting the message in the second language; and

distributing processing tasks among available computing resources including the hearing device, a computing device of the user and/or the speaker, and a remote computing system, by determining where to perform the processing tasks by identifying one of the computing resources that has available capacity, the processing tasks including determining speaker-related information and translating the utterance in the first language into a message in a second language.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 22, 2016
From: ELWHA LLC
To: MICROSOFT TECHNOLOGY LICENSING, LLC
Reel/Frame 037786/0907 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 8, 2012
From: LORD, RICHARD T.; LORD, ROBERT W.; MYHRVOLD, NATHAN P.; TEGREENE, CLARENCE T.; HYDE, RODERICK A.; WOOD, LOWELL L., JR.; ISHIKAWA, MURIEL Y.; WOOD, VICTORIA Y.H.; WHITMER, CHARLES; BAHL, PARAMVIR; BURGER, DOUGLAS C.; CHANDRA, RANVEER; GATES, WILLIAM H., III; HOLMAN, PAUL; KARE, JORDIN T.; MUNDIE, CRAIG J.; PAEK, TIM; TAN, DESNEY S.; ZHONG, LIN; DYOR, MATTHEW G.
To: ELWHA LLC
Reel/Frame 028176/0143 →
Continuity (3)
Continuation In Part 13309248 · Dec 1, 2011
Continuation In Part 13324232 · Dec 13, 2011
Related Publication 20130144595A1 · Jun 6, 2013