IP Library › Granted Patent US 12,561,289
Granted Patent B2
US 12,561,289 · App. 18/206,567 · Granted Feb 24, 2026

Systems and methods for generating and using shared natural language libraries

Inventor: Keyvan Mohajer (Los Gatos, CA)
Assignee: SOUNDHOUND AI IP, LLC
G06F16/1748G06F16/3334G06F16/3344G10L15/063G10L15/183G10L2015/0635
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,561,289
App. No.
18/206,567
Granted
Feb 24, 2026
Kind
B2
Abstract

Systems and methods for searching databases by sound data input are provided herein. A service provider may have a need to make their database(s) searchable through search technology. However, the service provider may not have the resources to implement such search technology. The search technology may allow for search queries using sound data input. The technology described herein provides a solution addressing the service provider's need, by giving a search technology that furnishes search results in a fast, accurate manner. In further embodiments, systems and methods to monetize those search results are also described herein.

Claims (26)

1 . A method for converting speech to text, the method comprising:

receiving a sound input, by a computer system, from a client device;

determining at least one natural language query from the sound input;

analyzing at least a portion of the natural language query, prior to fully converting the sound input into text, to determine a topic of the natural language query;

selecting a customized language model specifically trained as a large language model on the topic; and

fully converting the sound input of the natural language query into text using the customized language model;

wherein the customized language model is selected from a plurality of natural language libraries or sub-libraries, each aggregated from multiple service providers and stored in the cloud.

2 . The method according to claim 1 , wherein analyzing at least a portion of the natural language query to determine a topic of the natural language query comprises matching the at least a portion of the natural language query to a natural language library that is associated with the topic.

3 . The method according to claim 1 , the method further comprising:

updating content of the customized language model responsive to encountering a new type of search query.

4 . The method according to claim 3 , wherein updating the content of the customized language model is performed by a natural language library update.

5 . The method according to claim 1 , further comprising:

utilizing a natural language query processor to narrow down the possibilities in the language model to a smaller set.

6 . A non-transitory computer-readable storage medium storing instructions for converting speech to text, the instructions when executed by a computer processor performing actions comprising:

receiving a sound input, by a computer system, from a client device;

determining at least one natural language query from the sound input;

analyzing at least a portion of the natural language query, prior to fully converting the sound input into text, to determine a topic of the natural language query;

selecting a customized language model specifically trained as a large language model on the topic; and

fully converting the sound input of the natural language query into text using the customized language model;

wherein the customized language model is selected from a plurality of natural language libraries or sub-libraries, each aggregated from multiple service providers and stored in the cloud.

7 . The non-transitory computer-readable storage medium according to claim 6 , wherein analyzing at least a portion of the natural language query to determine a topic of the natural language query comprises matching the at least a portion of the natural language query to a natural language library that is associated with the topic.

8 . The non-transitory computer-readable storage medium according to claim 6 , the actions further comprising:

updating content of the customized language model responsive to encountering a new type of search query.

9 . The non-transitory computer-readable storage medium according to claim 8 , wherein updating the content of the customized language model is performed by a natural language library update.

10 . The non-transitory computer-readable storage medium according to claim 6 , the actions further comprising:

utilizing a natural language query processor to narrow down the possibilities in the language model to a smaller set.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 27, 2023
From: SOUNDHOUND AI IP HOLDING, LLC
To: SOUNDHOUND AI IP, LLC
Reel/Frame 064205/0676 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 23, 2023
From: SOUNDHOUND, INC.
To: SOUNDHOUND AI IP HOLDING, LLC
Reel/Frame 064083/0484 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 6, 2023
From: MOHAJER, KEYVAN
To: SOUNDHOUND, INC.
Reel/Frame 063872/0351 →
Continuity (6)
Continuation 15390441 · Dec 23, 2016
Continuation 14028166 · Sep 16, 2013
Continuation 13480400 · May 24, 2012
Continuation In Part 12861775 · Aug 23, 2010
Provisional Application 61368999 · Jul 29, 2010
Related Publication 20230325358A1 · Oct 12, 2023
References Cited (163)
US 5457768A · Tsuboi et al. · 1995 [cited by applicant]
US 5774841A · Salazar et al. · 1998 [cited by applicant]
US 5969283A · Looney et al. · 1999 [cited by applicant]
US 5974409A · Sanu et al. · 1999 [cited by applicant]
US 6246986B1 · Ammicht et al. · 2001 [cited by applicant]
US 6496799B1 · Pickering · 2002 [cited by applicant]
US 7092888B1 · McCarthy · 2006 [cited by applicant]
US 7379867B2 · Chelba · 2008 [cited by applicant]
US 7440895B1 · Miller · 2008 [cited by applicant]
US 7693720B2 · Kennewick et al. · 2010 [cited by applicant]
US 7765097B1 · Yu et al. · 2010 [cited by applicant]
US 8296179B1 · Rennison · 2012 [cited by applicant]
US 8843369B1 · Sharifi · 2014 [cited by applicant]
US 9437186B1 · Liu et al. · 2016 [cited by applicant]
US 9646628B1 · Carlson et al. · 2017 [cited by applicant]
US 9843861B1 · Termeulen · 2017 [cited by applicant]
US 10096328B1 · Markovich-Golan et al. · 2018 [cited by applicant]
US 10134425B1 · Johnson, Jr. · 2018 [cited by applicant]
US 10224030B1 · Kiss et al. · 2019 [cited by applicant]
US 10339918B2 · Hofer et al. · 2019 [cited by applicant]
US 10510340B1 · Fu et al. · 2019 [cited by applicant]
US 10854192B1 · Maas et al. · 2020 [cited by applicant]
US 10923111B1 · Fan et al. · 2021 [cited by applicant]
US 10943606B2 · Doshi et al. · 2021 [cited by applicant]
US 11308960B2 · Aguayo et al. · 2022 [cited by applicant]
US 11862162B2 · Aguayo et al. · 2024 [cited by applicant]
US 20010014891A1 · Hoffert et al. · 2001 [cited by applicant]
US 20020138630A1 · Solomon et al. · 2002 [cited by applicant]
US 20030061042A1 · Garudadri · 2003 [cited by applicant]
US 20030069880A1 · Harrison et al. · 2003 [cited by applicant]
US 20030078928A1 · Dorosario · 2003 [cited by examiner]
US 20030106413A1 · Samadani et al. · 2003 [cited by applicant]
US 20040019497A1 · Volk et al. · 2004 [cited by applicant]
US 20040193420A1 · Kennewick · 2004 [cited by examiner]
US 20050016360A1 · Zhang · 2005 [cited by applicant]
US 20050027699A1 · Awadallah et al. · 2005 [cited by applicant]
US 20050137939A1 · Calabria · 2005 [cited by examiner]
US 20050144064A1 · Calabria · 2005 [cited by examiner]
US 20050144065A1 · Calabria · 2005 [cited by examiner]
US 20050159945A1 · Otsuka et al. · 2005 [cited by applicant]
US 20060155694A1 · Chowdhury et al. · 2006 [cited by applicant]
US 20060200350A1 · Attwater et al. · 2006 [cited by applicant]
US 20060242017A1 · Libes et al. · 2006 [cited by applicant]
US 20070156392A1 · Balchandran et al. · 2007 [cited by applicant]
US 20070288444A1 · Nelken · 2007 [cited by examiner]
US 20080071520A1 · Sanford · 2008 [cited by applicant]
US 20080154951A1 · Martinez et al. · 2008 [cited by applicant]
US 20080228496A1 · Yu et al. · 2008 [cited by applicant]
US 20080249982A1 · Lakowske · 2008 [cited by applicant]
US 20080255937A1 · Chang et al. · 2008 [cited by applicant]
US 20090064029A1 · Corkran · 2009 [cited by examiner]
US 20100124892A1 · Issa · 2010 [cited by examiner]
US 20100205166A1 · Boulter et al. · 2010 [cited by applicant]
US 20100250497A1 · Redlich et al. · 2010 [cited by applicant]
US 20110071819A1 · Miller et al. · 2011 [cited by applicant]
US 20110153318A1 · Rossello et al. · 2011 [cited by applicant]
US 20110231185A1 · Kleffner et al. · 2011 [cited by applicant]
US 20120046936A1 · Kandekar · 2012 [cited by examiner]
US 20120065960A1 · Iwama et al. · 2012 [cited by applicant]
US 20120216178A1 · Gellerich et al. · 2012 [cited by applicant]
US 20130055223A1 · Xu · 2013 [cited by applicant]
US 20130060571A1 · Soemo et al. · 2013 [cited by applicant]
US 20140019483A1 · Mohajer · 2014 [cited by applicant]
US 20140039895A1 · Aravamudan et al. · 2014 [cited by applicant]
US 20150112679A1 · Zhang · 2015 [cited by applicant]
US 20160148615A1 · Lee et al. · 2016 [cited by applicant]
US 20160379632A1 · Hoffmeister et al. · 2016 [cited by applicant]
US 20170069309A1 · Aleksic et al. · 2017 [cited by applicant]
US 20170110118A1 · Tadpatrikar et al. · 2017 [cited by applicant]
US 20170178623A1 · Shamir et al. · 2017 [cited by applicant]
US 20170256259A1 · Froelich · 2017 [cited by applicant]
US 20180012616A1 · Salishev · 2018 [cited by applicant]
US 20180061399A1 · Rose et al. · 2018 [cited by applicant]
US 20180090127A1 · Hofer et al. · 2018 [cited by applicant]
US 20180102135A1 · Ebenezer · 2018 [cited by applicant]
US 20180102136A1 · Ebenezer · 2018 [cited by applicant]
US 20180192191A1 · TerMeulen et al. · 2018 [cited by applicant]
US 20180270565A1 · Ganeshkumar · 2018 [cited by applicant]
US 20180308489A1 · Pan et al. · 2018 [cited by applicant]
US 20180330723A1 · Acero et al. · 2018 [cited by applicant]
US 20190007540A1 · Shaik et al. · 2019 [cited by applicant]
US 20190098399A1 · Lashkari · 2019 [cited by applicant]
US 20190251955A1 · Degraye et al. · 2019 [cited by applicant]
US 20190311720A1 · Pasko · 2019 [cited by applicant]
US 20190385635A1 · Shahen Tov et al. · 2019 [cited by applicant]
US 20200213726A1 · Dyrholm · 2020 [cited by applicant]
US 20200302922A1 · Jazi et al. · 2020 [cited by applicant]
US 20210014599A1 · Apelqvist et al. · 2021 [cited by applicant]
US 20210043223A1 · Lee et al. · 2021 [cited by applicant]
US 20210306751A1 · Roach et al. · 2021 [cited by applicant]
US 20230110708A1 · Hui · 2023 [cited by applicant]
US 20240312466A1 · Reouveni et al. · 2024 [cited by applicant]
CN 1579059 · 2005 [cited by applicant]
EP 1679692 · 2006 [cited by applicant]
EP 2257034 · 2010 [cited by applicant]
JP 05189475 · 1993 [cited by applicant]
JP 2001014321 · 2001 [cited by applicant]
JP 2001142481 · 2001 [cited by applicant]
JP 2001169269 · 2001 [cited by applicant]
JP 2001243240 · 2001 [cited by applicant]
JP 2002197107 · 2002 [cited by applicant]
JP 2003223461 · 2003 [cited by applicant]
JP 2004102662 · 2004 [cited by applicant]
JP 2005332380 · 2005 [cited by applicant]
JP 2006039881 · 2006 [cited by applicant]
JP 2006065508 · 2006 [cited by applicant]
WO 0126093 · 2001 [cited by applicant]
WO 2008004181 · 2008 [cited by applicant]
WO 2009009522 · 2009 [cited by applicant]
WO 2013177213 · 2013 [cited by applicant]
WO 2016007563 · 2016 [cited by applicant]
Avery Li-Chun Wang, An Industrial-Strength Audio Search Algorithm, In ISMIR 2003, 4th Symposium Conference on Music Information Retrieval (Oct. 26, 2003), pp. 7-13. [cited by applicant]
Venkatachalam, V., Cazzanti, L., Chillon, N., Wells, M., Automatic Identification of Sound Recordings, Signal Processing Magazine, IEEE, Mar. 2004, 92-99, vol. 21, Issue 2. [cited by applicant]
Nelson, Jeffrey, V Cast Song ID from Verizon Wireless. May 21, 2007. [cited by applicant]
Gracenote: MusicID, available at http://www.gracenote.com/business solutions/music id/, last accessed Aug. 4, 2010. [cited by applicant]
Shazam: http://web.archive.org/web/20100501190631/http://www.shazam.com/. Last accessed May 1, 2010. [cited by applicant]
Gracenote Mobile MusicID: http://web.archive.org/web/20100123211802/http://www.gracenote.com/business_solutions/mobile Music/. Last accessed Jan. 23, 2010. [cited by applicant]
App Shopper MusicID: http://appshopper.com/music/musicid. Last changed Feb. 8, 2013. [cited by applicant]
Wang. “The Shazam Music Recognition Service,” Communications of the ACM (Magazine), Aug. 2006. vol. 49, No. 8, p. 44-48. http://dl.acm.org/citation.cfm?id=1145312. [cited by applicant]
Xu et al. “Music Identification Via Vocabulary Tree with MFCC Peaks,” MIRUM '11 Proceedings of the 1st international ACM workshop on Music information retrieval with user-centered and multimodal strategies, 2011. p. 21-… [cited by applicant]
Li et al. “Robust Audio Identification for MP3 Popular Music,” SIGIR '10 Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval, Jul. 2010. p. 627-634. http://dl.… [cited by applicant]
Yu et al. “A Query-By-Singing System for Retrieving Karaoke Music,” IEEE Transactions on Multimedia, Dec. 2008, vol. 10, No. 8, p. 1626-1637. http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=4694852. [cited by applicant]
Casey et al. “Content-Based Music Information Retrieval: Current Directions and Future Challenges,” Proceedings of the IEEE, 2008. vol. 96, No. 4, p. 668-696. http://research.yahoo.com/pub/2649. [cited by applicant]
Liu et al. “Content-Based Retrieval of MP3 Music Objects,” CIKM '01 Proceedings of the tenth international conference on Information and knowledge management, 2001. p. 506-511. http://dx.doi.org/10.1145/502585.502670. [cited by applicant]
OMRAS2—Ontology-Driven Music Retrieval & Annotation Sharing Service. Overview—Apr. 24, 2009 [Accessed Sep. 27, 2012—Archive.org] http://web.archive.org/web/20090424083019/http://www.omras2.orq/overview. [cited by applicant]
OMRAS2—AudioDB—Populating and Querying an AudioDB Instance. (No archived version available—accessed Sep. 27, 2012 via Google) http://omras2.orq/audioDB/tutorial1. [cited by applicant]
Benson et al. “Sync Kit: A Persistent Client-Side Database Caching Toolkit for Data Intensive Websites,” Proceedings Of The 19th International Conference On World Wide Web, Apr. 2010. oo. 121-130. http://dl.acm.orq/cita… [cited by applicant]
Larson et al. “NYT to Release Thesaurus and Enter Linked Data Cloud,” NY Times Biogs, Jun. 2009. http://open.blogs.nytimes.com/2009/06/26/nyt-to-release-thesaurusand-ent- erlinked-data-cloud/. [cited by applicant]
“Aurix Enhances Effectiveness Of Leading Search Software,” Aurix.com—News. Jun. 1, 2010. http://www.aurix.com/pages/3808/Aurix_enhances_effectiveness_of_leading_search_software.htm. [cited by applicant]
“Hearing it Loud & Clear at SpeechTEK 201 O,” Aurix.com—News. Jul. 21, 2010, http://www.aurix.com/pages/4161/State_of_the_art_speech_technology.htm. [cited by applicant]
Jamil. “A Natural Language Interface Plug-In For Cooperative Query Answering In Biological Databases,” BMC Genomics, Nov. 2011. (Accessed Sep. 27, 2012 via SpringerLink) http://www.biomedcentral.com/14 71-2164/13/S3/S4. [cited by applicant]
Feng. “A General Framework for Building Natural Language Understanding Modules in Voice Search,” 2010 IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP), Mar. 2010. (Accessed Sep. 27, 2012—… [cited by applicant]
Langanke. “Direct Voice Control Speech Data Entry and Database Query Models,” International Symposium on Logistics and Industrial Informatics, Sep. 2007. (Accessed Sep. 27, 2012—IEEE) http://ieeexplore.ieee.org/xpl/arti… [cited by applicant]
Indukuri et al. “Natural Language Querying Over Databases Using Cascaded CRFs,” Lecture Notes in Computer Science, Sep. 2010, http://www.springerlink.com/contenU5w1x27650475304m. [cited by applicant]
Kolias et al. “Design and implementation of a VoiceXML-driven wiki application for assistive environments on the web,” Personal and Ubiquitous Computing, Sep. 2010. vol. 14, No. 6, p. 527-539, http://www.icsd.aegean.gr/… [cited by applicant]
International Search Report & Written Opinion dated Dec. 2, 2013 in Application No. PCT/US13/42097, filed May 21, 2013. [cited by applicant]
United States Office Action, U.S. Appl. No. 13/480,400, May 7, 2013, 15 pages. [cited by applicant]
United States Office Action, U.S. Appl. No. 13/480,400, Nov. 29, 2012, 11 pages. [cited by applicant]
United States Office Action, U.S. Appl. No. 12/861,775, Dec. 10, 2012, 12 pages. [cited by applicant]
United States Office Action, U.S. Appl. No. 14/028,166, Sep. 28, 2016, 13 pages. [cited by applicant]
United States Office Action, U.S. Appl. No. 14/028,166, Feb. 9, 2016, 9 pages. [cited by applicant]
Japanese Office Action, Japanese Application No. 2015-514132, Apr. 4, 2017, 9 pages. [cited by applicant]
European Examination Report, European Application No. 13793693.6, Jul. 14, 2017, 7 pages. [cited by applicant]
Requirement for Restriction/Election dated Jun. 26, 2019 in U.S. Appl. No. 15/390,441. [cited by applicant]
Response to Requirement for Restriction/Election dated Oct. 22, 2019 in U.S. Appl. No. 15/390,441. [cited by applicant]
Office Action dated Jan. 27, 2020 in U.S. Appl. No. 15/390,441. [cited by applicant]
Response to Office Action dated May 26, 2020 in U.S. Appl. No. 15/390,441. [cited by applicant]
Final Office Action dated Aug. 11, 2020 in U.S. Appl. No. 15/390,441. [cited by applicant]
Response to Final Office Action dated Nov. 9, 2020 in U.S. Appl. No. 15/390,441. [cited by applicant]
Office Action dated Jun. 10, 2021 in U.S. Appl. No. 15/390,441. [cited by applicant]
Response to Office Action dated Sep. 10, 2021 in U.S. Appl. No. 15/390,441. [cited by applicant]
Final Office Action dated Dec. 10, 2021 in U.S. Appl. No. 15/390,441. [cited by applicant]
Appeal Brief dated Mar. 10, 2022 in U.S. Appl. No. 15/390,441. [cited by applicant]
Examiner's Answer dated Jun. 20, 2022 in U.S. Appl. No. 15/390,441. [cited by applicant]
Reply Brief dated Jun. 22, 2022 in U.S. Appl. No. 15/390,441. [cited by applicant]
Office Action dated Apr. 4, 2023 in U.S. Appl. No. 17/698,623. [cited by applicant]
English language Abstract for CN1579059 published Feb. 9, 2005. [cited by applicant]
Notice of Allowance dated Feb. 12, 2025 in U.S. Appl. No. 18/375,906. [cited by applicant]
Notice of Allowance dated Mar. 19, 2025 in U.S. Appl. No. 18/401,770. [cited by applicant]
Office Action dated Aug. 23, 2024 in U.S. Appl. No. 18/375,906. [cited by applicant]
Office Action dated Oct. 16, 2024 in U.S. Appl. No. 18/401,770. [cited by applicant]
Notice of Allowance dated Nov. 20, 2024 in U.S. Appl. No. 18/375,906. [cited by applicant]
Office Action dated Nov. 27, 2024 in U.S. Appl. No. 18/047,650. [cited by applicant]