IP Library › Granted Patent US 12,374,330
Granted Patent B1
US 12,374,330 · App. 17/725,540 · Granted Jul 29, 2025

Action registry management via assistant application and device software development toolkits

Inventors: Alexey Gennadyevich Surkov (Kirkland, WA); Nishant Singhal (Redmond, WA); Alexandru Petrescu (Honolulu, HI)
Assignee: Meta Platforms, Inc.
G10L15/22G10L15/083G10L15/1815H04L67/133G10L2015/223
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,374,330
App. No.
17/725,540
Filed
Apr 20, 2022
Granted
Jul 29, 2025
Kind
B1
Art Unit
2659
USPC
704/275
Abstract

In one embodiment, a method includes detecting a change in context of a computing system to a current context, changing a first registration status of a first set of actions with an assistant system associated with the computing system based on the change in context to the current context, where the first application is in a foreground of a user interface of the computing system, changing a second registration status of a second set of actions with the assistant system based on the change in context to the current context, where the second application is in a background of the user interface of the computing system, updating a list of registered actions for the assistant system based on the changed first and second registration statuses, and storing the updated list of registered actions on the computing system.

Claims (56)

1. A method comprising, by a computing system:

storing a first application manifest corresponding to a first application, wherein the first application manifest comprises a first set of actions with an assistant system associated with the computing system and one or more action identifiers corresponding to each action of the first set of actions;

storing a second application manifest corresponding to a second application, wherein the second application manifest comprises a second set of actions with the assistant system and one or more action identifiers corresponding to each action of the second set of actions;

detecting a change in context of the computing system to a current context;

changing a first registration status of a first set of actions with the assistant system associated with the computing system based on the change in context to the current context, wherein each action in the first set of actions corresponds to a voice-initiated action executable by the first application, via the assistant system, wherein the first application is in a foreground of a user interface of the computing system;

changing a second registration status of a second set of actions with the assistant system based on the change in context to the current context, wherein each action in the second set of actions corresponds to a voice-initiated action executable by the second application, via the assistant system, wherein the second application is in a background of the user interface of the computing system;

updating a list of registered actions for the assistant system based on the changed first and second registration statuses; and

storing the updated list of registered actions on the computing system.

2. The method of claim 1 , wherein the first application and the second application are different applications currently running on the computing system.

3. The method of claim 1 , wherein the first application and the second application are the same application, and wherein the change in context is a change in view state within the application.

4. The method of claim 1 , wherein the change in context comprises one or more of:

a change in geographic location of the computing system,

a change in pose of the computing system,

a change in pose of a user of the computing system,

a selection of the first application, or

a termination of the second application.

5. The method of claim 1 , wherein the change in context comprises moving the first application to the foreground of the user interface of the computing system and moving the second application to the background of the user interface of the computing system.

6. The method of claim 1 , further comprising: receiving a voice input from a first user;

determining the voice input corresponds to a first action from the first set of actions;

determining whether the first action is a registered action based on the updated list of registered actions;

executing, via the assistant system, the first action responsive to determining the first action is a registered action; and

presenting an output of the executed first action in the user interface of the computing system.

7. The method of claim 1 , further comprising: receiving a voice input from a first user;

determining the voice input corresponds to a second action from the second set of actions; determining whether the second action is a registered action based on the updated list of registered actions; executing, via the assistant system, the second action responsive to determining the second action is a registered action; and presenting results of the executed second action in the user interface of the computing system.

8. The method of claim 1 , further comprising: receiving a voice input from a first user;

determining the voice input corresponds to a third action from a third set of actions; identifying the third action is an unregistered action based on the updated list of registered actions; and

presenting an indication the third action cannot be executed in the user interface of the computing system.

9. The method of claim 1 , wherein the computing system comprises a message bus configured to multiplex application programming interface (API) calls to and from the assistant system.

10. The method of claim 9 , further comprising receiving, from the first application or the second application, an API call to add one or more actions to the first set of actions or the second set of actions.

11. The method of claim 9 , wherein changing the first registration status of the first set of actions comprises registering the first set of actions via an API call from the first application to the assistant system.

12. The method of claim 9 , wherein changing the first registration status of the first set of actions comprises deregistering the first set of actions via an API call from the first application to the assistant system.

13. The method of claim 9 , wherein changing the second registration status of the second set of actions comprises registering the second set of actions via an API call from the second application to the assistant system.

14. The method of claim 9 , wherein changing the second registration status of the second set of actions comprises deregistering the second set of actions via an API call from the second application to the assistant system.

15. The method of claim 1 , further comprising: receiving a voice input from a first user; and

parsing the voice input to identify one or more intents and one or more slots, wherein the identified one or more intents corresponds to a respective one or more first actions of the first set of actions or a respective one or more second actions of the second set of actions.

16. The method of claim 1 ,

wherein the first application manifest further comprises one or more deep links corresponding to the first application.

17. The method of claim 1 ,

wherein the second application manifest further comprises one or more deep links corresponding to the second application.

18. The method of claim 1 , wherein the first set of actions comprises a first set of foreground actions and a first set of background actions, and wherein the second set of actions comprises a second set of foreground actions and a second set of background actions, wherein each of the foreground actions is a voice-initiated action that is registered to be performed by the respective application when the respective application is in the foreground of the user interface of the computing system, and wherein each of the background actions is a voice-initiated action that is registered to be performed by the respective application when the respective application is in the background of the user interface of the computing system.

19. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

store a first application manifest corresponding to a first application, wherein the first application manifest comprises a first set of actions with an assistant system associated with the computing system and one or more action identifiers corresponding to each action of the first set of actions;

store a second application manifest corresponding to a second application, wherein the second application manifest comprises a second set of actions with the assistant system and one or more action identifiers corresponding to each action of the second set of actions;

detect a change in context of the computing system to a current context;

change a first registration status of a first set of actions with the assistant system associated with the computing system based on the change in context to the current context, wherein each action in the first set of actions corresponds to a voice-initiated action executable by the first application, via the assistant system, wherein the first application is in a foreground of a user interface of the computing system;

change a second registration status of a second set of actions with the assistant system based on the change in context to the current context, wherein each action in the second set of actions corresponds to a voice-initiated action executable by the second application, via the assistant system, wherein the second application is in a background of the user interface of the computing system;

update a list of registered actions for the assistant system based on the changed first and second registration statuses; and

store the updated list of registered actions on the computing system.

20. A system comprising: one or more processors; and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:

store a first application manifest corresponding to a first application, wherein the first application manifest comprises a first set of actions with an assistant system associated with the computing system and one or more action identifiers corresponding to each action of the first set of actions;

store a second application manifest corresponding to a second application, wherein the second application manifest comprises a second set of actions with the assistant system and one or more action identifiers corresponding to each action of the second set of actions;

detect a change in context of the computing system to a current context;

change a first registration status of a first set of actions with the assistant system associated with the computing system based on the change in context to the current context, wherein each action in the first set of actions corresponds to a voice-initiated action executable by the first application, via the assistant system, wherein the first application is in a foreground of a user interface of the computing system;

change a second registration status of a second set of actions with the assistant system based on the change in context to the current context, wherein each action in the second set of actions corresponds to a voice-initiated action executable by the second application, via the assistant system, wherein the second application is in a background of the user interface of the computing system;

update a list of registered actions for the assistant system based on the changed first and second registration statuses; and

store the updated list of registered actions on the computing system.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 14, 2022
From: SURKOV, ALEXEY GENNADYEVICH; SINGHAL, NISHANT; PETRESCU, ALEXANDRU
To: META PLATFORMS, INC.
Reel/Frame 060509/0556 →
Continuity (1)
Provisional Application 63177812 · Apr 21, 2021
References Cited (332)
US 5282267A · Woo, Jr. · 1994 [cited by examiner]
US 5926537A · Birze · 1999 [cited by applicant]
US 6014439A · Walker et al. · 2000 [cited by applicant]
US 6504571B1 · Narayanaswami et al. · 2003 [cited by applicant]
US 7124123B1 · Roskind et al. · 2006 [cited by applicant]
US 7158678B2 · Nagel et al. · 2007 [cited by applicant]
US 7397912B2 · Aasman et al. · 2008 [cited by applicant]
US 7613287B1 · Stifelman et al. · 2009 [cited by applicant]
US 7889853B2 · Sutcliffe · 2011 [cited by applicant]
US 8027451B2 · Arendsen et al. · 2011 [cited by applicant]
US 8204200B2 · Wang et al. · 2012 [cited by applicant]
US 8560564B1 · Hoelzle et al. · 2013 [cited by applicant]
US 8660247B1 · Brown · 2014 [cited by applicant]
US 8677377B2 · Cheyer et al. · 2014 [cited by applicant]
US 8935192B1 · Ventilla et al. · 2015 [cited by applicant]
US 8983383B1 · Haskin · 2015 [cited by applicant]
US 9154739B1 · Nicolaou et al. · 2015 [cited by applicant]
US 9251471B2 · Pinckney et al. · 2016 [cited by applicant]
US 9299059B1 · Marra et al. · 2016 [cited by applicant]
US 9304736B1 · Whiteley et al. · 2016 [cited by applicant]
US 9338242B1 · Suchland et al. · 2016 [cited by applicant]
US 9338493B2 · Van Os et al. · 2016 [cited by applicant]
US 9390724B2 · List · 2016 [cited by applicant]
US 9418658B1 · David et al. · 2016 [cited by applicant]
US 9472206B2 · Ady · 2016 [cited by applicant]
US 9479931B2 · Ortiz, Jr. et al. · 2016 [cited by applicant]
US 9576574B2 · Van Os · 2017 [cited by applicant]
US 9639608B2 · Freeman · 2017 [cited by applicant]
US 9659577B1 · Langhammer · 2017 [cited by applicant]
US 9720955B1 · Cao et al. · 2017 [cited by applicant]
US 9747895B1 · Jansche et al. · 2017 [cited by applicant]
US 9792281B2 · Sarikaya · 2017 [cited by applicant]
US 9858925B2 · Gruber et al. · 2018 [cited by applicant]
US 9865260B1 · Vuskovic et al. · 2018 [cited by applicant]
US 9875233B1 · Tomkins et al. · 2018 [cited by applicant]
US 9875741B2 · Gelfenbeyn et al. · 2018 [cited by applicant]
US 9881077B1 · Alfonseca et al. · 2018 [cited by applicant]
US 9886953B2 · Lemay et al. · 2018 [cited by applicant]
US 9990591B2 · Gelfenbeyn et al. · 2018 [cited by applicant]
US 10042032B2 · Scott et al. · 2018 [cited by applicant]
US 10127220B2 · Bellegarda et al. · 2018 [cited by applicant]
US 10134395B2 · Typrin · 2018 [cited by applicant]
US 10162886B2 · Wang et al. · 2018 [cited by applicant]
US 10199051B2 · Binder et al. · 2019 [cited by applicant]
US 10241752B2 · Lemay et al. · 2019 [cited by applicant]
US 10276170B2 · Gruber et al. · 2019 [cited by applicant]
US 10311833B1 · Qiu et al. · 2019 [cited by applicant]
US 10387464B2 · Weston et al. · 2019 [cited by applicant]
US 10453454B2 · Homma et al. · 2019 [cited by applicant]
US 10462422B1 · Harrison et al. · 2019 [cited by applicant]
US D868793S · Germe · 2019 [cited by applicant]
US 10511808B2 · Harrison et al. · 2019 [cited by applicant]
US 10521185B1 · Kim et al. · 2019 [cited by applicant]
US D881883S · Germe · 2020 [cited by applicant]
US D882567S · Parfitt · 2020 [cited by applicant]
US D882570S · Germe · 2020 [cited by applicant]
US 10623897B1 · Wu et al. · 2020 [cited by applicant]
US 10685669B1 · Lan et al. · 2020 [cited by applicant]
US 10719786B1 · Treseler et al. · 2020 [cited by applicant]
US 10782986B2 · Martin · 2020 [cited by applicant]
US 10803050B1 · Salkola · 2020 [cited by applicant]
US 10841249B2 · Lim et al. · 2020 [cited by applicant]
US 10854206B1 · Liu et al. · 2020 [cited by applicant]
US 10878805B2 · Agarwal et al. · 2020 [cited by applicant]
US 10885819B1 · Verbeke et al. · 2021 [cited by applicant]
US 10896295B1 · Shenoy · 2021 [cited by applicant]
US 10949616B1 · Shenoy et al. · 2021 [cited by applicant]
US 10958599B1 · Penov et al. · 2021 [cited by applicant]
US 10977258B1 · Liu et al. · 2021 [cited by applicant]
US 10978056B1 · Challa et al. · 2021 [cited by applicant]
US 11003669B1 · Nelson et al. · 2021 [cited by applicant]
US 11003839B1 · Hatch · 2021 [cited by examiner]
US 11038974B1 · Koukoumidis et al. · 2021 [cited by applicant]
US 11042554B1 · Balakrishnan et al. · 2021 [cited by applicant]
US 11086858B1 · Koukoumidis et al. · 2021 [cited by applicant]
US 11175646B2 · Koch · 2021 [cited by applicant]
US 11195057B2 · Zadeh et al. · 2021 [cited by applicant]
US 11222627B1 · Qian et al. · 2022 [cited by applicant]
US 11308284B2 · Huang et al. · 2022 [cited by applicant]
US 11416556B2 · Misra et al. · 2022 [cited by applicant]
US 11437027B1 · Guo et al. · 2022 [cited by applicant]
US 20080240379A1 · Maislos et al. · 2008 [cited by applicant]
US 20080300884A1 · Smith · 2008 [cited by applicant]
US 20090055778A1 · Abdelazim · 2009 [cited by examiner]
US 20090282033A1 · Alshawi · 2009 [cited by applicant]
US 20110246383A1 · Gibson et al. · 2011 [cited by applicant]
US 20120245944A1 · Gruber et al. · 2012 [cited by applicant]
US 20120246191A1 · Xiong · 2012 [cited by applicant]
US 20120265528A1 · Gruber et al. · 2012 [cited by applicant]
US 20120311126A1 · Jadallah et al. · 2012 [cited by applicant]
US 20130035930A1 · Ferrucci et al. · 2013 [cited by applicant]
US 20130268839A1 · Lefebvre et al. · 2013 [cited by applicant]
US 20130275138A1 · Gruber et al. · 2013 [cited by applicant]
US 20130275164A1 · Gruber et al. · 2013 [cited by applicant]
US 20130278634A1 · Xu et al. · 2013 [cited by applicant]
US 20140012926A1 · Narayanan et al. · 2014 [cited by applicant]
US 20140032659A1 · Marini et al. · 2014 [cited by applicant]
US 20140074483A1 · Van Os · 2014 [cited by applicant]
US 20140129266A1 · Perl et al. · 2014 [cited by applicant]
US 20140164506A1 · Tesch et al. · 2014 [cited by applicant]
US 20140225918A1 · Mittal et al. · 2014 [cited by applicant]
US 20140244712A1 · Walters et al. · 2014 [cited by applicant]
US 20140278343A1 · Tran · 2014 [cited by applicant]
US 20140280017A1 · Indarapu et al. · 2014 [cited by applicant]
US 20140280302A1 · Morsi et al. · 2014 [cited by applicant]
US 20140297284A1 · Gruber et al. · 2014 [cited by applicant]
US 20140314338A1 · Hamel et al. · 2014 [cited by applicant]
US 20150081674A1 · Ali et al. · 2015 [cited by applicant]
US 20150134430A1 · Ellis et al. · 2015 [cited by applicant]
US 20150142420A1 · Sarikaya et al. · 2015 [cited by applicant]
US 20150142704A1 · London · 2015 [cited by applicant]
US 20150169284A1 · Quast et al. · 2015 [cited by applicant]
US 20150169572A1 · Taubman · 2015 [cited by examiner]
US 20150169744A1 · Walkingshaw et al. · 2015 [cited by applicant]
US 20150179168A1 · Hakkani-Tur et al. · 2015 [cited by applicant]
US 20150186156A1 · Brown et al. · 2015 [cited by applicant]
US 20150207765A1 · Brantingham et al. · 2015 [cited by applicant]
US 20150220888A1 · Iyer · 2015 [cited by applicant]
US 20150235447A1 · Abovitz et al. · 2015 [cited by applicant]
US 20150347375A1 · Tremblay et al. · 2015 [cited by applicant]
US 20160019290A1 · Ratnaparkhi et al. · 2016 [cited by applicant]
US 20160037311A1 · Cho · 2016 [cited by applicant]
US 20160063118A1 · Campbell et al. · 2016 [cited by applicant]
US 20160179462A1 · Bjorkengren · 2016 [cited by applicant]
US 20160196491A1 · Chandrasekaran et al. · 2016 [cited by applicant]
US 20160225370A1 · Kannan et al. · 2016 [cited by applicant]
US 20160255082A1 · Rathod · 2016 [cited by applicant]
US 20160260436A1 · Lemay et al. · 2016 [cited by applicant]
US 20160293164A1 · Shi · 2016 [cited by examiner]
US 20160306505A1 · Vigneras et al. · 2016 [cited by applicant]
US 20160308799A1 · Schubert et al. · 2016 [cited by applicant]
US 20160328096A1 · Tran et al. · 2016 [cited by applicant]
US 20160378849A1 · Myslinski · 2016 [cited by applicant]
US 20160378861A1 · Eledath et al. · 2016 [cited by applicant]
US 20170026318A1 · Daniel et al. · 2017 [cited by applicant]
US 20170076720A1 · Gopalan et al. · 2017 [cited by applicant]
US 20170083285A1 · Meyers et al. · 2017 [cited by applicant]
US 20170091168A1 · Bellegarda et al. · 2017 [cited by applicant]
US 20170092264A1 · Hakkani-Tur et al. · 2017 [cited by applicant]
US 20170132019A1 · Karashchuk et al. · 2017 [cited by applicant]
US 20170193390A1 · Weston et al. · 2017 [cited by applicant]
US 20170220119A1 · Potts et al. · 2017 [cited by applicant]
US 20170286401A1 · He et al. · 2017 [cited by applicant]
US 20170337742A1 · Powderly et al. · 2017 [cited by applicant]
US 20170353469A1 · Selekman et al. · 2017 [cited by applicant]
US 20170358304A1 · Castillo Sanchez et al. · 2017 [cited by applicant]
US 20170359707A1 · Diaconu et al. · 2017 [cited by applicant]
US 20180013699A1 · Sapoznik et al. · 2018 [cited by applicant]
US 20180018562A1 · Jung · 2018 [cited by applicant]
US 20180018987A1 · Zass · 2018 [cited by applicant]
US 20180040020A1 · Kurian et al. · 2018 [cited by applicant]
US 20180045963A1 · Hoover et al. · 2018 [cited by applicant]
US 20180046151A1 · Ahmed · 2018 [cited by applicant]
US 20180046861A1 · Patel et al. · 2018 [cited by applicant]
US 20180052824A1 · Ferrydiansyah et al. · 2018 [cited by applicant]
US 20180054523A1 · Zhang et al. · 2018 [cited by applicant]
US 20180096071A1 · Green · 2018 [cited by applicant]
US 20180096072A1 · He et al. · 2018 [cited by applicant]
US 20180107917A1 · Hewavitharana et al. · 2018 [cited by applicant]
US 20180121508A1 · Halstvedt · 2018 [cited by applicant]
US 20180189629A1 · Yatziv et al. · 2018 [cited by applicant]
US 20180196880A1 · Carter · 2018 [cited by applicant]
US 20180210874A1 · Fuxman et al. · 2018 [cited by applicant]
US 20180211665A1 · Park et al. · 2018 [cited by applicant]
US 20180293484A1 · Wang et al. · 2018 [cited by applicant]
US 20180308486A1 · Saddler et al. · 2018 [cited by applicant]
US 20180329998A1 · Thomson et al. · 2018 [cited by applicant]
US 20180364887A1 · Bell et al. · 2018 [cited by applicant]
US 20190012470A1 · Ross et al. · 2019 [cited by applicant]
US 20190034412A1 · Gupta · 2019 [cited by examiner]
US 20190035152A1 · Kazansky · 2019 [cited by applicant]
US 20190035390A1 · Howard et al. · 2019 [cited by applicant]
US 20190066670A1 · White et al. · 2019 [cited by applicant]
US 20190080698A1 · Miller · 2019 [cited by applicant]
US 20190082221A1 · Jain et al. · 2019 [cited by applicant]
US 20190087491A1 · Bax · 2019 [cited by applicant]
US 20190096129A1 · Rhodes et al. · 2019 [cited by applicant]
US 20190096398A1 · Sereshki · 2019 [cited by applicant]
US 20190103108A1 · Song et al. · 2019 [cited by applicant]
US 20190132265A1 · Nowak-Przygodzki et al. · 2019 [cited by applicant]
US 20190139150A1 · Brownhill et al. · 2019 [cited by applicant]
US 20190171943A1 · Pao et al. · 2019 [cited by applicant]
US 20190179611A1 · Wojogbe et al. · 2019 [cited by applicant]
US 20190180144A1 · Tsishkou et al. · 2019 [cited by applicant]
US 20190213490A1 · White et al. · 2019 [cited by applicant]
US 20190311036A1 · Shanmugam et al. · 2019 [cited by applicant]
US 20190311720A1 · Pasko · 2019 [cited by applicant]
US 20190324527A1 · Presant et al. · 2019 [cited by applicant]
US 20190324553A1 · Liu et al. · 2019 [cited by applicant]
US 20190324780A1 · Zhu et al. · 2019 [cited by applicant]
US 20190325042A1 · Yu et al. · 2019 [cited by applicant]
US 20190325080A1 · Natarajan et al. · 2019 [cited by applicant]
US 20190325081A1 · Liu et al. · 2019 [cited by applicant]
US 20190325084A1 · Peng et al. · 2019 [cited by applicant]
US 20190325864A1 · Anders et al. · 2019 [cited by applicant]
US 20190327330A1 · Natarajan et al. · 2019 [cited by applicant]
US 20190327331A1 · Natarajan et al. · 2019 [cited by applicant]
US 20190347837A1 · Donalek et al. · 2019 [cited by applicant]
US 20190348033A1 · Chen et al. · 2019 [cited by applicant]
US 20190361408A1 · Tokuchi · 2019 [cited by applicant]
US 20200013380A1 · Kakoyiannis · 2020 [cited by examiner]
US 20200027427A1 · Allen et al. · 2020 [cited by applicant]
US 20200211546A1 · Schairer et al. · 2020 [cited by applicant]
US 20200213663A1 · Okamoto · 2020 [cited by applicant]
US 20200221223A1 · Zhou · 2020 [cited by applicant]
US 20200228692A1 · Wakamatsu et al. · 2020 [cited by applicant]
US 20200241646A1 · Hebbalaguppe et al. · 2020 [cited by applicant]
US 20200274773A1 · Mortsolf · 2020 [cited by examiner]
US 20200285855A1 · Brebner · 2020 [cited by examiner]
US 20200304880A1 · Diaz Delgado · 2020 [cited by examiner]
US 20200314330A1 · Takayama · 2020 [cited by applicant]
US 20200394843A1 · Ramachandra Iyer · 2020 [cited by applicant]
US 20200409457A1 · Terrano et al. · 2020 [cited by applicant]
US 20200410012A1 · Moon et al. · 2020 [cited by applicant]
US 20200410891A1 · Baeuerle et al. · 2020 [cited by applicant]
US 20210048883A1 · Kelly et al. · 2021 [cited by applicant]
US 20210081698A1 · Lindeman et al. · 2021 [cited by applicant]
US 20210082403A1 · Agarwal et al. · 2021 [cited by applicant]
US 20210117479A1 · Liu et al. · 2021 [cited by applicant]
US 20210117623A1 · Aly et al. · 2021 [cited by applicant]
US 20210117712A1 · Huang et al. · 2021 [cited by applicant]
US 20210117780A1 · Malik et al. · 2021 [cited by applicant]
US 20210118440A1 · Peng et al. · 2021 [cited by applicant]
US 20210118442A1 · Poddar et al. · 2021 [cited by applicant]
US 20210120206A1 · Liu et al. · 2021 [cited by applicant]
US 20210247846A1 · Shriram et al. · 2021 [cited by applicant]
US 20220121884A1 · Zadeh et al. · 2022 [cited by applicant]
US 20220129123A1 · Nair · 2022 [cited by examiner]
US 20220219206A1 · Narayan · 2022 [cited by applicant]
US 20220269762A1 · Zhao et al. · 2022 [cited by applicant]
CA 3142624A1 · 2020 [cited by applicant]
GB 2581621A · 2020 [cited by applicant]
WO WO9401824A1 · 1994 [cited by examiner]
WO 2012116241A2 · 2012 [cited by applicant]
WO 2017027339A1 · 2017 [cited by applicant]
Co-pending U.S. Appl. No. 15/953,957, inventors Kemal; El Moujahid et al., filed Apr. 16, 2018. [cited by applicant]
Co-pending U.S. Appl. No. 15/967,193, inventors Testuggine; Davide et al., filed Apr. 30, 2018. [cited by applicant]
Co-pending U.S. Appl. No. 15/967,279, inventors Fuchun; Peng et al., filed Apr. 30, 2018. [cited by applicant]
Co-pending U.S. Appl. No. 16/025,317, inventors Gupta; Sonal et al., filed Jul. 2, 2018. [cited by applicant]
Co-pending U.S. Appl. No. 16/036,827, inventors Emmanouil; Koukoumidis et al., filed Jul. 16, 2018. [cited by applicant]
Co-pending U.S. Appl. No. 16/038,120, inventors Schissel; Jason et al., filed Jul. 17, 2018. [cited by applicant]
Co-pending U.S. Appl. No. 16/048,049, inventor Markku; Salkola, filed Jul. 27, 2018. [cited by applicant]
Co-pending U.S. Appl. No. 16/048,072, inventor Salkola; Markku, filed Jul. 27, 2018. [cited by applicant]
Co-pending U.S. Appl. No. 16/048,101, inventor Salkola; Markku, filed Jul. 27, 2018. [cited by applicant]
Co-pending U.S. Appl. No. 16/057,414, inventors Kahn; Jeremy Gillmor et al., filed Aug. 7, 2018. [cited by applicant]
Co-pending U.S. Appl. No. 16/103,775, inventors Zheng; Zhou et al., filed Aug. 14, 2018. [cited by applicant]
Co-pending U.S. Appl. No. 16/107,601, inventor Rajesh; Krishna Shenoy, filed Aug. 21, 2018. [cited by applicant]
Co-pending U.S. Appl. No. 16/107,847, inventors Rajesh; Krishna Shenoy et al., filed Aug. 21, 2018. [cited by applicant]
Co-pending U.S. Appl. No. 16/121,393, inventors Zhou; Zheng et al., filed Sep. 4, 2018. [cited by applicant]
Co-pending U.S. Appl. No. 16/127,173, inventors Zheng; Zhou et al., filed Sep. 10, 2018. [cited by applicant]
Co-pending U.S. Appl. No. 16/129,638, inventors Vivek; Natarajan et al., filed Sep. 12, 2018. [cited by applicant]
Co-pending U.S. Appl. No. 16/135,752, inventors Liu; Xiaohu et al., filed Sep. 19, 2018. [cited by applicant]
Co-pending U.S. Appl. No. 16/150,184, inventors Francislav; P. Penov et al., filed Oct. 2, 2018. [cited by applicant]
Co-pending U.S. Appl. No. 16/151,040, inventors Brian; Nelson et al., filed Oct. 3, 2018. [cited by applicant]
Co-pending U.S. Appl. No. 16/168,536, inventors Dumoulin; Benoit F. et al., filed Oct. 23, 2018. [cited by applicant]
Co-pending U.S. Appl. No. 16/176,081, inventors Anusha; Balakrishnan et al., filed Oct. 31, 2018. [cited by applicant]
Co-pending U.S. Appl. No. 16/176,312, inventors Koukoumidis; Emmanouil et al., filed Oct. 31, 2018. [cited by applicant]
Co-pending U.S. Appl. No. 16/182,542, inventors Michael; Hanson et al., filed Nov. 6, 2018. [cited by applicant]
Co-pending U.S. Appl. No. 16/183,650, inventors Xiaohu; Liu et al., filed Nov. 7, 2018. [cited by applicant]
Co-pending U.S. Appl. No. 16/192,538, inventor Koukoumidis; Emmanouil, filed Nov. 15, 2018. [cited by applicant]
Co-pending U.S. Appl. No. 16/222,923, inventors Jason; Schissel et al., filed Dec. 17, 2018. [cited by applicant]
Co-pending U.S. Appl. No. 16/222,957, inventors Emmanouil; Koukoumidis et al., filed Dec. 17, 2018. [cited by applicant]
Co-pending U.S. Appl. No. 16/229,828, inventors Xiaohu; Liu et al., filed Dec. 21, 2018. [cited by applicant]
Co-pending U.S. Appl. No. 16/247,439, inventors Xiaohu; Liu et al., filed Jan. 14, 2019. [cited by applicant]
Co-pending U.S. Appl. No. 16/264,173, inventors Ashwini; Challa et al., filed Jan. 31, 2019. [cited by applicant]
Co-pending U.S. Appl. No. 16/376,832, inventors Liu; Honglei et al., filed Apr. 5, 2019. [cited by applicant]
Co-pending U.S. Appl. No. 16/389,634, inventors Crook; Paul Anthony et al., filed Apr. 19, 2019. [cited by applicant]
Co-pending U.S. Appl. No. 16/389,728, inventors William; Presant et al., filed Apr. 19, 2019. [cited by applicant]
Co-pending U.S. Appl. No. 16/389,738, inventors Peng; Fuchun et al., filed Apr. 19, 2019. [cited by applicant]
Co-pending U.S. Appl. No. 16/389,769, inventors Liu; Honglei et al., filed Apr. 19, 2019. [cited by applicant]
Co-pending U.S. Appl. No. 16/434,010, inventors Dogaru; Sergiu et al., filed Jun. 6, 2019. [cited by applicant]
Co-pending U.S. Appl. No. 16/552,559, inventors Seungwhan; Moon et al., filed Aug. 27, 2019. [cited by applicant]
Co-pending U.S. Appl. No. 16/557,055, inventors Moon; Seungwhan et al., filed Aug. 30, 2019. [cited by applicant]
Co-pending U.S. Appl. No. 16/659,070, inventors Huang; Lisa Xiaoyi et al., filed Oct. 21, 2019. [cited by applicant]
Co-pending U.S. Appl. No. 16/659,203, inventors Huang; Lisa Xiaoyi et al., filed Oct. 21, 2019. [cited by applicant]
Co-pending U.S. Appl. No. 16/659,419, inventor Huang; Lisa Xiaoyi, filed Oct. 21, 2019. [cited by applicant]
Co-pending U.S. Appl. No. 16/703,700, inventors Ahmed; Aly et al., filed Dec. 4, 2019. [cited by applicant]
Co-pending U.S. Appl. No. 16/733,044, inventors Francislav; P. Penov et al., filed Jan. 2, 2020. [cited by applicant]
Co-pending U.S. Appl. No. 16/741,630, inventors Crook; Paul Anthony et al., filed Jan. 13, 2020. [cited by applicant]
Co-pending U.S. Appl. No. 16/741,642, inventors Fuchun; Peng et al., filed Jan. 13, 2020. [cited by applicant]
Co-pending U.S. Appl. No. 16/742,668, inventors Xiaohu; Liu et al., filed Jan. 14, 2020. [cited by applicant]
Co-pending U.S. Appl. No. 16/742,769, inventors Liu; Xiaohu et al., filed Jan. 14, 2020. [cited by applicant]
Co-pending U.S. Appl. No. 16/790,497, inventors Gao; Yang et al., filed Feb. 13, 2020. [cited by applicant]
Co-pending U.S. Appl. No. 16/815,960, inventors Malik; Kshitiz et al., filed Mar. 11, 2020. [cited by applicant]
Co-pending U.S. Appl. No. 16/815,990, inventors Kshitiz; Malik et al., filed Mar. 11, 2020. [cited by applicant]
Co-pending U.S. Appl. No. 16/842,366, inventors Sravani; Kamisetty et al., filed Apr. 7, 2020. [cited by applicant]
Co-pending U.S. Appl. No. 16/914,966, inventor Behar; Noam Yakob, filed Jun. 29, 2020. [cited by applicant]
Co-pending U.S. Appl. No. 16/917,664, inventors Liu; Xiaohu et al., filed Jun. 30, 2020. [cited by applicant]
Co-pending U.S. Appl. No. 16/921,665, inventors Liu; Honglei et al., filed Jul. 6, 2020. [cited by applicant]
Co-pending U.S. Appl. No. 16/998,423, inventors Armen; Aghajanyan et al., filed Aug. 20, 2020. [cited by applicant]
Co-pending U.S. Appl. No. 17/006,260, inventors William; Presant et al., filed Aug. 28, 2020. [cited by applicant]
Co-pending U.S. Appl. No. 17/006,339, inventors Shivani; Poddar et al., filed Aug. 28, 2020. [cited by applicant]
Co-pending U.S. Appl. No. 17/009,542, inventor Kottur; Satwik, filed Sep. 1, 2020. [cited by applicant]
Co-pending U.S. Appl. No. 17/035,253, inventors Khemka; Piyush et al., filed Sep. 28, 2020. [cited by applicant]
Co-pending U.S. Appl. No. 17/120,013, inventors Botros; Fadi et al., filed Dec. 11, 2020. [cited by applicant]
Co-pending U.S. Appl. No. 17/136,636, inventors Greenberg; Michael et al., filed Dec. 29, 2020. [cited by applicant]
Co-pending U.S. Appl. No. 17/139,363, inventors Cheng; Daniel Manhon et al., filed Dec. 31, 2020. [cited by applicant]
Co-pending U.S. Appl. No. 17/186,459, inventors Liu; Bing et al., filed Feb. 26, 2021. [cited by applicant]
Co-pending U.S. Appl. No. 17/336,716, inventors Chaland; Christophe et al., filed Jun. 2, 2021. [cited by applicant]
Co-pending U.S. Appl. No. 17/351,501, inventors Sethi; Pooja et al., filed Jun. 18, 2021. [cited by applicant]
Co-pending U.S. Appl. No. 17/391,765, inventors Pu; Yiming et al., filed Aug. 2, 2021. [cited by applicant]
Co-pending U.S. Appl. No. 17/394,096, inventors Wang; Emily et al., filed Aug. 4, 2021. [cited by applicant]
Co-pending U.S. Appl. No. 17/394,159, inventors Santoro; Elizabeth Kelsey et al., filed Aug. 4, 2021. [cited by applicant]
Co-pending U.S. Appl. No. 17/407,922, inventors Pu; Yiming et al., filed Aug. 20, 2021. [cited by applicant]
Co-pending U.S. Appl. No. 17/504,276, inventors Kottur; Satwik et al., filed Oct. 18, 2021. [cited by applicant]
Co-pending U.S. Appl. No. 17/512,478, inventors Chen; Zhiyu et al., filed Oct. 27, 2021. [cited by applicant]
Co-pending U.S. Appl. No. 17/512,490, inventors Presant; William Crosby et al., filed Oct. 27, 2021. [cited by applicant]
Co-pending U.S. Appl. No. 17/512,508, inventors Vincent; Joshuah et al., filed Oct. 27, 2021. [cited by applicant]
Co-pending U.S. Appl. No. 17/521,623, inventors Martinson; Leif Haven et al., filed Nov. 8, 2021. [cited by applicant]
Co-pending U.S. Appl. No. 17/524,598, inventors Hanson; Michael Robert et al., filed Nov. 11, 2021. [cited by applicant]
Co-pending U.S. Appl. No. 17/525,510, inventors Shrivastava; Akshat et al., filed Nov. 12, 2021. [cited by applicant]
Co-pending U.S. Appl. No. 17/543,178, inventors Desai; Shrey et al., filed Dec. 6, 2021. [cited by applicant]
Co-pending U.S. Appl. No. 17/566,467, inventors Goel; Swati et al., filed Dec. 30, 2021. [cited by applicant]
Co-pending U.S. Appl. No. 17/732,104, inventors Bi; Sheng et al., filed Apr. 28, 2022. [cited by applicant]
U.S. Appl. No. 62/660,876, inventors Kumar; Anuj et al., filed Apr. 20, 2018. [cited by applicant]
U.S. Appl. No. 62/675,090, inventors Hanson; Michael Robert et al., filed May 22, 2018. [cited by applicant]
U.S. Appl. No. 62/747,628, inventors Liu; Honglei et al., filed Oct. 18, 2018. [cited by applicant]
U.S. Appl. No. 62/749,608, inventors Challa; Ashwini et al., filed Oct. 23, 2018. [cited by applicant]
U.S. Appl. No. 62/750,746, inventors Liu; Honglei et al., filed Oct. 25, 2018. [cited by applicant]
U.S. Appl. No. 62/923,342, inventors Hanson; Michael Robert et al., filed Oct. 18, 2019. [cited by applicant]
Huang J., et al., “An Augmented Reality Sign-Reading Assistant for Users with Reduced Vision,” PLOS One, vol. 14, No. 1, Jan. 16, 2019, 9 pages. [cited by applicant]
International Search Report and Written Opinion for International Application No. PCT/US2022/024163, mailed Oct. 10, 2022, 9 pages. [cited by applicant]
Khan M.D., et al., “An AI-Based Visual Aid With Integrated Reading Assistant for the Completely Blind,” IEEE Tra1'isactions on Human-Machine Systems, vol. 50, No. 6, Dec. 2020, pp. 507-517. [cited by applicant]
Pintado D., et al., “Deep Learning Based Shopping Assistant for the Visually Impaired,” IEEE International Conference Onconsumer Electronics (ICCE), Jan. 11, 2019, 6 pages. [cited by applicant]
Co-pending U.S. Appl. No. 17/006,377, inventors Shivani; Poddar et al., filed Aug. 28, 2020. [cited by applicant]
International Search Report and Written Opinion for International Application No. PCT/US2022/025653, mailed Jul. 21, 2022, 11 pages. [cited by applicant]
International Search Report and Written Opinion for International Application No. PCT/US2022/025675, mailed Jul. 22, 2022, 14 pages. [cited by applicant]
International Search Report and Written Opinion for International Application No. PCT/US2022/025844, mailed Jul. 25, 2022, 14 pages. [cited by applicant]
EPO—Office Action mailed Jan. 2, 2025 for European Patent Application No. 22729316.4, filed on Apr. 20, 2022, 9 pages. [cited by applicant]
EPO—Office Action mailed Jan. 3, 2025 for European Patent Application No. 22729836.1, filed on Apr. 21, 2022, 5 pages. [cited by applicant]
EPO—Office Action mailed Dec. 23, 2024 for European Patent Application No. 22726559.2, filed on Apr. 21, 2022, 9 pages. [cited by applicant]
USPTO—Office Action mailed on Dec. 16, 2024 for related U.S. Appl. No. 17/725,540, 72 pages. [cited by applicant]
Cited By (3)
US 12,544,919 US 12,566,649 US 12,646,517