IP Library › Granted Patent US 12,243,521
Granted Patent B2
US 12,243,521 · App. 17/387,608 · Granted Mar 4, 2025

Multimodal transmission of packetized data

Inventors: Gaurav Bhaya (Sunnyvale, CA); Robert Stets (Mountain View, CA)
Assignee: GOOGLE LLC
G10L15/22G06F3/165G10L15/14G10L15/1822G10L15/26G10L15/30H04L47/25G10L2015/088
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,243,521
App. No.
17/387,608
Granted
Mar 4, 2025
Kind
B2
Abstract

A system of multi-modal transmission of packetized data in a voice activated data packet based computer network environment is provided. A natural language processor component can parse an input audio signal to identify a request and a trigger keyword. Based on the input audio signal, a direct action application programming interface can generate a first action data structure, and a content selector component can select a content item. An interface management component can identify first and second candidate interfaces, and respective resource utilization values. The interface management component can select, based on the resource utilization values, the first candidate interface to present the content item. The interface management component can provide the first action data structure to the client computing device for rendering as audio output, and can transmit the content item converted for a first modality to deliver the content item for rendering from the selected interface.

Claims (59)

1. A system to transmit data in a voice-based computing environment, comprising:

a data processing system comprising one or more processors and memory to:

receive, via an interface of the data processing system, data packets comprising an input audio signal detected by a sensor of a client computing device associated with an electronic account;

parse the input audio signal to identify a request;

select a content item responsive to the request;

identify a plurality of interfaces executed by one or more client computing devices linked with the electronic account, the one or more client computing devices comprising the client computing device;

determine a characteristic of each of the plurality of interfaces;

select, based on the characteristic of each of the plurality of interfaces executed by the one or more client computing devices linked with the electronic account, a first interface of the plurality of interfaces having a first characteristic, the first interface executed by a first computing device; and

provide, to the first computing device, the content item for presentation as audio output via the first interface.

2. The system of claim 1 , wherein the first computing device is the client computing device that detects the input audio signal.

3. The system of claim 1 , wherein the first computing device is different from the client computing device that detects the input audio signal.

4. The system of claim 1 , comprising the data processing system to:

generate, based on the request, a first action data structure; and

transmit the first action data structure to the first computing device with the content item.

5. The system of claim 1 , comprising:

the data processing system to provide the content item in a modality compatible with the first interface.

6. The system of claim 1 , comprising the data processing system to:

determine a capability of the first interface; and

convert the content item to a modality compatible with the capability of the first interface.

7. The system of claim 1 , wherein the first interface comprises an audio interface, comprising:

the data processing system to provide the content item for presentation via the audio interface.

8. The system of claim 1 , comprising the data processing system to:

select a second content item based on the first characteristic of the first interface; and

provide the second content item to the client computing device for presentation via the first interface.

9. The system of claim 1 , comprising the data processing system to:

parse the input audio signal to identify a keyword corresponding to the request; and

select the content item based at least on the keyword.

10. The system of claim 1 , comprising the data processing system to:

select a second content item; and

provide the second content item to the client computing device for presentation via a second interface of the client computing device that has a different characteristic than the first characteristic.

11. A method of transmitting data in a voice-based computing environment, comprising:

receiving, by a data processing system comprising one or more processors and memory, via an interface of the data processing system, data packets comprising an input audio signal detected by a sensor of a client computing device associated with an electronic account;

parsing, by the data processing system, the input audio signal to identify a request;

selecting, by the data processing system, a content item responsive to the request;

identifying, by the data processing system, a plurality of interfaces executed by one or more client computing devices linked with the electronic account, the one or more client computing devices comprising the client computing device;

determining, by the data processing system, a characteristic of each of the plurality of interfaces;

selecting, by the data processing system, based on the characteristic of each of the plurality of interfaces executed by the one or more client computing devices linked with the electronic account, a first interface of the plurality of interfaces having a first characteristic, the first interface executed by a first computing device; and

providing, by the data processing system, to the first computing device, the content item for presentation as audio output via the first interface.

12. The method of claim 11 , wherein the first computing device is the client computing device that detects the input audio signal.

13. The method of claim 11 , wherein the first computing device is different from the client computing device that detects the input audio signal.

14. The method of claim 11 , comprising:

generating, by the data processing system based on the request, a first action data structure; and

transmitting, by the data processing system, the first action data structure to the first computing device with the content item.

15. The method of claim 11 , comprising:

providing, by the data processing system, the content item in a modality compatible with the first interface.

16. The method of claim 11 , comprising:

determining, by the data processing system, a capability of the first interface; and

converting, by the data processing system, the content item to a modality compatible with the capability of the first interface.

17. The method of claim 11 , wherein the first interface comprises an audio interface, comprising:

providing, by the data processing system, the content item for presentation via the audio interface.

18. The method of claim 11 , comprising:

selecting, by the data processing system, a second content item based on the first characteristic of the first interface; and

providing, by the data processing system, the second content item to the client computing device for presentation via the first interface.

19. The method of claim 11 , comprising:

parsing, by the data processing system, the input audio signal to identify a keyword corresponding to the request; and

selecting, by the data processing system, the content item based at least on the keyword.

20. The method of claim 11 , comprising:

selecting, by the data processing system, a second content item; and

providing, by the data processing system, the second content item to the client computing device for presentation via a second interface of the client computing device that has a different characteristic than the first characteristic.

Continuity (4)
Continuation 16696622 · Nov 26, 2019
Continuation 16039202 · Jul 18, 2018
Continuation 15395703 · Dec 30, 2016
Related Publication 20210358498A1 · Nov 18, 2021
References Cited (400)
US 5675788A · Husick et al. · 1997 [cited by applicant]
US 5724521A · Dedrick · 1998 [cited by applicant]
US 5740549A · Reilly et al. · 1998 [cited by applicant]
US 5749069A · Komori et al. · 1998 [cited by applicant]
US 5848397A · Marsh et al. · 1998 [cited by applicant]
US 5850433A · Rondeau · 1998 [cited by applicant]
US 5930773A · Crooks et al. · 1999 [cited by applicant]
US 5948061A · Merriman et al. · 1999 [cited by applicant]
US 6026368A · Brown et al. · 2000 [cited by applicant]
US 6044376A · Kurtzman, II · 2000 [cited by applicant]
US 6078914A · Redfern · 2000 [cited by applicant]
US 6088688A · Crooks et al. · 2000 [cited by applicant]
US 6125284A · Moore et al. · 2000 [cited by applicant]
US 6144944A · Kurtzman et al. · 2000 [cited by applicant]
US 6167382A · Sparks et al. · 2000 [cited by applicant]
US 6189003B1 · Leal · 2001 [cited by applicant]
US 6259127B1 · Pan · 2001 [cited by applicant]
US 6259405B1 · Stewart et al. · 2001 [cited by applicant]
US 6269361B1 · Davis et al. · 2001 [cited by applicant]
US 6275806B1 · Pertrushin · 2001 [cited by applicant]
US 6332127B1 · Bandera et al. · 2001 [cited by applicant]
US 6401075B1 · Mason et al. · 2002 [cited by applicant]
US 6446045B1 · Stone et al. · 2002 [cited by applicant]
US 6600930B1 · Sakurai et al. · 2003 [cited by applicant]
US 6647257B2 · Owensby · 2003 [cited by applicant]
US 6665293B2 · Thornton et al. · 2003 [cited by applicant]
US 6677894B2 · Sheynblat et al. · 2004 [cited by applicant]
US 6684249B1 · Frerichs et al. · 2004 [cited by applicant]
US 6731612B1 · Koss · 2004 [cited by applicant]
US 6857007B1 · Bloomfield · 2005 [cited by applicant]
US 6956816B1 · Alexander et al. · 2005 [cited by applicant]
US 6983331B1 · Mitchell et al. · 2006 [cited by applicant]
US 6985882B1 · Del Sesto · 2006 [cited by applicant]
US 7016343B1 · Mermel et al. · 2006 [cited by applicant]
US 7039599B2 · Merriman et al. · 2006 [cited by applicant]
US 7120235B2 · Altberg et al. · 2006 [cited by applicant]
US 7136875B2 · Anderson et al. · 2006 [cited by applicant]
US 7240025B2 · Stone et al. · 2007 [cited by applicant]
US 7376640B1 · Anderson et al. · 2008 [cited by applicant]
US 7406434B1 · Chang et al. · 2008 [cited by applicant]
US 7613637B2 · Chan et al. · 2009 [cited by applicant]
US 7668950B2 · Horowitz et al. · 2010 [cited by applicant]
US 7853255B2 · Karaoguz et al. · 2010 [cited by applicant]
US 7881936B2 · Longe et al. · 2011 [cited by applicant]
US 7904460B2 · Armour et al. · 2011 [cited by applicant]
US 7920682B2 · Byrne et al. · 2011 [cited by applicant]
US 8041709B2 · Permandala et al. · 2011 [cited by applicant]
US 8068604B2 · Leeds et al. · 2011 [cited by applicant]
US 8108383B2 · Lin et al. · 2012 [cited by applicant]
US 8175914B1 · Benson et al. · 2012 [cited by applicant]
US 8195133B2 · Ramer et al. · 2012 [cited by applicant]
US 8204881B2 · Holt et al. · 2012 [cited by applicant]
US 8214342B2 · Meiresonne · 2012 [cited by applicant]
US 8234275B2 · Grant et al. · 2012 [cited by applicant]
US 8312014B2 · Lu et al. · 2012 [cited by applicant]
US 8326637B2 · Baldwin et al. · 2012 [cited by applicant]
US 8386386B1 · Zhu · 2013 [cited by applicant]
US 8453058B1 · Coccaro et al. · 2013 [cited by applicant]
US 8612226B1 · Epstein et al. · 2013 [cited by applicant]
US 8762156B2 · Chen · 2014 [cited by applicant]
US 8903716B2 · Chen et al. · 2014 [cited by applicant]
US 9153106B1 · Kuscher et al. · 2015 [cited by applicant]
US 9275637B1 · Salvador et al. · 2016 [cited by applicant]
US 9318107B1 · Sharifi · 2016 [cited by applicant]
US 9424841B2 · Foerster et al. · 2016 [cited by applicant]
US 9431006B2 · Bellegarda · 2016 [cited by applicant]
US 9542941B1 · Weksler et al. · 2017 [cited by applicant]
US 9653075B1 · Chen et al. · 2017 [cited by applicant]
US 10032452B1 · Bhaya · 2018 [cited by examiner]
US 10165091B1 · Bittfield et al. · 2018 [cited by applicant]
US 10193465B2 · Dai et al. · 2019 [cited by applicant]
US 10535348B2 · Bhaya · 2020 [cited by examiner]
US 11087760B2 · Bhaya · 2021 [cited by examiner]
US 20010020236A1 · Cannon · 2001 [cited by applicant]
US 20010025275A1 · Tanaka et al. · 2001 [cited by applicant]
US 20020029226A1 · Li et al. · 2002 [cited by applicant]
US 20020042829A1 · Mizuhara et al. · 2002 [cited by applicant]
US 20020077130A1 · Owensby · 2002 [cited by applicant]
US 20020082938A1 · Borger et al. · 2002 [cited by applicant]
US 20020091571A1 · Thomas et al. · 2002 [cited by applicant]
US 20020128908A1 · Levin et al. · 2002 [cited by applicant]
US 20020161646A1 · Gailey et al. · 2002 [cited by applicant]
US 20020164977A1 · Link, II et al. · 2002 [cited by applicant]
US 20020188680A1 · McCormack et al. · 2002 [cited by applicant]
US 20030008661A1 · Joyce et al. · 2003 [cited by applicant]
US 20030018479A1 · Oh et al. · 2003 [cited by applicant]
US 20030028529A1 · Cheung et al. · 2003 [cited by applicant]
US 20030032406A1 · Minear et al. · 2003 [cited by applicant]
US 20030032409A1 · Hutcheson et al. · 2003 [cited by applicant]
US 20030033079A1 · Endicott · 2003 [cited by applicant]
US 20030033292A1 · Meisel et al. · 2003 [cited by applicant]
US 20030046161A1 · Kamangar et al. · 2003 [cited by applicant]
US 20030061211A1 · Shultz et al. · 2003 [cited by applicant]
US 20030081594A1 · Lee · 2003 [cited by applicant]
US 20030105677A1 · Skinner · 2003 [cited by applicant]
US 20030125977A1 · Morioka et al. · 2003 [cited by applicant]
US 20030154072A1 · Young et al. · 2003 [cited by applicant]
US 20030220835A1 · Barnes, Jr. · 2003 [cited by applicant]
US 20040023666A1 · Moon et al. · 2004 [cited by applicant]
US 20040030556A1 · Bennett · 2004 [cited by applicant]
US 20040043770A1 · Amit et al. · 2004 [cited by applicant]
US 20040044565A1 · Kumar et al. · 2004 [cited by applicant]
US 20040044571A1 · Bronnimann et al. · 2004 [cited by applicant]
US 20040059712A1 · Dean et al. · 2004 [cited by applicant]
US 20040076279A1 · Taschereau · 2004 [cited by applicant]
US 20040083133A1 · Nicholas et al. · 2004 [cited by applicant]
US 20040085894A1 · Wang et al. · 2004 [cited by applicant]
US 20040120323A1 · Viikari et al. · 2004 [cited by applicant]
US 20040172389A1 · Galai et al. · 2004 [cited by applicant]
US 20040220778A1 · Imai et al. · 2004 [cited by applicant]
US 20050027659A1 · Kumar et al. · 2005 [cited by applicant]
US 20050065999A1 · Acharya et al. · 2005 [cited by applicant]
US 20050074102A1 · Altberg et al. · 2005 [cited by applicant]
US 20050076017A1 · Rein et al. · 2005 [cited by applicant]
US 20050086104A1 · McFadden · 2005 [cited by applicant]
US 20050097204A1 · Horowitz et al. · 2005 [cited by applicant]
US 20050137939A1 · Calabria et al. · 2005 [cited by applicant]
US 20050144065A1 · Calabria et al. · 2005 [cited by applicant]
US 20050144069A1 · Wiseman et al. · 2005 [cited by applicant]
US 20050187823A1 · Howes · 2005 [cited by applicant]
US 20050225810A1 · Sun · 2005 [cited by applicant]
US 20050234879A1 · Zeng et al. · 2005 [cited by applicant]
US 20060004627A1 · Baluja · 2006 [cited by applicant]
US 20060149624A1 · Baluja et al. · 2006 [cited by applicant]
US 20060190974A1 · Lee · 2006 [cited by applicant]
US 20060195819A1 · Chory et al. · 2006 [cited by applicant]
US 20060247913A1 · Huerta et al. · 2006 [cited by applicant]
US 20060274869A1 · Morse · 2006 [cited by applicant]
US 20060287919A1 · Rubens et al. · 2006 [cited by applicant]
US 20070005570A1 · Hurst-Hiller et al. · 2007 [cited by applicant]
US 20070088609A1 · Reller et al. · 2007 [cited by applicant]
US 20070097975A1 · Rakers et al. · 2007 [cited by applicant]
US 20070100822A1 · Freeman et al. · 2007 [cited by applicant]
US 20070127688A1 · Doulton · 2007 [cited by applicant]
US 20070282612A1 · Kaneko et al. · 2007 [cited by applicant]
US 20070294229A1 · Au · 2007 [cited by applicant]
US 20080021604A1 · Bouvier et al. · 2008 [cited by applicant]
US 20080021792A1 · Steelberg et al. · 2008 [cited by applicant]
US 20080049696A1 · Stewart · 2008 [cited by applicant]
US 20080052219A1 · Sandholm et al. · 2008 [cited by applicant]
US 20080071536A1 · Nagashima · 2008 [cited by applicant]
US 20080103781A1 · Wasson et al. · 2008 [cited by applicant]
US 20080146292A1 · Gilmore et al. · 2008 [cited by applicant]
US 20080227484A1 · Auvray et al. · 2008 [cited by applicant]
US 20080228494A1 · Cross · 2008 [cited by applicant]
US 20080243785A1 · Stading · 2008 [cited by applicant]
US 20080249855A1 · Collins et al. · 2008 [cited by applicant]
US 20080270223A1 · Collins et al. · 2008 [cited by applicant]
US 20080270224A1 · Portman et al. · 2008 [cited by applicant]
US 20080294609A1 · Liu et al. · 2008 [cited by applicant]
US 20080305778A1 · Aaltonen et al. · 2008 [cited by applicant]
US 20090006145A1 · Duggal et al. · 2009 [cited by applicant]
US 20090013255A1 · Yuschik et al. · 2009 [cited by applicant]
US 20090210491A1 · Thakkar et al. · 2009 [cited by applicant]
US 20090228281A1 · Singleton et al. · 2009 [cited by applicant]
US 20090240670A1 · Tiyyagura et al. · 2009 [cited by applicant]
US 20100114944A1 · Adler et al. · 2010 [cited by applicant]
US 20100138416A1 · Bellotti · 2010 [cited by applicant]
US 20100198772A1 · Silverman et al. · 2010 [cited by applicant]
US 20100306229A1 · Timm et al. · 2010 [cited by applicant]
US 20110010240A1 · Veach · 2011 [cited by applicant]
US 20110014925A1 · Antic et al. · 2011 [cited by applicant]
US 20110022460A1 · Bhatia et al. · 2011 [cited by applicant]
US 20110087660A1 · Yu et al. · 2011 [cited by applicant]
US 20110141925A1 · Velenko et al. · 2011 [cited by applicant]
US 20110177799A1 · Ramer et al. · 2011 [cited by applicant]
US 20110202494A1 · Shin et al. · 2011 [cited by applicant]
US 20110264644A1 · Grant et al. · 2011 [cited by applicant]
US 20110295682A1 · Liu · 2011 [cited by applicant]
US 20110295990A1 · St. Jean et al. · 2011 [cited by applicant]
US 20110307436A1 · Cai et al. · 2011 [cited by applicant]
US 20110320114A1 · Buxton et al. · 2011 [cited by applicant]
US 20120016897A1 · Tulumbas et al. · 2012 [cited by applicant]
US 20120030015A1 · Brunsman et al. · 2012 [cited by applicant]
US 20120036226A1 · Chor · 2012 [cited by applicant]
US 20120101776A1 · Brower et al. · 2012 [cited by applicant]
US 20120102020A1 · Pearson · 2012 [cited by applicant]
US 20120102087A1 · Chor · 2012 [cited by applicant]
US 20120138671A1 · Gaede et al. · 2012 [cited by applicant]
US 20120155838A1 · Gerhards et al. · 2012 [cited by applicant]
US 20120158954A1 · Heffernan et al. · 2012 [cited by applicant]
US 20120166277A1 · Gnanamani et al. · 2012 [cited by applicant]
US 20120221724A1 · Chor · 2012 [cited by applicant]
US 20120265528A1 · Gruber et al. · 2012 [cited by applicant]
US 20120297174A1 · Frank et al. · 2012 [cited by applicant]
US 20130013749A1 · Kane et al. · 2013 [cited by applicant]
US 20130073400A1 · Heath · 2013 [cited by applicant]
US 20130117022A1 · Chen et al. · 2013 [cited by applicant]
US 20130124606A1 · Carpenter et al. · 2013 [cited by applicant]
US 20130144720A1 · Hari et al. · 2013 [cited by applicant]
US 20130185074A1 · Gruber et al. · 2013 [cited by applicant]
US 20130191226A1 · Smallwood et al. · 2013 [cited by applicant]
US 20130275164A1 · Gruber et al. · 2013 [cited by applicant]
US 20130304758A1 · Gruber et al. · 2013 [cited by applicant]
US 20130304789A1 · Herlein · 2013 [cited by applicant]
US 20130325449A1 · Levien et al. · 2013 [cited by applicant]
US 20130339030A1 · Ehsani et al. · 2013 [cited by applicant]
US 20140074895A1 · Ingerman et al. · 2014 [cited by applicant]
US 20140095583A1 · Houle · 2014 [cited by applicant]
US 20140115635A1 · Mashimo · 2014 [cited by applicant]
US 20140222436A1 · Binder et al. · 2014 [cited by applicant]
US 20140229184A1 · Shires · 2014 [cited by applicant]
US 20140337028A1 · Wang et al. · 2014 [cited by applicant]
US 20140350938A1 · Schroeter · 2014 [cited by applicant]
US 20150081288A1 · Kim · 2015 [cited by applicant]
US 20150106085A1 · Lindahl · 2015 [cited by applicant]
US 20150221305A1 · Sharifi · 2015 [cited by applicant]
US 20160012686A1 · Kuscher et al. · 2016 [cited by applicant]
US 20160077710A1 · Lewis et al. · 2016 [cited by applicant]
US 20160105718A1 · Xu et al. · 2016 [cited by applicant]
US 20160118044A1 · Bondarev et al. · 2016 [cited by applicant]
US 20160180846A1 · Lee · 2016 [cited by applicant]
US 20160217790A1 · Sharifi · 2016 [cited by applicant]
US 20160223640A1 · Vilermo et al. · 2016 [cited by applicant]
US 20160246929A1 · Zenati et al. · 2016 [cited by applicant]
US 20160259497A1 · Foss et al. · 2016 [cited by applicant]
US 20170069317A1 · Kwon · 2017 [cited by applicant]
US 20170092278A1 · Evermann et al. · 2017 [cited by applicant]
US 20170110130A1 · Sharifi et al. · 2017 [cited by applicant]
US 20170110144A1 · Sharifi et al. · 2017 [cited by applicant]
US 20170132019A1 · Karashchuk et al. · 2017 [cited by applicant]
US 20170147585A1 · Kleindienst et al. · 2017 [cited by applicant]
US 20170236524A1 · Ray et al. · 2017 [cited by applicant]
US 20170289596A1 · Krasadakis et al. · 2017 [cited by applicant]
US 20170358301A1 · Raitio et al. · 2017 [cited by applicant]
US 20170358302A1 · Orr et al. · 2017 [cited by applicant]
US 20180322536A1 · Zhang et al. · 2018 [cited by applicant]
CN 102016502 · 2011 [cited by applicant]
CN 104396287 · 2015 [cited by applicant]
CN 104683294 · 2015 [cited by applicant]
CN 105849758 · 2016 [cited by applicant]
EP 1271458 · 2003 [cited by applicant]
EP 1286288 · 2003 [cited by applicant]
GB 2305747 · 1997 [cited by applicant]
JP H04287556 · 1992 [cited by applicant]
JP 11065950A · 1999 [cited by applicant]
JP 11265347 · 1999 [cited by applicant]
JP 2001236410 · 2001 [cited by applicant]
JP 2001282982A · 2001 [cited by applicant]
JP 2001297256A · 2001 [cited by applicant]
JP 2001312646 · 2001 [cited by applicant]
JP 2001312649 · 2001 [cited by applicant]
JP 2002007253 · 2002 [cited by applicant]
JP 2002016970 · 2002 [cited by applicant]
JP 2002073666 · 2002 [cited by applicant]
JP 2002099822 · 2002 [cited by applicant]
JP 2002132827 · 2002 [cited by applicant]
JP 2002140359 · 2002 [cited by applicant]
JP 2002169744 · 2002 [cited by applicant]
JP 2002170027 · 2002 [cited by applicant]
JP 2002245048 · 2002 [cited by applicant]
JP 2002288541A · 2002 [cited by applicant]
JP 2003016348 · 2003 [cited by applicant]
JP 2003122781A · 2003 [cited by applicant]
JP 2003223591 · 2003 [cited by applicant]
JP 2003263584A · 2003 [cited by applicant]
JP 2003337893 · 2003 [cited by applicant]
JP 2004032037 · 2004 [cited by applicant]
JP 2006236324A · 2006 [cited by applicant]
JP 2011192102 · 2010 [cited by applicant]
JP 2013540306A · 2013 [cited by applicant]
JP 2015001815A · 2015 [cited by applicant]
JP 2016111406A · 2016 [cited by applicant]
JP 2016519805 · 2016 [cited by applicant]
KR 20000054165A · 2000 [cited by applicant]
KR 200054165A · 2000 [cited by applicant]
KR 200269767A · 2002 [cited by applicant]
KR 200339736A · 2003 [cited by applicant]
KR 20100126796A · 2010 [cited by applicant]
KR 1020150092996A · 2015 [cited by applicant]
KR 20150097816A · 2015 [cited by applicant]
WO WO9721183A · 1997 [cited by applicant]
WO WO0042544 · 2000 [cited by applicant]
WO WO0159546A2 · 2001 [cited by applicant]
WO WO0193138 · 2001 [cited by applicant]
WO WO2012094329 · 2012 [cited by applicant]
WO WO2014004489A1 · 2014 [cited by applicant]
WO WO2015133022A1 · 2015 [cited by applicant]
A. Narayanaswami and C. Vassiliadis, “An online prototype speech-enabled information access tool using Java Speech Application Programming Interface,” Proceedings of the 33rd Southeastern Symposium on System Theory (Cat… [cited by examiner]
Machine Translated Chinese Search Report Corresponding to Application No. 202210606342.2 on Dec. 26, 2922. [cited by applicant]
U.S. Appl. No. 95/001,061, filed Jul. 7, 2008. [cited by applicant]
U.S. Appl. No. 95/001,068, filed Jul. 14, 2008. [cited by applicant]
U.S. Appl. No. 95/001,069, filed Jul. 21, 2008. [cited by applicant]
U.S. Appl. No. 95/001,073, filed Jul. 30, 2008. [cited by applicant]
“10 Tips to Improve the Performance of Google Product Listing Ads”, Foghorn Labs, retrieved Mar. 18, 2013 from URL: http://www.foghornlabs.com/2012/11/21/product-listing-ads-best-practices/ (5 pages). [cited by applicant]
“About Ad-Start”, Appendix C, Ad-Star, Apr. 12, 1997 and Feb. 1, 1997, retrieved from URL: http://web.archive.org/web/19970412171433/www.adstar.com/about.html (43 pages). [cited by applicant]
“AdForce User Guide, Version 2.6”, ADFORCE, Inc., 1998 (285 pages). [cited by applicant]
“Best Practices for Creating Adaptive User Interfaces with the Mobile Internet Toolkit”, Microsoft Corporation, Jan. 2002 (16 pages). [cited by applicant]
“Chapter 12, Managing the Product, Class Notes”, University of Delaware, http://www.udel.edu/alex/chapt12.html, accessed Mar. 13, 2015 (10 pages). [cited by applicant]
“‘Dolphin’ attacks fool Amazon, Google voice assistants”, BBC News, Sep. 7, 2017 (8 pages). [cited by applicant]
“EStara Push to Talk: The most popular click to call service in the world,” downloaded from http://www.estara.com on Jun. 29, 20014 (2 pages). [cited by applicant]
“Global Network, Inc. Enters Into Agreement in Principle With Major Advertising Agency”, Business Wire, Inc., Appendix L, Oct. 4, 1999 (2 pages). [cited by applicant]
“Google Assistant SDK”, Google Developers Newsletter, Google Developers, developers.google.com, accessed on Jul. 12, 2017 (2 pages). [cited by applicant]
“Products Feed Specification”, Google Merchant Center Help, Google, Inc., retrieved Mar. 18, 2013 from URL: http://www.support.google.com/merchants/bin/answer.py?hl=en&answer=188494#US (6 pages). [cited by applicant]
“Reviewer's Guide”, AdKnowledge, Campaign Manager: Part of the AdKnowledge System, Appendix E1, Aug. 1998 (23 pages). [cited by applicant]
“S-1/A SEC Filing”, Adforce, Inc., Edgar Online, Appendix F2, May 6, 1999 (9 pages). [cited by applicant]
“Supported File Formats”, Google Merchant Center Help, Google, Inc., retrieved Mar. 18, 2013 from URL: http://www.support.google.com/merchants/bin/answer.py?hl=en&answer=160567 (1 page). [cited by applicant]
“Walmart and Google to offer voice-enabled shopping”, BBC News, Aug. 23, 2017 (10 pages). [cited by applicant]
“Wiley Encyclopedia of Computer Science and Engineering: vol. I”, Wiley-Interscience, Nov. 2008 (2365 pages). [cited by applicant]
U.S. Appl. No. 95/001,061 Reexamination—Miscellaneous Action dated Jul. 2, 2013 (4 pages). [cited by applicant]
U.S. Appl. No. 95/001,061 Reexamination—Reexamination Terminated dated Jul. 2, 2013 (4 pages). [cited by applicant]
U.S. Appl. No. 95/001,068 Reexamination—“Certificate of Service” dated Mar. 24, 2012 (1 page). [cited by applicant]
U.S. Appl. No. 95/001,068 Reexamination—“Inter Partes Reexamination Certificate” for 6829587-C1 dated Jan. 8, 2013 (2 pages). [cited by applicant]
U.S. Appl. No. 95/001,068 Reexamination—Appeal No. 2011-013241, “Decision on Appeal” dated Jan. 27, 2012 (34 pages). [cited by applicant]
U.S. Appl. No. 95/001,068 Reexamination—Appeal No. 2011-013241, “Notice of Appeal” dated Mar. 24, 2012 (3 pages). [cited by applicant]
U.S. Appl. No. 95/001,068 Reexamination—Case 12-1379, Document 20, “Order” by the US Court of Appeals for the Federal Circuit dated Sep. 6, 2012 (2 pages). [cited by applicant]
U.S. Appl. No. 95/001,069 Reexamination—“Certificate of Service” dated Mar. 24, 2012 (1 page). [cited by applicant]
U.S. Appl. No. 95/001,069 Reexamination—“Inter Partes Reexamination Certificate” for U.S. Pat. No. 7,249,059-C1 dated Jul. 23, 2013 (2 pages). [cited by applicant]
U.S. Appl. No. 95/001,069 Reexamination—Appeal No. 2011-010893, “Decision on Appeal” dated Jan. 27, 2012 (34 pages). [cited by applicant]
U.S. Appl. No. 95/001,069 Reexamination—Appeal No. 2011-010893, “Notice of Appeal” dated Mar. 24, 2012 (3 pages). [cited by applicant]
U.S. Appl. No. 95/001,069 Reexamination—Case 12-1380, Document 54-1, Decision by the US Court of Appeals for the Federal Circuit dated Mar. 7, 2013 (8 pages). [cited by applicant]
U.S. Appl. No. 95/001,069 Reexamination—Case 12-1380, Document 54-2, “Information Sheet” dated Mar. 7, 2013 (1 page). [cited by applicant]
U.S. Appl. No. 95/001,069 Reexamination—Case 12-1380, Document 54-3, “Questions and Answers: Petitions for Panel Rehearing (Fed. Cir. R. 40) and Petitions for Hearing or Rehearing En Banc (Fed. Cir. R. 35)” dated Mar. 7… [cited by applicant]
U.S. Appl. No. 95/001,069 Reexamination—Case 12-1380, Document 54-4, “Notice of Entry of Judgment Accompanied by Opinion” dated Mar. 7, 2013 (2 pages). [cited by applicant]
U.S. Appl. No. 95/001,069 Reexamination—Case 12-1380, Document 55-3, “Mandate” dated Apr. 29, 2013 (2 pages). [cited by applicant]
U.S. Appl. No. 95/001,073 Reexamination—“Certificate of Service” dated Mar. 24, 2012 (1 page). [cited by applicant]
U.S. Appl. No. 95/001,073 Reexamination—“Inter Partes Reexamination Certificate” for U.S. Pat. No. 7240025-C1 dated Jul. 23, 2013 (2 pages). [cited by applicant]
U.S. Appl. No. 95/001,073 Reexamination—Appeal No. 2011-010719, “Decision on Appeal” dated Jan. 27, 2012 (27 pages). [cited by applicant]
U.S. Appl. No. 95/001,073 Reexamination—Appeal No. 2011-010719, “Notice of Appeal” dated Mar. 24, 2012 (3 pages). [cited by applicant]
U.S. Appl. No. 95/001,073 Reexamination—Case 12-1380, Document 54-1, Decision by the US Court of Appeals for the Federal Circuit dated Mar. 7, 2013 (8 pages). [cited by applicant]
U.S. Appl. No. 95/001,073 Reexamination—Case 12-1380, Document 55-3, “Mandate” by the US Court of Appeals for the Federal Circuit dated Apr. 29, 2013 (2 pages). [cited by applicant]
Abrams, Brad, “Help users find, interact & re-engage with your app on the Google Assistant”, Google Developers Blog, Nov. 15, 2017 (16 pages). [cited by applicant]
AdKnowledge, “Reviewer's Guide”, MarketMatch Planner, Appendix E2, May 1998 (26 pages). [cited by applicant]
Albrecht, Chris, “Alexa, How Can You Be Used in Restaurants?”, The Spoon, Dec. 10, 2017 (6 pages). [cited by applicant]
Amazon, “Echo Look | Hands-Free Camera and Style Assistant”, retrieved Aug. 22, 2017 from URL: https://www.amazon.com/gp/product/B0186JAEWK?ref%5F=cm%5Fsw%5Fr%5Ffa%5Fdp%5Ft2%5FC5oazbJTKCB18&pldnSite=1 (7 pages). [cited by applicant]
AU 2005259861, “Statement of Grounds and Particulars in Support of Opposition” dated Jan. 22, 2010 (13 pages). [cited by applicant]
Barr, Jeff, “AWS DeepLens—Get Hands-On Experience with Deep Learning With Our New Video Camera”, AWS News Blog, Nov. 29, 2017 (11 pages). [cited by applicant]
Baseview Products, Inc., “AdManagerPro Administrator's Manual v. 2.0”, Appendix H, Dec. 1998 (226 pages). [cited by applicant]
Baseview Products, Inc., “ClassManagerPro Administration Manual v. 1.0.5”, Feb. 1, 1997 (157 pages). [cited by applicant]
Broussard, Mitchel, “Chatbot-Like Siri Patent Includes Intelligent Image, Video, and Audio Recognition Within Messages”, Mac Rumors, May 11, 2017, retrieved Jul. 12, 2017 from URL: https://www.macrumors.com/2017/05/11/s… [cited by applicant]
Buckland et al., “Amazon's Alexa Takes Open-Source Route to Beat Google Into Cars”, Bloomberg, Feb. 27, 2018 (6 pages). [cited by applicant]
Chen, Lulu Yilun, “Alibaba Challenges Google, Amazon With New Echo-Like Device”, Bloomberg, Jul. 5, 2017, retrieved Jul. 12, 2017 from URL: https://www.bloomberg.com/news/articles/2017-07-05/alibaba-challenges-google-am… [cited by applicant]
Close, Kerry, “Amazon Echo Will Give You These Deals if You Order Through Alexa This Weekend,” Time.com, Nov. 18, 2016 (2 pages). [cited by applicant]
Clover, Juli, “Amazon and Google Want to Turn Their Smart Home Speakers Into Telephone Replacements”, MacRumors, Feb. 15, 2017 (5 pages). [cited by applicant]
Coberly, Cohen, “Apple patent filing reveals potential whispering Siri functionality”, Techspot, Dec. 14, 2017 (4 pages). [cited by applicant]
Collins, et al., “Can Twitter Save Itself?”, CNET, Apr. 26, 2017, retrieved Aug. 22, 2017 from URL: https://www.cnet.com/news/twitter-q1-2017-earnings/ (3 pages). [cited by applicant]
Cook, John, “A Siri for advertising: These mobile ads talk back to you,” GeekWire, Apr. 1, 2013, geekwire.com (7 pages). [cited by applicant]
Corrected Notice of Allowance for U.S. Appl. No. 15/395,703 dated May 17, 2018 (5 pages). [cited by applicant]
Crist, Ry, “Logitech Harmony's Alexa skill just got a whole lot better”, CNET, Jul. 13, 2017 (2 pages). [cited by applicant]
Decision of Rejection for JP Appln. Ser. No. 2007-519374 dated Jun. 15, 2010 (6 pages). [cited by applicant]
Decision of Rejection for JP Appln. Ser. No. 2007-519407 dated Feb. 23, 2010 (5 pages). [cited by applicant]
Decision of Rejection for JP AppIn. Ser. No. 2010-232591 dated Jan. 27, 2014 (4 pages). [cited by applicant]
Decision to Dismiss the Amendment for KR Appln. Ser. No. 10-2007-7002383 dated Jul. 9, 2010 (3 pages). [cited by applicant]
Dedrick, Rick, “A Consumption Model for Targeted Electronic Advertising,” Intel Architecture Labs, IEEE MultiMedia, dated 1995 (9 pages). [cited by applicant]
Dedrick, Rick, “Interactive Electronic Advertising”, Intel Corp. Media Delivery Lab, IEEE, dated 1994 (12 pages). [cited by applicant]
Dickinger et. al, “An Investigation of Conceptual Model of SMS Marketing”, Proceedings of 37th Hawaii International Conference on System Sciences, Jan. 2004, pp. 1-10 (10 pages). [cited by applicant]
Estes, Adam Clark, “Amazon's Newest Gadget is a Tablet That's Also an Echo”, Gizmodo, Sep. 19, 2017 (3 pages). [cited by applicant]
Examination Report for AU Appln. Ser. No. 2017384996 dated Nov. 25, 2019 (2 pages). [cited by applicant]
Examination Report for AU Appln. Ser. No. 2020217377 dated Jul. 7, 2021 (2 pages). [cited by applicant]
Examination Report for EP Appln. Ser. No. 05763971.8 dated Mar. 29, 2011 (4 pages). [cited by applicant]
Examination Report for EP Appln. Ser. No. 05763971.8-2221 dated Mar. 29, 2011 (4 pages). [cited by applicant]
Examination Report for EP Appln. Ser. No. 05768157.9 dated Aug. 21, 2014 (6 pages). [cited by applicant]
Examination Report for EP Appln. Ser. No. 05768157.9 dated Jan. 27, 2014 (5 pages). [cited by applicant]
Examination Report for EP Appln. Ser. No. 05768157.9 dated Jul. 23, 2015 (7 pages). [cited by applicant]
Examination Report for EP Appln. Ser. No. 05768157.9-1958 dated Feb. 15, 2013 (5 pages). [cited by applicant]
Examination Report for GB Appln. Ser. No. 1804023.8 dated Jun. 9, 2021 (4 pages). [cited by applicant]
Examination Report for IN Appln. Ser. No. 201747040182 dated Aug. 12, 2020 (7 pages). [cited by applicant]
Examiner's Re-Examination Report for AU Appln. Ser. No. 2005259861 dated Dec. 1, 2011 (2 pages). [cited by applicant]
Feng et al., “Active Profiling of Physical Devices at Internet Scale”, IEEE, 2016 (9 pages). [cited by applicant]
Final Office Action for U.S. Appl. No. 10/880,868 dated Apr. 13, 2010 (8 pages). [cited by applicant]
Final Office Action for U.S. Appl. No. 10/880,868 dated Jul. 17, 2009 (7 pages). [cited by applicant]
Final Office Action for U.S. Appl. No. 10/880,868 dated Mar. 24, 2015 (16 pages). [cited by applicant]
Final Office Action for U.S. Appl. No. 11/026,507 dated Apr. 6, 2015 (9 pages). [cited by applicant]
Final Office Action for U.S. Appl. No. 11/026,507 dated Jan. 12, 2010 (19 pages). [cited by applicant]
Final Office Action for U.S. Appl. No. 11/026,507 dated Mar. 24, 2011 (19 pages). [cited by applicant]
Final Office Action for U.S. Appl. No. 13/441,298 dated Dec. 30, 2013 (12 pages). [cited by applicant]
Final Office Action for U.S. Appl. No. 13/441,298 dated Nov. 20, 2015 (16 pages). [cited by applicant]
Final Office Action for U.S. Appl. No. 13/478,998 dated Feb. 27, 2017 (21 pages). [cited by applicant]
Final Office Action for U.S. Appl. No. 13/478,998 dated Jan. 20, 2016 (22 pages). [cited by applicant]
Final Office Action for U.S. Appl. No. 13/840,380 dated Aug. 3, 2018 (12 pages). [cited by applicant]
Final Office Action for U.S. Appl. No. 13/840,380 dated Feb. 27, 2017 (7 pages). [cited by applicant]
Final Office Action for U.S. Appl. No. 13/840,380 dated Oct. 22, 2015 (32 pages). [cited by applicant]
Final Office Action for U.S. Appl. No. 14/172,353 dated Apr. 4, 2018 (22 pages). [cited by applicant]
Final Office Action for U.S. Appl. No. 14/172,353 dated Aug. 10, 2017 (15 pages). [cited by applicant]
Final Office Action for U.S. Appl. No. 14/172,353 dated Dec. 14, 2018 (21 pages). [cited by applicant]
Final Office Action for U.S. Appl. No. 14/172,353 dated Jan. 21, 2016 (28 pages). [cited by applicant]
Final Office Action for U.S. Appl. No. 14/230,508 dated Jun. 3, 2015 (20 pages). [cited by applicant]
Final Office Action for U.S. Appl. No. 15/189,826 dated Jun. 3, 2019 (31 pages). [cited by applicant]
Final Office Action for U.S. Appl. No. 15/491,734 dated May 28, 2019 (24 pages). [cited by applicant]
Final Office Action for U.S. Appl. No. 15/584,970 dated Jun. 25, 2021 (17 pages). [cited by applicant]
Final Office Action for U.S. Appl. No. 15/584,970 dated May 31, 2019 (23 pages). [cited by applicant]
Final Office Action for U.S. Appl. No. 15/587,132 dated Jul. 9, 2020 (27 pages). [cited by applicant]
Final Office Action for U.S. Appl. No. 15/587,132 dated Jun. 25, 2021 (18 pages). [cited by applicant]
Final Office Action for U.S. Appl. No. 15/603,701 dated Apr. 2, 2019 (14 pages). [cited by applicant]
Final Office Action for U.S. Appl. No. 15/638,291 dated Jan. 30, 2020 (24 pages). [cited by applicant]
Final Office Action for U.S. Appl. No. 15/638,295 dated Jan. 30, 2020 (25 pages). [cited by applicant]
Final Office Action for U.S. Appl. No. 15/638,295 dated Jul. 21, 2021 (26 pages). [cited by applicant]
Final Office Action for U.S. Appl. No. 16/266,927 dated Dec. 1, 2020 (8 pages). [cited by applicant]
First Office Action for CN Appln. Ser. No. 201780001651.4 dated Mar. 31, 2021 (27 pages). [cited by applicant]
First Office Action for JP Appln. Ser. No. 2007-519374 dated May 29, 2012 (47 pages). [cited by applicant]
First Office Action for JP Appln. Ser. No. 2010-142707 dated Jul. 23, 2013 (4 pages). [cited by applicant]
First Office Action for JP Appln. Ser. No. 2010-142707 dated Nov. 13, 2012 (4 pages). [cited by applicant]
First Office Action for JP Appln. Ser. No. 2010-232591 dated Feb. 19, 2013 (7 pages). [cited by applicant]
First Office Action for JP Appln. Ser. No. 2010-232591 dated Jun. 9, 2015 (7 pages). [cited by applicant]
First Office Action for JP Appln. Ser. No. 2014-010608 dated Nov. 18, 2014 (4 pages). [cited by applicant]
Fisher et al., “The Role of Text Analytics and Information Retrieval in the Accounting Domain,” Journal of Emerging Technologies in Accounting, vol. 7, Dec. 2010 (25 pages). [cited by applicant]
Flinn et al., “Balancing Performance, Energy, and Quality in Pervasive Computing”, Proceedings of the 22nd International Conference on Distributed Computing Systems (ICDCS'02), IEEE, 2002, pp. 1-10, retrieved Feb. 28, 2… [cited by applicant]
Forrest, Conner, “Essential Home wants to be ‘bridge’ between Amazon Alexa, Apple's Siri, and Google Assistant,” TechRepublic, May 31, 2017 (9 pages). [cited by applicant]
Foxx, Chris, “Apple reveals HomePod smart speaker”, BBC, Jun. 5, 2017 (9 pages). [cited by applicant]
Gebhart, Andrew, “Google Assistant is spreading, but it needs its own ‘Echo Dot’”, CNET, May 20, 2017 (6 pages). [cited by applicant]