IP Library Granted Patent US 11,397,558
Granted Patent B2
US 11,397,558 · App. 15/935,992 · Granted Jul 26, 2022

Optimizing display engagement in action automation

Inventor: Rajat Mukherjee (San Jose, CA)
Assignee: PELOTON INTERACTIVE, INC.
G06F3/167G06F3/0484G06F9/45512G06F9/541G10L15/22G10L15/30H04L12/2816H04L67/141H04W76/14G06F3/048G10L15/26G10L2015/223H04L67/306
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,397,558
App. No.
15/935,992
Granted
Jul 26, 2022
Kind
B2
Abstract

Embodiments of the present invention provide systems, methods, and computer storage media directed to optimizing engagement with a display during digital assistant-performed operations in response to a received command. The digital assistant generates an overlay having user interface elements that present information determined to be relevant to a user based on the received command and contextual data. The overlay is presented over the underlying operations performed on corresponding applications to mask the visible steps of the operations being performed. In this way, the digital assistant optimizes display resources that are typically rendered useless during the processing of digital assistant-performed operations.

Claims (45)

1. A non-transitory computer storage medium storing computer-useable instructions that, when used by at least one computing device, cause the at least one computing device to perform operations to optimize display engagement in action automation, comprising:

obtaining a library that includes a plurality of commands, each command in the plurality of commands being associated with one of a stored plurality of action datasets, each action dataset in the stored plurality of action datasets defining a corresponding plurality of automated operations to perform utilizing a corresponding application installed on a computing device;

selecting a first action dataset of the stored plurality of action datasets based at least in part on a received command determined to correspond to a first command of the plurality of commands, wherein the selected first action dataset defines a first plurality of automated operations for a first application installed on the computing device;

generating an overlay interface that includes at least a first user interface element, the first user interface element presenting content that is determined relevant to at least a first portion of the received command;

initiating the selected first action dataset, the initiating causing the defined first plurality of automated operations to be performed utilizing the first application; and

responsive to initiating the selected first action dataset, providing for display the generated overlay interface to mask visual output data generated for display via the first application, the generated overlay interface being displayed for at least a latency period associated with the performance of the defined first plurality of automated operations.

2. The non-transitory computer storage medium of claim 1 , the operations further comprising:

terminating the performance of the defined first plurality of automated operations responsive to an input received via at least a portion of the displayed overlay interface during the latency period.

3. The non-transitory computer storage medium of claim 1 , wherein the content of the first user interface element is determined relevant to at least the first portion of the received command based on a determined contextual relationship there between.

4. The non-transitory computer storage medium of claim 3 , wherein the content of the first user interface element includes a piece of third-party content.

5. The non-transitory computer storage medium of claim 3 , wherein the content of the first user interface element includes a command suggestion, the received command is a first voice command, and the command suggestion is a text representation of a second voice command.

6. The non-transitory computer storage medium of claim 5 , wherein the command suggestion corresponds to one of:

a second action dataset of the stored plurality of action datasets, determined to define one of a second plurality of automated operations for the first application, or

a third action dataset of the stored plurality of action datasets, associated with a second command of the plurality of commands and determined to correspond to at least one of the first portion of the received command or obtained contextual data.

7. The non-transitory computer storage medium of claim 6 , wherein the obtained contextual data includes at least one of a second portion of the received command, location data determined by the at least one computing device, a profile associated with the at least one computing device, or historical data associated with the profile.

8. The non-transitory computer storage medium of claim 1 , further comprising:

removing, after the latency period, the displayed overlay interface to reveal the first application.

9. The non-transitory computer storage medium of claim 8 , further comprising:

based on the displayed overlay interface being removed, providing for display over or adjacent to at least a portion of the first application, a second user interface element having content associated with the first user interface element.

10. The non-transitory computer storage medium of claim 9 , further comprising:

receiving an interaction via the displayed second user interface element, wherein the displayed second user interface corresponds to one of a second action dataset of the stored plurality of action datasets, an embedded hyperlink, or an embedded deep link; and

accessing a second plurality of automated operations for a second application defined in the second action dataset, the embedded hyperlink, or the embedded deep link, in response to the received interaction.

11. A computer-implemented method to optimize display engagement in action automation, comprising:

storing, by a computing device, a library that includes a plurality of commands to a memory of the computing device, each command in the plurality of commands being associated with one of a stored plurality of action datasets, each action dataset of the stored plurality of action datasets defining a corresponding plurality of executable operations to automatically execute, when the action dataset is initiated, utilizing a corresponding application installed on the computing device;

selecting, by the computing device, a first action dataset of the stored plurality of action datasets based at least in part on a received command determined to correspond to a first command of the plurality of commands, wherein the selected first action dataset defines a first plurality of executable operations for a first application installed on the computing device;

generating, by the computing device, an overlay interface that includes at least a first user interface element, the first user interface element presenting content determined to be contextually relevant to at least a first portion of the received command;

initiating, by the computing device, the selected first action dataset, the initiating causing the defined first plurality of executable operations to be automatically executed utilizing the first application; and

responsive to initiating the selected first action dataset, providing for display, by the computing device, the generated overlay interface to mask visual output data generated for display via the first application, the generated overlay interface being displayed for at least a latency period that corresponds to a duration associated with the defined first plurality of executable operations being automatically executed.

12. The computer-implemented method of claim 11 , wherein a plurality of executable operations includes at least one of a touch event, a deep link, an inclusion of at least one parameter, or an entry of input data.

13. The computer-implemented method of claim 12 , wherein the at least one parameter and the input data is determined based on contextual data obtained by the computing device.

14. The computer-implemented method of claim 13 , wherein the obtained contextual data includes at least one of the first portion of the received command, a second portion of the received command, location data determined by the computing device, a profile associated with the computing device, or historical data associated with the profile.

15. A computerized system to optimize display engagement in action automation, comprising:

at least one processor; and

at least one computer storage media storing computer-usable instructions that, when used by the at least one processor, cause the at least one processor to:

obtain a library that includes a plurality of commands, each command in the plurality of commands being associated with one of a stored plurality of action datasets, each action dataset of the stored plurality of action datasets defining a corresponding plurality of automated operations to perform utilizing a corresponding application installed on a computing device;

select a first action dataset of the stored plurality of action datasets based at least in part on a received command determined corresponding to a first command of the plurality of commands, wherein the selected first action dataset defines a first plurality of automated operations for a first application installed on the computing device;

generate an overlay interface that includes at least a first user interface element, the first user interface element presenting content determined contextually relevant to at least one parameter included in the received command;

initiate the selected first action dataset, causing the defined first plurality of automated operations to be performed utilizing the first application; and

responsive to the selected first action dataset being initiated, provide for display the generated overlay interface to mask visual output data generated for display via the first application, the generated overlay being displayed for at least a latency period that corresponds to the defined first plurality of automated operations being performed.

16. The computerized system of claim 15 , wherein the instructions further cause the at least one processor to:

remove, after the latency period, the displayed overlay interface to reveal resulting visual output data generated for display via the first application based on a completed performance of the defined first plurality of automated operations.

17. The computerized system of claim 15 , wherein a plurality of automated operations includes at least one of an emulated touch event, an invocation of a deep link, an automated inclusion of at least one parameter, or an automated entry of input data.

18. The computerized system of claim 15 , wherein the command is received based on generated speech-to-text data.

19. The non-transitory computer storage medium of claim 1 , wherein the overlay interface is generated based on the selected first action dataset.

20. The non-transitory computer storage medium of claim 1 , wherein the generated overlay interface is provided for display to mask an entirety of visual output data generated for display via the first application.

Assignments (3)
PATENT SECURITY AGREEMENT Recorded Jun 1, 2022
From: PELOTON INTERACTIVE, INC.
To: JPMORGAN CHASE BANK, N.A.
Reel/Frame 060247/0453 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 3, 2021
From: AIQUDO, INC.
To: PELOTON INTERACTIVE INC.
Reel/Frame 058284/0392 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 23, 2018
From: MUKHERJEE, RAJAT
To: AIQUDO, INC.
Reel/Frame 045613/0684 →
Continuity (6)
Provisional Application 62576766 · Oct 25, 2017
Provisional Application 62539866 · Aug 1, 2017
Provisional Application 62576804 · Oct 25, 2017
Provisional Application 62509534 · May 22, 2017
Provisional Application 62508181 · May 18, 2017
Related Publication 20180336010A1 · Nov 22, 2018
Cited By (1)
US 12,470,620