Granted Patent B1

US 12,498,836 · App. 19/271,494 · Granted Dec 16, 2025

Inventors: Manoj Kumar Jhawar (Encinitas, CA); Aman Manoj Jhawar (San Marcos, CA)

Assignee: AUGMENTALIS INC.

G06F3/0481G06F3/011G06F3/013G06F3/017G06F3/167G06F40/284

Abstract

Claims (66)

1 . One or more non-transitory computer readable mediums storing instructions that, when executed by one or more processors of a system, cause the system to perform operations comprising:

obtaining data associated with an input by a user at an application interface indicating an intent to control the application interface;

tokenizing, using a tokenizer, the data associated with the input by the user to create tokenized user input data;

obtaining tokenized user interface (UI) element data corresponding to a tokenized record of actionable UI elements associated with the application interface;

generating, using a trained large language model (LLM), based at least on the tokenized UI element data and the tokenized user input data, one or more events to inject into the application interface to control one or more of the actionable UI elements;

injecting the one or more events into the application interface to control the one or more actionable UI elements in accordance with the input by the user; and

providing feedback to the user in accordance with injecting the one or more events into the application interface,

wherein obtaining the tokenized UI element data corresponding to the tokenized record of actionable UI elements associated with the application interface comprises:

scraping the application interface to create a record of the actionable UI elements associated with the application interface; and

tokenizing, using the tokenizer, the record of the actionable UI elements associated with the application interface to create the tokenized UI element data.

2 . The one or more non-transitory computer readable mediums of claim 1 , wherein scraping the application interface to create the record of the actionable UI elements associated with the application interface comprises:

identifying a first actionable UI element that does not have associated metadata in the application interface;

identifying an action associated with the first actionable UI element;

assigning a unique identifier to the first actionable UI element; and

storing, in a datastore, the unique identifier and the action associated with the first actionable UI element.

3 . The one or more non-transitory computer readable mediums of claim 2 , wherein the first actionable UI element is a UI element that can be invoked via an accessibility service of an operating system on which the application interface runs.

4 . The one or more non-transitory computer readable mediums of claim 1 , wherein:

the operations further comprise:

receiving, using a microphone of the system, a vocal command from the user to control the application interface; and

converting the vocal command to vocal command text; and

obtaining the data associated with input by the user comprises obtaining the vocal command text.

5 . The one or more non-transitory computer readable mediums of claim 4 , wherein:

the data associated with the input by the user is fused input data; and

obtaining the data associated with input by the user further comprises:

obtaining user motion data in response to an inertial measurement unit (IMU) of the system detecting head motion of the user, or obtaining user gaze data in response to an eye tracking unit of the system detecting eye motion of the user; and

fusing the vocal command text with the user motion data or the user gaze data to obtain the fused input data.

6 . The one or more non-transitory computer readable mediums of claim 1 , wherein

the one or more events to inject into the application interface to control one or more of the actionable UI elements comprise a plurality of events to inject into the application interface to control a plurality of the actionable UI elements; and

injecting the one or more events into the application interface comprises: injecting the plurality of events into the application interface to control the plurality of actionable UI elements in accordance with the input by the user.

7 . The one or more non-transitory computer readable mediums of claim 6 , wherein:

the plurality of events comprise a first event that is initiated by an operating system on which the application interface runs, and is not interpretable by the application interface; and

injecting the plurality of events comprises injecting, via the operating system, the first event into the application interface.

8 . The one or more non-transitory computer readable mediums of claim 7 , wherein:

the plurality of events further comprise a second event that is interpretable by the application interface; and

injecting the plurality of events further comprises injecting, via the application interface, the second event into the application interface.

9 . The one or more non-transitory computer readable mediums of claim 5 , wherein the system is a head mounted display system.

10 . The one or more non-transitory computer readable mediums of claim 1 , wherein the tokenizer and the trained LLM are configured to run locally on the system.

11 . The one or more non-transitory computer readable mediums of claim 1 , wherein providing feedback to the user in accordance with injecting the one or more events into the application interface comprises: presenting, on a display of the system, an updated rendering of the application interface in response to controlling the one or more actionable UI elements.

12 . The one or more non-transitory computer readable mediums of claim 11 , wherein providing the feedback to the user in accordance with injecting the one or more events into the application interface further comprises: playing sound on a speaker of the system, or delivering haptic feedback to the user using a haptic feedback device of the system.

13 . A method, comprising:

obtaining data associated with an input by a user at an application interface indicating an intent to control the application interface;

tokenizing, using a tokenizer, the data associated with the input by the user to create tokenized user input data;

obtaining tokenized user interface (UI) element data corresponding to a tokenized record of actionable UI elements associated with the application interface;

generating, using a trained large language model (LLM) model, based at least on the tokenized UI element data and the tokenized user input data, one or more events to inject into the application interface to control one or more of the actionable UI elements;

injecting the one or more events into the application interface to control the one or more actionable UI elements in accordance with the input by the user; and

providing feedback to the user in accordance with injecting the one or more events into the application interface,

wherein obtaining the tokenized UI element data corresponding to the tokenized record of actionable UI elements associated with the application interface comprises:

scraping the application interface to create a record of the actionable UI elements associated with the application interface; and

tokenizing, using the tokenizer, the record of the actionable UI elements associated with the application interface to create the tokenized UI element data.

14 . One or more non-transitory computer readable mediums storing instructions that, when executed by one or more processors of a system, cause the system to perform operations comprising:

obtaining audio data corresponding to a vocal command by a user indicating an intent to perform a task with an object present in a field of view of a camera of the system;

capturing, using the camera, one or more images of the object;

recognizing, based on the one or more images, using a trained image classification model, the object;

tokenizing, using a tokenizer, the audio data to obtain tokenized audio data;

obtaining tokenized object document data corresponding to a tokenized record of object document data obtained from one or more documents describing the object;

generating, using a trained large language model (LLM), based at least on the tokenized object document data and the tokenized audio data, a workflow to perform to complete the task, the workflow comprising a series of steps; and

displaying, via an application interface, a first graphical representation of the workflow and a second graphical representation of the object.

15 . The one or more non-transitory computer readable mediums of claim 14 , wherein the operations further comprise: instantiating one or more actionable UI elements for manipulating the first graphical representation and the second graphical representation in the application interface.

16 . The one or more non-transitory computer readable mediums of claim 15 , wherein the operations further comprise:

obtaining multimodal user input data associated with multimodal input by the user at the application interface indicating an intent to control the one or more actionable UI elements; and

processing the multimodal user input data to provide feedback to the user, the feedback comprising: presenting, on a display, an updated rendering of the application interface.

17 . The one or more non-transitory computer readable mediums of claim 16 , wherein the obtaining the multimodal user input data comprises processing a natural language voice command by the user.

18 . The one or more non-transitory computer readable mediums of claim 15 , wherein obtaining the tokenized object document data comprises:

extracting the object document data from one or more documents describing the object; and

tokenizing the object document data that was extracted to obtain the tokenized object document data.

19 . The one or more non-transitory computer readable mediums of claim 18 , wherein the tokenized object document data comprises spatial tokens relating one or more equipment components to one or more respective physical locations.

Assignments (1)

Continuity (1)

References Cited (51)