FACILITATED LIVE ANALYSIS OF SCREEN CONTENT
A computer system is provided. The computer system includes a memory and at least one processor coupled to the memory. The at least one processor is configured to implement a rule processor to receive a UI element recognition rule comprising one or more UI element specifications and a response action from a workspace server, and generate a task identifier for the received UI element recognition rule; implement a computer vision (CV) processor to receive the task identifier from the rule processor, and recognize, based on the one or more UI element specifications and the task identifier, a UI element presented at the client computer system; and implement an action handler configured to execute the response action based on the task identifier and in response to the recognized UI element.
1 . A client computer system configured to recognize and respond to user interface (UI) elements, the client computer system comprising:
a memory; and
at least one processor coupled to the memory and configured to:
receive a recognition rule comprising one or more UI element specifications and a response action from a server;
generate a task identifier corresponding to the received recognition rule;
associate the task identifier with the recognition rule;
recognize, based on the one or more UI element specifications of the recognition rule and the task identifier, a UI element rendered at the client computer system; and
execute, in response to recognition of the UI element, the response action of the recognition rule based on the task identifier.
2 . The client computer system of claim 1 , wherein to recognize the UI element based on the recognition rule further comprises to execute a CV process.
3 . The client computer system of claim 2 , wherein to execute the CV process further comprises one or more of:
to identify a rectangle;
to filter out a user interface (UI) element;
to identify a detail of a dialog or menu item group; or
to classify a rectangle using a guided heuristic.
4 . The client computer system of claim 1 , wherein to recognize the UI element based on the recognition rule further comprises to execute a machine learning (ML) process.
5 . The client computer system of claim 1 , wherein the at least one processor is configured to:
generate a numerical hash value by executing a hash function on one or more characteristics of the rendered UI element;
send the numerical hash value to the server; and
further comprising the server, wherein the server is configured to track a number of instances of the numerical hash value.
6 . The client computer system of claim 1 , wherein the one or more UI element specifications describe one or more of:
a size or shape of a dialog or window;
text of a dialog or window;
an icon or image of a dialog or window;
a progress bar or scroll bar of a dialog or window;
a button or other control of a dialog or window;
a menu item; or
a number of instances or affected users associated with the UI element.
7 . The client computer system of claim 1 , wherein the recognition rule comprises a structured recognition rule tag including:
one or more structured UI element specification tags encoding the one or more UI element specifications; and
a structured response action tag encoding the response action.
8 . The client computer system of claim 1 , wherein the UI element rendered at the client computer system comprises one or more of:
an error notification;
a password entry field;
a delay notifier;
a pop-up menu item;
personal information; or
a personal message.
9 . A method of recognizing and responding to user interface (UI) elements, the method comprising:
receiving a recognition rule comprising one or more UI element specifications and a response action from a server;
recognizing, based on the one or more UI element specifications, a UI element rendered at the client computer; and
executing, in response to the recognition of the UI element, the response action of the received recognition rule.
10 . The method of claim 9 , further comprising executing a CV process to recognize the UI element based on the recognition rule, wherein the CV process comprises one or more of:
identifying a rectangle;
filtering out a user interface (UI) element;
identifying a detail of a dialog or menu item group; or
classifying a rectangle using a guided heuristic.
11 . The method of claim 9 , further comprising executing a machine learning (ML) process to recognize the UI element based on the recognition rule.
12 . The method of claim 9 , further comprising:
generating a numerical hash value by executing a hash function on one or more characteristics of the rendered UI element;
sending the numerical hash value to the server; and
tracking, by the server, a number of instances of the numerical hash value.
13 . The method of claim 9 , wherein the one or more UI element specifications describe one or more of:
a size or shape of a dialog or window;
text of a dialog or window;
an icon or image of a dialog or window;
a progress bar or scroll bar of a dialog or window;
a button or other control of a dialog or window;
a menu item; or
a number of instances or affected users associated with the UI element.
14 . The method of claim 9 , wherein the recognition rule comprises a structured recognition rule tag including:
one or more structured UI element specification tags encoding the one or more UI element specifications; and
a structured response action tag encoding the response action.
15 . A non-transitory computer readable medium storing executable sequences of instructions to recognize and respond to user interface (UI) elements, the sequences of instructions comprising instructions to:
receive a recognition rule comprising one or more UI element specifications and a response action from a server;
generate a task identifier corresponding to the received recognition rule;
associate the task identifier with the recognition rule;
recognize, based on the one or more UI element specifications of the recognition rule and the task identifier, a UI element rendered at the client computer system; and
execute, in response to the recognition of the UI element, the response action of the recognition rule based on the task identifier.
16 . The non-transitory computer readable medium of claim 15 , wherein to recognize the UI element comprises to execute a CV process for one or more of:
to identify a rectangle;
to filter out a user interface (UI) element;
to identify a detail of a dialog or menu item group; or
to classify a rectangle using a guided heuristic.
17 . The non-transitory computer readable medium of claim 15 , wherein the instructions further comprise instructions to:
generate a numerical hash value by executing a hash function on one or more characteristics of the rendered UI element; and
send the numerical hash value to the server, the server being configured to track a number of instances of the numerical hash value.
18 . The non-transitory computer readable medium of claim 15 , wherein the one or more UI element specifications describe one or more of:
a size or shape of a dialog or window;
text of a dialog or window;
an icon or image of a dialog or window;
a progress bar or scroll bar of a dialog or window;
a button or other control of a dialog or window;
a menu item; or
a number of instances or affected users associated with the UI element.
19 . The non-transitory computer readable medium of claim 15 , wherein to recognize the UI element comprises to recognize one or more of:
an error notification;
a password entry field;
a delay notifier;
a pop-up menu item;
personal information; or
a personal message.
20 . The non-transitory computer readable medium of claim 15 , wherein the recognition rule comprises a structured recognition rule tag including:
one or more structured UI element specification tags encoding the one or more UI element specifications; and
a structured response action tag encoding the response action.