Systems and methods for user interface detection
Embodiments described include systems and methods for user interface (UI) anomaly detection. One or more processors of a client device can execute an application undergoing UI anomaly detection. The application can be injected with a detection engine. The detection engine can determine, while executing as a thread of the application on the one or more processors of the client device, that a dimension of a text-designated region of a first user interface element of the application is less than that of corresponding text for rendering on the user interface element. The detection engine can provide, to a server responsive to the determination, an indication of a first UI anomaly. The indication can include information about a position and size of the first user interface element.
1. A method for user interface (UI) anomaly detection, the method comprising:
executing, by one or more processors of a client device, an application undergoing UI anomaly detection, the application injected with a detection engine;
detecting, by the detection engine, while executing as a thread of the application on the one or more processors of the client device, an overlap between a first user interface element and a second user interface element of the application;
determining, by the detection engine, that the detected overlap is less than an area of any of the first user interface element or the second user interface element; and
sending, by the detection engine to a server responsive to determining that the detected overlap is less than an area of any of the first user interface element or the second user interface element, an image of the first user interface element and of the second user interface element.
2. The method of claim 1 , further comprising:
receiving, by the detection engine, a trigger from a test utility;
determining, by the detection engine, while executing as the thread of the application on the one or more processors of the client device, that a dimension of a text-designated region of a third user interface element of the application is less than a dimension of a corresponding text for rendering on the third user interface element;
determining, by the detection engine responsive to receiving the trigger, whether the dimension of the text-designated region of the third user interface element is less than that of the corresponding text; and
providing, by the detection engine to a server responsive to the determination, an indication of a first UI anomaly, the indication comprising position and size values of the third user interface element.
3. The method of claim 1 , wherein the application comprises an embedded browser providing user access to a network application, the first user interface element being a user interface element of the embedded browser or the network application.
4. The method of claim 1 , further comprising:
providing, by the detection engine to the server responsive to the detection of the overlap, information comprising at least one of:
information about a position and size of the first user interface element,
information about a position and size of the second user interface element,
a screenshot image A of the first user interface element, the second user interface element and the overlap,
an image a 1 of the first user interface element,
an image a 2 of the second user interface element,
a screenshot image b 1 with the first user interface element set as transparent, or
a screenshot image b 2 with the second user interface element set as transparent.
5. The method of claim 4 , further comprising:
causing the server to compare, at least over a region corresponding to the position and the size of the first user interface element, the image a 1 superimposed onto the screenshot image b 1 , with the screenshot image A, and to determine that the detected overlap corresponds to a first UI anomaly when the comparison indicates a mismatch.
6. The method of claim 1 , further comprising:
determining, by the detection engine, whether the detected overlap corresponds to a transparent portion of at least one of the first user interface element or the second user interface element to determine whether the detected overlap is a UI anomaly.
7. The method of claim 2 , comprising providing, by the detection engine to the server, the indication of the first UI anomaly via a hypertext transfer protocol (HTTP) message.
8. A system for user interface (UI) anomaly detection, the system comprising:
one or more processors of a client device, configured to execute an application undergoing UI anomaly detection; and
a detection engine, injected into the application, the detection engine configured to, while executing as a thread of the application on the one or more processors of the client device:
detect an overlap between a first user interface element and a second user interface element of the application;
determine that the detected overlap is less than an area of any of the first user interface element or the second user interface element; and
send, to a server responsive to the determination that the detected overlap is less than an area of any of the first user interface element or the second user interface element, an image of the first user interface element and of the second user interface element.
9. The system of claim 8 , wherein the detection engine is further configured to:
receive a trigger from a test utility;
determine that a dimension of a text-designated region of a third user interface element of the application is less than a dimension of a corresponding text for rendering on the third user interface element;
determine, responsive to receiving the trigger, whether the dimension of the text-designated region of the first user interface element is less than that of the corresponding text; and
provide, to the server responsive to the determination, an indication of a first UI anomaly, the indication comprising position and size values of the third user interface element.
10. The system of claim 8 , wherein the application comprises an embedded browser providing user access to a network application, the first user interface element being a user interface element of the embedded browser or the network application.
11. The system of claim 8 , wherein the detection engine is further configured to provide to the server responsive to the detected overlap, information comprising at least one of:
information about a position and size of the first user interface element,
information about a position and size of the second user interface element,
a screenshot image A of the first user interface element, the second user interface element and the overlap,
an image a 1 of the first user interface element,
an image a 2 of the second user interface element,
a screenshot image b 1 with the first user interface element set as transparent, or
a screenshot image b 2 with the second user interface element set as transparent.
12. The system of claim 11 , wherein the detection engine is further configured to cause the server to compare, at least over a region corresponding to the position and the size of the first user interface element, the image a 1 superimposed onto the screenshot image b 1 , with the screenshot image A, and to determine that the detected overlap corresponds to a first UI anomaly when the comparison indicates a mismatch.
13. The system of claim 8 , wherein the detection engine is further configured to:
determine whether the detected overlap corresponds to a transparent portion of at least one of the first user interface element or the second user interface element
to determine whether the detected overlap is a UT anomaly.
14. The system of claim 9 , wherein the detection engine is configured to provide to the server the indication of the first UI anomaly via a hypertext transfer protocol (HTTP) message.