System and method for detecting a change in context of an application
A digital adoption platform includes a page change identification technique. The page change identification technique is platform and application agnostic. Changes in screenshots of a UI of an application are compared. Differences between screenshots are compared using a cross-correlation technique. Trigger conditions are defined to reduce the computational resources required. Additional optimizations to reduce processing resources and optimize accuracy may also be included.
1 . A computer-implemented method of detecting page changes of a graphical user interface (GUI) application screen, comprising:
detecting, by a page-change identification engine, a trigger condition based on at least one operating system event selected from mouse events, keyboard events, focus events, and reorder events;
in response to the trigger condition, capturing at least two screenshots of the GUI at different time instances;
dividing each captured screenshot into a plurality of spatial segments and identifying one or more non-blank segments containing active image content; computing, for each non-blank segment, a cross-correlation value between corresponding segments of the captured at least two screenshots;
determining that a page change has occurred when the cross-correlation value for any segment is below a predetermined threshold; and
transmitting an indication of the detected page change to a digital adoption platform to dynamically update contextual guidance corresponding to a new application state.
2 . The computer-implemented method of claim 1 , wherein a cross-correlation function is evaluated based on a reference screenshot captured in response to the trigger condition and a subsequent screenshot.
3 . The computer-implemented method of claim 1 , wherein the trigger condition comprises a first trigger condition comprising detecting mouse events and keyboard events followed by a second trigger condition comprising detecting reorder events and focus events.
4 . The computer-implemented method of claim 1 , wherein the threshold value is a range between 0.5 to 1.0.
5 . The computer-implemented method of claim 1 , further comprising performing a stability test, based on screenshots of an application, to determine if the user interface is stable.
6 . The computer-implemented method of claim 5 , wherein the stability test is based on performing a cross-correlation check for at least two screenshots taken after detecting a change in page content.
7 . The computer-implemented method of claim 1 , further comprising performing at least one retry if a cross-correlation function does not have a value below a threshold value indicative of a page change.
8 . The computer-implemented method of claim 1 , wherein a sequence of at least two consecutive screenshots is converted into grayscale images to reduce the number of dimensions for analysis.
9 . The computer-implemented method of claim 8 , further comprising caching grayscale versions of at least a first screenshot in a sequence of at least two consecutive screenshots.
10 . The computer-implemented method of claim 1 , wherein a sequence of at least two consecutive screenshots comprises screenshots of at least two different color images, with the cross-correlation function being performed on color images.
11 . A system for identifying page changes of a graphical user interface (GUI), comprising:
a page-change identification engine configured to:
detect a trigger condition based on at least one operating-system event selected from mouse events, keyboard events, focus events, and reorder events;
in response to detecting the trigger condition, capture at least two screenshots of the GUI at different time instances;
divide each captured screenshot into a plurality of spatial segments and identify one or more non-blank segments containing active image content;
compute, for each non-blank segment, a cross-correlation value between corresponding segments of the captured screenshots; and
determine that a page change has occurred when the cross-correlation value for any segment is below a predetermined threshold; and
a context-communication module configured to transmit an indication of a detected page change to a digital-adoption platform so that contextual guidance for the application is dynamically updated in response to the detected page change.
12 . The system of claim 11 , wherein the cross-correlation value is evaluated based on a reference screenshot captured in response to the trigger condition and a subsequent screenshot.
13 . The system of claim 11 , wherein the trigger condition comprises a first trigger condition comprising detecting mouse events and keyboard events followed by a second trigger condition comprising detecting reorder events and focus events.
14 . The system of claim 11 , wherein the threshold value is in a range between 0.5 to 1.0.
15 . The system of claim 11 , wherein the system is further configured to perform a stability test, based on screenshots of the application, to determine if the user interface is stable.
16 . The system of claim 15 , wherein the stability test is based on performing a cross-correlation check for at least two screenshots taken after detecting a change in page content.
17 . The system of claim 11 , wherein the system is further configured to perform at least one retry if a cross-correlation function does not have a value below a threshold value indicative of a page change.
18 . The system of claim 11 , wherein a sequence of at least two consecutive screenshots is converted into grayscale images to reduce the number of dimensions for analysis.
19 . The system of claim 18 , wherein the system is further configured to cache grayscale versions of at least a first screenshot in the sequence of at least two consecutive screenshots.
20 . The system of 11 , wherein a cross-correlation function is performed on screenshots of color images.
21 . A computer-implemented method of detecting page changes of a graphical user interface (GUI) application screen, comprising:
determining if a trigger condition is satisfied for a page change, the trigger condition being based on a combination of at least two different operating system events associated with a likely page change of an application;
subsequent to satisfying the trigger condition, capturing a sequence of at least two screenshots of the application taken at different times, and comparing the at least two screenshots using a cross correlation function;
identifying a page change in response to the cross-correlation function having a value below a first threshold value indicative of a page change;
evaluating page stability using a stability score;
in response to identifying the page change and that the page is stable, providing an indication of the identified page change to a digital adoption platform to dynamically update contextual guidance corresponding to a new application state.