Methods and systems for automatically recognizing actions in a call center environment using screen capture technology

1. A method for recognizing processes performed by a call center agent during a session, the method comprising:

collecting, by an electronic device from one or more input devices in communication with the electronic device, input data comprising data pertaining to one or more interactions that a call center agent has with one or more programs running on the electronic device that cause one or more graphical user interfaces to be displayed on a desktop of the electronic device during a session;

analyzing, by the electronic device, the input data to generate one or more events, wherein each event comprises a consolidation of at least a portion of the input data;

generating, by the electronic device, a mid-level event log comprising one or more of the events;

performing, by the electronic device, action recognition on the mid-level event log to ascertain one or more actions that were performed within the one or more graphical user interfaces by the call center agent during the session, by:

for one or more events in the mid-level event log:

identifying a before-event image for the event, wherein the before-event image is a screen shot of the desktop that was automatically captured by the electronic device in real-time before the event occurred and in anticipation of the event,

identifying an after-event image for the event, wherein the after-event image is a screen shot of the desktop of the electronic device that was automatically captured by the electronic device in real-time after the event occurred,

performing image analysis on the before-event image and the after-event to generate a delta image showing a region of interest that includes one or more changes between the before-event image and the after-event image, and

automatically classifying the event as an action based on the delta image, and

generate an event log that comprises an indication of the event and corresponding action.

2. The method of claim 1 , wherein collecting input data comprises one or more of the following:

initiating a keyboard logging thread to collect one or more keystrokes that are entered via a keyboard in communication with the electronic device during the session;

initiating a mouse logging thread to collect data associated with movement or operation of the mouse during the session; or

initiating an active window logging thread to collect data associated with active windows during the session.

3. The method of claim 1 , wherein generating the mid-level event log based on the input data comprises:

identifying consecutive keyboard entries in the input data; and

in response to determining that a time between each keyboard entry is less than a threshold value, merging the keyboard entries into a single typing event.

4. The method of claim 1 , wherein generating the mid-level event log based on the input data comprises:

identifying from the input data representing a mousedown event corresponding to a mouse followed by data representing a mouseup event corresponding to the mouse such that there is no change in a position of the mouse; and

consolidating the mousedown event and the mouse up event into a single event represented as a mouse click event.

5. The method of claim 1 , wherein generating the mid-level event log based on the input data comprises:

identifying from the input data consecutive mouse clicks corresponding to a mouse; and

consolidating the consecutive mouse clicks into a single event represented as a double click mouse click event.

6. The method of claim 1 , wherein generating the mid-level event log based on the input data comprises:

identifying from the input data a mousedown event followed by a plurality of mouse movement events followed by a mouseup event; and

consolidating the mousedown event, the mouse movement events and the mouseup event into a single event represented as a mouse select event.

7. The method of claim 1 , wherein generating the mid-level event log based on the input data comprises:

identifying from the input data consecutive mouse wheel events; and

consolidating the consecutive mouse wheel events into a single event represented as a mouse scroll event.

8. The method of claim 1 , wherein performing image analysis on the before-event image and the after-event to generate a delta image comprises:

identifying a screen to which the before-event image and the after-event image correspond by comparing the before-event image and the after-event image to one or more screens in a unique screen library, wherein the unique screen library stores each unique screen that the call agent may encounter during a session along with metadata associated with each unique screen;

analyzing the screen to identify a region on the screen that corresponds to the region of interest; and

analyzing the metadata associated with the screen to identify a data field located within the region.

9. The method of claim 8 , wherein:

the event is a mouse click,

analyzing the metadata associated with the screen to identify a data field located within the region comprises analyzing the metadata to identify a dropdown menu,

classifying the event as an action comprises classifying the event as a select dropdown menu action.

10. The method of claim 1 , further comprising applying a process discovery technique to the event log to generate a process map showing the actions performed by the call center agent during the session.

11. A system for recognizing processes performed by a call center agent during a session, the system comprising:

an electronic device; and

a computer-readable storage medium in communication with the electronic device, wherein the computer-readable storage medium comprises one or more programming instructions that, when executed, cause the electronic device to:

collect from one or more input devices in communication with the electronic device, input data comprising data pertaining to one or more interactions that a call center agent has with one or more programs running on the electronic device that cause one or more graphical user interfaces to be displayed on a desktop of the electronic device during a session,

analyze the input data to generate one or more events, wherein each event comprises a consolidation of at least a portion of the input data,

generate a mid-level event log comprising one or more of the events,

perform action recognition on the mid-level event log to ascertain one or more actions that were performed within the one or more graphical user interfaces by the call center agent during the session, by:

for one or more events in the mid-level event log:

identify a before-event image for the event, wherein the before-event image is a screen shot of the desktop that was automatically captured by the electronic device in real-time before the event occurred and in anticipation of the event,

identify an after-event image for the event, wherein the after-event image is a screen shot of the desktop of the electronic device that was automatically captured by the electronic device in real-time after the event occurred,

perform image analysis on the before-event image and the after-event to generate a delta image showing a region of interest that includes one or more changes between the before-event image and the after-event image, and

automatically classify the event as an action based on the delta image, and

generate an event log that comprises an indication of the event and corresponding action.

12. The system of claim 11 , wherein the one or more programming instructions that, when executed, cause the electronic device to collect input data comprise one or more programming instructions that, when executed, cause the electronic device to perform one or more of the following:

initiate a keyboard logging thread to collect one or more keystrokes that are entered via a keyboard in communication with the electronic device during the session;

initiate a mouse logging thread to collect data associated with movement or operation of the mouse during the session; or

initiate an active window logging thread to collect data associated with active windows during the session.

13. The system of claim 11 , wherein the one or more programming instructions that, when executed, cause the electronic device to generate the mid-level event log based on the input data comprise one or more programming instructions that, when executed, cause the electronic device to:

identify consecutive keyboard entries in the input data; and

in response to determining that a time between each keyboard entry is less than a threshold value, merge the keyboard entries into a single typing event.

14. The system of claim 11 , wherein the one or more programming instructions that, when executed, cause the electronic device to generate the mid-level event log based on the input data comprise one or more programming instructions that, when executed, cause the electronic device to:

identify from the input data representing a mousedown event corresponding to a mouse followed by data representing a mouseup event corresponding to the mouse such that there is no change in a position of the mouse; and

consolidate the mousedown event and the mouse up event into a single event represented as a mouse click event.

15. The system of claim 11 , wherein the one or more programming instructions that, when executed, cause the electronic device to generate the mid-level event log based on the input data comprise one or more programming instructions that, when executed, cause the electronic device to:

identify from the input data consecutive mouse clicks corresponding to a mouse; and

consolidate the consecutive mouse clicks into a single event represented as a double click mouse click event.

16. The system of claim 11 , wherein the one or more programming instructions that, when executed, cause the electronic device to generate the mid-level event log based on the input data comprise one or more programming instructions that, when executed, cause the electronic device to:

identify from the input data a mousedown event followed by a plurality of mouse movement events followed by a mouseup event; and

consolidate the mousedown event, the mouse movement events and the mouseup event into a single event represented as a mouse select event.

17. The system of claim 11 , wherein the one or more programming instructions that, when executed, cause the electronic device to generate the mid-level event log based on the input data comprise one or more programming instructions that, when executed, cause the electronic device to:

identify from the input data consecutive mouse wheel events; and

consolidate the consecutive mouse wheel events into a single event represented as a mouse scroll event.

18. The system of claim 11 , wherein the one or more programming instructions that, when executed, cause the electronic device to perform image analysis on the before-event image and the after-event to generate a delta image comprise one or more programming instructions that, when executed, cause the electronic device to:

identify a screen to which the before-event image and the after-event image correspond by comparing the before-event image and the after-event image to one or more screens in a unique screen library, wherein the unique screen library stores each unique screen that the call agent may encounter during a session along with metadata associated with each unique screen;

analyze the screen to identify a region on the screen that corresponds to the region of interest; and

analyze the metadata associated with the screen to identify a data field located within the region.

19. The system of claim 18 , wherein:

the event is a mouse click,

the one or more programming instructions that, when executed, cause the electronic device to analyze the metadata associated with the screen to identify a data field located within the region comprise one or more programming instructions that, when executed, cause the electronic device to analyze the metadata to identify a dropdown menu,

the one or more programming instructions that, when executed, cause the electronic device to classify the event as an action comprise one or more programming instructions that, when executed, cause the electronic device to classify the event as a select dropdown menu action.

20. The system of claim 11 , wherein the computer-readable storage medium further comprises one or more programming instructions that, when executed, cause the electronic device to apply a process discovery technique to the event log to generate a process map showing the actions performed by the call center agent during the session.