Methods and systems for automatically recognizing actions in a call center environment using video data

1. A method for recognizing processes performed by a call center agent during a session, the method comprising:

receiving, by a receiver of an electronic device, input data from a screen recording system, wherein the input data comprises a video signal representing a plurality of frames from a time-series video recording or video stream of a desktop of a user electronic device, wherein the user electronic device is associated with a call center agent in a call center environment, wherein each frame comprises an image of one or more screens displayed on the desktop during a session;

processing the video signal to identify one or more frames of interest from the plurality of frames by:

identifying a first frame from the plurality of frames as a background frame, wherein the background frame is associated with a unique identifier,

identifying a second frame from the plurality of frames, wherein the second frame occurs after the background frame in the time-series video recording,

determining whether a difference in layout exists between the background frame and the second frame by:

comparing the second frame to one or more unique screen images from a unique screen library to determine a unique screen image from the unique screen library that corresponds to the second frame,

identifying a screen identifier associated with the unique screen image that corresponds to the second frame, and

determining that the difference in layout exists in response to the screen identifier associated with the unique screen image that corresponds to the second frame being different than the unique identifier associated with the background frame,

in response to determining that the difference in layout exists between the background and the second frame, identifying the second frame as a frame of interest and updating the background frame to reflect the second frame,

automatically identifying within one or more of the frames of interest one or more actions performed by the call center agent, wherein each action represents an interaction that the call center agent had with one or more screens corresponding to the frame of interest; and

generating an event log that comprises a timeline depicting a visual representation of the identified actions.

2. The method of claim 1 , wherein receiving input data from the screen recording system comprises receiving the input data from one or more video recording sensors that support video compression methods.

3. The method of claim 1 , wherein comparing the second frame to one or more unique screen images comprises:

for one or more of the unique screen images:

applying a first scale to the second frame and the unique screen image to determine a first score, wherein the first score is indicative of how well a high level layout of the second frame corresponds to a high level layout of the unique screen image,

applying a second scale to the second frame and the unique screen image to determine a second score, wherein the second score is indicative of how well a low level layout of the second frame corresponds to a low level layout of the unique screen image,

determining a final match score for the unique screen image by summing the first score and the second score; and

selecting the unique screen image having the highest corresponding final match score.

4. The method of claim 1 , wherein automatically identifying within one or more of the frames of interest one or more actions performed by the call center agent comprises:

performing frame differencing on the one or more frames of interest to detect one or more regions of interest, wherein each region of interest represents a change in pixel level;

determining whether the region of interest comprises a desktop window; and

in response to determining that the region of interest comprises a desktop window, identifying a window opening as a performed action.

5. The method of claim 1 , wherein automatically identifying within one or more of the frames of interest one or more actions performed by the call center agent comprises:

performing frame differencing on the one or more frames of interest to detect one or more regions of interest, wherein each region of interest represents a change in pixel level;

determining whether the region of interest comprises a movement in a position of a mouse cursor; and

in response to determining that the region of interest comprises a movement in a position of a mouse cursor, identifying a mouse event as a performed action.

6. The method of claim 1 , wherein generating an event log that comprises a timeline depicting a visual representation of the identified actions comprises:

identifying whether the identified actions comprise a plurality of short-duration typing events at a same text-field;

in response to identifying that the identified actions comprise a plurality of short-duration typing events at a same text-field, combining the short-duration typing events into a single typing event; and

adding the single typing event to the event log.

7. The method of claim 1 , wherein the input data further comprises audio data corresponding to a conversation between the call center agent and a caller during the session, the method further comprising:

analyzing the audio data for one or more periods of dead time during which neither the call center agent nor the caller is speaking; and

for one or more of the periods of dead time, analyzing the identified actions that occurred during the periods of dead time to determine one or more possible reasons for the periods of dead time.

8. A system for recognizing processes performed by a call center agent during a session, the system comprising:

an electronic device; and

a computer-readable storage medium comprising one or more programming instructions that, when executed, cause the electronic device to:

receive input data from a screen recording system, wherein the input data comprises a video signal representing a plurality of frames from a time-series video recording or video stream of a desktop of a user electronic device, wherein the user electronic device is associated with a call center agent in a call center environment, wherein each frame comprises an image of one or more screens displayed on the desktop during a session,

process the video signal to identify one or more frames of interest from the plurality of frames by:

identifying a first frame from the plurality of frames as a background frame, wherein the background frame is associated with a unique identifier,

identifying a second frame from the plurality of frames, wherein the second frame occurs after the background frame in the time-series video recording,

determining whether a difference in layout exists between the background frame and the second frame by:

comparing the second frame to one or more unique screen images from a unique screen library to determine a unique screen image from the unique screen library that corresponds to the second frame,

identifying a screen identifier associated with the unique screen image that corresponds to the second frame, and

determining that the difference in layout exists in response to the screen identifier associated with the unique screen image that corresponds to the second frame being different than the unique identifier associated with the background frame,

in response to determining that the difference in layout exists between the background and the second frame, identifying the second frame as a frame of interest and updating the background frame to reflect the second frame,

automatically identify within one or more of the frames of interest one or more actions performed by the call center agent, wherein each action represents an interaction that the call center agent had with one or more screens corresponding to the frame of interest, and

generate an event log that comprises a timeline depicting a visual representation of the identified actions.

9. The system of claim 8 , wherein the one or more programming instructions that, when executed, cause the electronic device to receive input data from the screen recording system comprise one or more programming instructions that, when executed, cause the electronic device to receive the input data from one or more video recording sensors that support video compression methods.

10. The system of claim 8 , wherein the one or more programming instructions that, when executed, cause the electronic device to compare the second frame to one or more unique screen images comprise one or more programming instructions that, when executed, cause the electronic device to:

for one or more of the unique screen images:

apply a first scale to the second frame and the unique screen image to determine a first score, wherein the first score is indicative of how well a high level layout of the second frame corresponds to a high level layout of the unique screen image,

apply a second scale to the second frame and the unique screen image to determine a second score, wherein the second score is indicative of how well a low level layout of the second frame of interest corresponds to a low level layout of the unique screen image, and

determine a final match score for the unique screen image by summing the first score and the second score; and

select the unique screen image having the highest corresponding final match score.

11. The system of claim 8 , wherein the one or more programming instructions that, when executed, cause the electronic device to automatically identify within one or more of the frames of interest one or more actions performed by the call center agent comprise one or more programming instructions that, when executed, cause the electronic device to:

perform frame differencing on the one or more frames of interest to detect one or more regions of interest, wherein each region of interest represents a change in pixel level;

determine whether the region of interest comprises a desktop window; and

in response to determining that the region of interest comprises a desktop window, identify a window opening as a performed action.

12. The system of claim 8 , wherein the one or more programming instructions that, when executed, cause the electronic device to automatically identify within one or more of the frames of interest one or more actions performed by the call center agent comprise one or more programming instructions that, when executed, cause the electronic device to:

perform frame differencing on the one or more frames of interest to detect one or more regions of interest, wherein each region of interest represents a change in pixel level;

determine whether the region of interest comprises a movement in a position of a mouse cursor; and

in response to determining that the region of interest comprises a movement in a position of a mouse cursor, identify a mouse event as a performed action.

13. The system of claim 8 , wherein the one or more programming instructions that, when executed, cause the electronic device to generate an event log that comprises a timeline depicting a visual representation of the identified actions comprise one or more programming instructions that, when executed, cause the electronic device to:

identify whether the identified actions comprise a plurality of short-duration typing events at a same text-field;

in response to identifying that the identified actions comprise a plurality of short-duration typing events at a same text-field, combine the short-duration typing events into a single typing event; and

add the single typing event to the event log.

14. The system of claim 8 , wherein:

the input data further comprises audio data corresponding to a conversation between the call center agent and a caller during the session,

the computer-readable storage medium further comprises one or more programming instructions that, when executed, cause the electronic device to:

analyze the audio data for one or more periods of dead time during which neither the call center agent nor the caller is speaking; and

for one or more of the periods of dead time, analyze the identified actions that occurred during the periods of dead time to determine one or more possible reasons for the periods of dead time.