Remote supervision of client device activity
A server device may receive a series of at least two screen capture representations of a graphical user interface on a client device. A first active window for a first screen capture representation of the series and a second active window for a second screen capture representation of the series may be determined. The first screen capture representation may have been screen captured by the client device at a first time and the second screen capture representation may have been screen captured by the client device at a second time. A first application associated with the first active window and a second application associated with the second active window may be identified, at least one of which may be a pre-determined target application. Based on the identified applications, an activity log for the client device may be determined.
1. A method comprising:
receiving, by a server device, a series of screen capture representations of a graphical user interface on a client device, wherein a first screen capture representation of the series was captured by the client device at a first time and a second screen capture representation of the series was captured by the client device at a second time, and wherein the first screen capture representation includes a representation of multiple windows from the graphical user interface;
sliding rectangular outlines of varying sizes over the first screen capture representation;
identifying when one of the rectangular outlines surrounds four borders of a rectangular shape in the first screen capture representation;
determining that the rectangular shape is a first active window;
identifying a first application associated with the first active window;
identifying a second application associated with a second active window from the second screen capture representation, wherein at least one of the first application or the second application is a pre-determined target application; and
based on the identification of the first active window and the second active window, determining, by the server device, an activity log for the client device, wherein the activity log identifies one or more idle periods and one or more busy periods for the pre-determined target application.
2. The method of claim 1 , wherein the first screen capture representation consists of the first active window and the second screen capture representation consists of the second active window.
3. The method of claim 1 , wherein the first application was executing in the first active window at the first time and the second application was executing in the second active window at the second time.
4. The method of claim 1 , wherein identifying the first application associated with the first active window comprises performing character recognition on text in a title bar of the first active window, wherein the recognized characters identify the first application.
5. The method of claim 1 , wherein identifying the first application associated with the first active window comprises recognizing a logo within the first active window, wherein the recognized logo identifies the first application.
6. The method of claim 1 , wherein identifying the first application associated with the first active window comprises:
classifying characteristics of the first active window into a vector;
comparing the vector to a database of vectors, wherein vectors in the database represent characteristics of particular applications; and
based on the comparison, selecting one of the particular applications as the first application.
7. The method of claim 6 , wherein the characteristics of the first active window include text and color patterns within the active window, and wherein the characteristics of particular applications include text and color patterns of the particular applications.
8. The method of claim 1 , wherein a user of the client device is associated with a reliability factor, and wherein a frequency at which the client device performs the screen captures is inversely proportional to the reliability factor.
9. The method of claim 1 , wherein determining the activity log for the client device comprises:
determining that the first screen capture representation begins a contiguous subset of the screen capture representations, the screen capture representations in the subset containing respective active windows identified as the pre-determined target application;
determining that the second screen capture representation ends the contiguous subset of the screen capture representations; and
identifying, in the activity log, from the first time to the second time as a busy period.
10. The method of claim 1 , wherein an idle timer is calibrated to a duration of t contiguous screen captures, and wherein determining the activity log for the client device comprises:
determining that an ith screen capture representation of the series begins a contiguous subset of m essentially identical screen capture representations, the screen capture representations in the subset containing respective active windows identified as the pre-determined target application;
determining that a (i+m)th screen capture representation ends the contiguous subset of the screen capture representations, wherein m is greater than t;
identifying, in the activity log, a duration of the ith screen capture representation through the (i+t)th screen capture representation as a busy period; and
identifying, in the activity log, a duration of the (i+t)th screen capture representation to the (i+m)th screen capture representation as an idle period.
11. The method of claim 1 , wherein determining the activity log for the client device comprises:
determining that the first screen capture representation begins a contiguous subset of the screen capture representations, none of the screen capture representations in the subset containing respective active windows identified as being the pre-determined target application;
determining that the second screen capture representation ends the contiguous subset of the screen capture representations; and
identifying, in the activity log, from the first time to the second time as an idle period.
12. The method of claim 1 , further comprising:
based on the activity log, determining a first total time that a user of the client device used the client device and a second total time that the user actively used the pre-determined target application on the client device, wherein the first total time includes idle and busy periods, wherein the second total time includes only busy periods, and wherein the first total time is greater than the second total time.
13. An article of manufacture including a non-transitory computer-readable medium, having stored thereon program instructions that, upon execution by a computing device, cause the computing device to perform operations comprising:
receiving a series of screen capture representations of a graphical user interface on a client device, wherein a first screen capture representation of the series was captured by the client device at a first time and a second screen capture representation of the series was captured by the client device at a second time, and wherein the first screen capture representation includes a representation of multiple windows from the graphical user interface;
sliding rectangular outlines of varying sizes over the first screen capture representation;
identifying when one of the rectangular outlines surrounds four borders of a rectangular shape in the first screen capture representation;
determining that the rectangular shape is a first active window;
identifying a first application associated with the first active window;
identifying a second application associated with a second active window from the second screen capture representation, wherein at least one of the first application or the second application is a pre-determined target application; and
based on the identification of the first active window and the second active window, determining an activity log for the client device, wherein the activity log identifies one or more idle periods and one or more busy periods for the pre-determined target application.
14. The article of manufacture of claim 13 , wherein the first application was executing in the first active window at the first time and the second application was executing in the second active window at the second time.
15. The article of manufacture of claim 13 , wherein identifying the first application associated with the first active window comprises performing character recognition on text in a title bar of the first active window, wherein the recognized characters identify the first application.
16. The article of manufacture of claim 13 , wherein identifying the first application associated with the first active window comprises recognizing a logo within the first active window, wherein the recognized logo identifies the first application.
17. A computing device comprising:
at least one processor;
memory; and
program instructions, stored in the memory, that upon execution by the at least one processor cause the computing device to perform operations comprising:
receiving a series of screen capture representations of a graphical user interface on a client device, wherein a first screen capture representation of the series was captured by the client device at a first time and a second screen capture representation of the series was captured by the client device at a second time, and wherein the first screen capture representation includes a representation of multiple windows from the graphical user interface;
sliding rectangular outlines of varying sizes over the first screen capture representation;
identifying when one of the rectangular outlines surrounds four borders of a rectangular shape in the first screen capture representation;
determining that the rectangular shape is a first active window;
identifying a first application associated with the first active window;
identifying a second application associated with a second active window from the second screen capture representation, wherein at least one of the first application or the second application is a pre-determined target application; and
based on the identification of the first active window and the second active window, determining an activity log for the client device, wherein the activity log identifies one or more idle periods and one or more busy periods for the pre-determined target application.
18. The computing device of claim 17 , wherein the first application was executing in the first active window at the first time and the second application was executing in the second active window at the second time.
19. The computing device of claim 17 , wherein identifying the first application associated with the first active window comprises performing character recognition on text in a title bar of the first active window, wherein the recognized characters identify the first application.
20. The computing device of claim 17 , wherein identifying the first application associated with the first active window comprises recognizing a logo within the first active window, wherein the recognized logo identifies the first application.