Capturing and processing interactions with a user interface of a native application
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for evaluating interactions with a user interface of an application are disclosed. In one aspect, a method includes, for each of a plurality of different user sessions of a native application, accessing frame bundles that each include data representing content presented by a frame of a user interface of the native application at a given time. Each frame bundle includes at least a portion of a view tree of the native application used to generate the user interface at the given time and data specifying content presented by each view of the portion of the view tree. Based on the frame bundles, playback data are generated that present visual changes of the user interface corresponding to changes to the view trees.
1. A method comprising:
initiating, at a client device, a user session with a user interface of a native application running on the client device;
initiating, at the client device, an instrumentation module for collecting event data for use in recreating playback of the user session at a computing device different from the client device;
passing, into each drawing method of a proper subset of a set of drawing methods of the native application, a canvas recorder subclass that captures drawing operations performed by the proper subset of drawing methods;
for each frame of multiple frames of the user interface displayed during the user session:
obtaining, by the instrumentation module, data representing a view tree used to generate display of the frame, wherein the view tree comprises a plurality of views drawn to a display of the client device to generate display of the frame; and
obtaining, by the instrumentation module and for each view of one or more of the plurality of views, data specifying one or more drawing operations processed by the native application to draw content of the view on the display of the client device, wherein, for views that include one or more drawing methods of the proper subset of drawing methods, the one or more drawing operations comprise at least one drawing operation captured by the canvas recorder subclass;
generating the event data comprising, for each one or more of the multiple frames, the data representing the view tree and the data specifying the one or more drawing operations; and
sending the event data to the computing device.
2. The method of claim 1 , further comprising identifying, for inclusion in the proper subset of the set of drawing methods, each occurrence of a drawing method in the native application that is in a list of drawing methods into which the canvas recorder subclass is eligible to be passed.
3. The method of claim 1 , further comprising instrumenting, by the instrumentation module, one or more drawing methods using one or more reflection techniques such that calls of each drawing method of the one or more drawing methods are replaced or augmented with a stub method that collects data related to calls of the drawing method.
4. The method of claim 1 , further comprising:
identifying a given drawing method that renders bits of control and inserting a shim that intercepts calls to the given drawing method to collect results of the given drawing method; and
including, in the event data, the results of the given drawing method for at least one call of the drawing method during the user session.
5. The method of claim 1 , further comprising mapping image data with resource names by instrumenting methods called by the native application with shims.
6. The method of claim 1 , further comprising initializing, by the instrumentation module, a view scanner configured to obtain data from view trees of the native application.
7. The method of claim 6 , further comprising, for each frame of the user interface displayed during the user session:
iterating, by the view scanner, through each view attached to each window;
collecting, by the view scanner, data for each view; and
generating, by the view scanner, the view tree for the frame using the collected data.
8. The method of claim 7 , wherein the data for each view comprises at least one of (i) a position on screen of the view, (ii) transparency of the view, or (iii) view transform for the view.
9. The method of claim 7 , further comprising:
identifying, for a given frame, a clipped view that is obscured by one or more other views; and
discarding the clipped view from the view tree for the given frame.
10. A system, comprising:
a data store; and
one or more hardware processors that interact with the data store and execute instructions that cause the one or more one or more hardware processors to perform operations comprising:
initiating, at a client device, a user session with a user interface of a native application running on the client device;
initiating, at the client device, an instrumentation module for collecting event data for use in recreating playback of the user session at a computing device different from the client device;
passing, into each drawing method of a proper subset of a set of drawing methods of the native application, a canvas recorder subclass that captures drawing operations performed by the proper subset of drawing methods;
for each of frame of multiple frames of the user interface displayed during the user session:
obtaining, by the instrumentation module, data representing a view tree used to generate display of the frame, wherein the view tree comprises a plurality of views drawn to a display of the client device to generate display of the frame; and
obtaining, by the instrumentation module and for each view of one or more of the plurality of views, data specifying one or more drawing operations processed by the native application to draw content of the view on the display of the client device, wherein, for views that include one or more drawing methods of the proper subset of drawing methods, the one or more drawing operations comprise at least one drawing operation captured by the canvas recorder subclass;
generating the event data comprising, for each one or more of the multiple frames, the data representing the view tree and the data specifying the one or more drawing operations; and
sending the event data to the computing device.
11. The system of claim 10 , wherein the operations comprise identifying, for inclusion in the proper subset of the set of drawing methods, each occurrence of a drawing method in the native application that is in a list of drawing methods into which the canvas recorder subclass is eligible to be passed.
12. The system of claim 10 , wherein the operations comprise instrumenting, by the instrumentation module, one or more drawing methods using one or more reflection techniques such that calls of each drawing method of the one or more drawing methods are replaced or augmented with a stub method that collects data related to calls of the drawing method.
13. The system of claim 10 , wherein the operations comprise:
identifying a given drawing method that renders bits of control and inserting a shim that intercepts calls to the given drawing method to collect results of the given drawing method; and
including, in the event data, the results of the given drawing method for at least one call of the drawing method during the user session.
14. The system of claim 10 , wherein the operations comprise mapping image data with resource names by instrumenting methods called by the native application with shims.
15. The system of claim 10 , wherein the operations comprise initializing, by the instrumentation module, a view scanner configured to obtain data from view trees of the native application.
16. The system of claim 15 , wherein the operations comprise, for each frame of the user interface displayed during the user session:
iterating, by the view scanner, through each view attached to each window;
collecting, by the view scanner, data for each view; and
generating, by the view scanner, the view tree for the frame using the collected data.
17. The system of claim 16 , wherein the data for each view comprises at least one of (i) a position on screen of the view, (ii) transparency of the view, or (iii) view transform for the view.
18. The system of claim 16 , wherein the operations comprise:
identifying, for a given frame, a clipped view that is obscured by one or more other views; and
discarding the clipped view from the view tree for the given frame.
19. A non-transitory computer readable medium storing instructions that upon execution by one or more processors cause the one or more processors to perform operations comprising:
initiating, at a client device, a user session with a user interface of a native application running on the client device;
initiating, at the client device, an instrumentation module for collecting event data for use in recreating playback of the user session at a computing device different from the client device;
passing, into each drawing method of a proper subset of a set of drawing methods of the native application, a canvas recorder subclass that captures drawing operations performed by the proper subset of drawing methods;
for each frame of multiple frames of the user interface displayed during the user session:
obtaining, by the instrumentation module, data representing a view tree used to generate display of the frame, wherein the view tree comprises a plurality of views drawn to a display of the client device to generate display of the frame; and
obtaining, by the instrumentation module and for each view of one or more of the plurality of views, data specifying one or more drawing operations processed by the native application to draw content of the view on the display of the client device, wherein, for views that include one or more drawing methods of the proper subset of drawing methods, the one or more drawing operations comprise at least one drawing operation captured by the canvas recorder subclass;
generating the event data comprising, for each one or more of the multiple frames, the data representing the view tree and the data specifying the one or more drawing operations; and
sending the event data to the computing device.