CAPTURING AND PROCESSING INTERACTIONS WITH A USER INTERFACE OF A NATIVE APPLICATION
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for evaluating interactions with a user interface of an application are disclosed. In one aspect, a method includes receiving frame bundles for a user session with a native application. Each frame bundle can include data specifying, for each of one or more points in time, a presentation position of each presentation object used by the native application to generate a user interface of the native application at the point in time, and, for one or more presentation objects, one or more drawing operations performed to generate the visual representation of the presentation object. Playback data that presents visual changes of the user interface corresponding to the drawing operations performed to generate the visual representation of each presentation object is generated based on the data specified by the frame bundles.
1 . A method comprising:
receiving a plurality of frame bundles for a user session with a native application at a client device, each frame bundle including data specifying:
for each of one or more points in time:
a presentation position of each presentation object used by the native application to generate a user interface of the native application at the point in time, each presentation object being an object that generates a visual representation of itself within a portion of the user interface; and
for one or more presentation objects, data representing one or more drawing operations performed to generate the visual representation of the presentation object at the client device; and
generating, based on the data specified by the plurality of frame bundles, playback data that presents playback of the user session including visual changes of the user interface that occurred during the user session, the generating including processing the data representing the one or more drawing operations to redraw the one or more presentation objects during playback of the user session.
2 . The method of claim 1 , wherein:
each presentation object comprises at least one of a view or a layer and wherein each frame bundle includes data specifying a hierarchical representation of views and/or layers for each of the one or more points in time; and
processing the data representing the one or more drawing operations to redraw the one or more presentation objects during playback of the user session comprises processing at least one drawing operation for each layer or view to redraw the layer or view during playback of the user session.
3 . The method of claim 2 , wherein the native application comprises a window scanner configured to obtain the data specifying the hierarchical representation of the views and/or layers and to obtain, using the hierarchical representation, the at least one drawing operation for each layer and/or view.
4 . The method of claim 1 , wherein generating the playback data comprises recreating the visual representation of each presentation object based on the data representing the drawing operations used to generate the presentation object during the user session.
5 . The method of claim 1 , wherein instrumentation code of the native application generates each frame bundle, including causing each layer of each presentation object to draw itself to an offscreen drawing context and storing data specifying drawing operations performed to draw each layer of each presentation object in the offscreen drawing context.
6 . The method of claim 5 , wherein the drawing context is a Portable Document Format (PDF) based drawing context and data specifying each drawing operation is stored in a PDF document.
7 . The method of claim 6 , wherein the instrumentation code creates a respective PDF page for each presentation object.
8 . The method of claim 6 , wherein the instrumentation code causes, for each given presentation object, a drawing library of an operating system on which the native application runs to draw, to the offscreen drawing context, a start marker for the presentation object prior to the view being drawn to the offscreen drawing context and an end marker for the presentation object after the presentation object has been drawn to the offscreen drawing context.
9 . The method of claim 8 , wherein the instrumentation code assigns each drawing operation of the offscreen drawing context between the start marker and the end marker to the given presentation object.
10 . The method of claim 1 , wherein:
each frame bundle includes data specifying one or more user interface events that occurred during the presentation of one or more presentation objects; and
generating the playback data comprises recreating the one or more user interface events during presentation of the one or more presentation objects.
11 . A system, comprising:
a data store; and
one or more computers that interact with the data store and execute instructions that cause the one or more computers to perform operations comprising:
receiving a plurality of frame bundles for a user session with a native application at a client device, each frame bundle including data specifying:
for each of one or more points in time:
a presentation position of each presentation object used by the native application to generate a user interface of the native application at the point in time, each presentation object being an object that generates a visual representation of itself within a portion of the user interface; and
for one or more presentation objects, data representing one or more drawing operations performed to generate the visual representation of the presentation object at the client device; and
generating, based on the data specified by the plurality of frame bundles, playback data that presents playback of the user session including visual changes of the user interface that occurred during the user session, the generating including processing the data representing the one or more drawing operations to redraw the one or more presentation objects during playback of the user session.
12 . The system of claim 9 , wherein:
each presentation object comprises at least one of a view or a layer and wherein each frame bundle includes data specifying a hierarchical representation of views and/or layers for each of the one or more points in time; and
processing the data representing the one or more drawing operations to redraw the one or more presentation objects during playback of the user session comprises processing at least one drawing operation for each layer or view to redraw the layer or view during playback of the user session.
13 . The system of claim 12 , wherein the native application comprises a window scanner configured to obtain the data specifying the hierarchical representation of the views and/or layers and to obtain, using the hierarchical representation, the at least one drawing operation for each layer and/or view.
14 . The system of claim 11 , wherein generating the playback data comprises recreating the visual representation of each presentation object based on the data representing the drawing operations used to generate the presentation object during the user session.
15 . The system of claim 11 , wherein instrumentation code of the native application generates each frame bundle, including causing each layer of each presentation object to draw itself to an offscreen drawing context and storing data specifying drawing operations performed to draw each layer of each presentation object in the offscreen drawing context.
16 . The system of claim 15 , wherein the drawing context is a Portable Document Format (PDF) based drawing context and data specifying each drawing operation is stored in a PDF document.
17 . The method of claim 16 , wherein the instrumentation code creates a respective PDF page for each presentation object.
18 . The system of claim 16 , wherein the instrumentation code causes, for each given presentation object, a drawing library of an operating system on which the native application runs to draw, to the offscreen drawing context, a start marker for the presentation object prior to the view being drawn to the offscreen drawing context and an end marker for the presentation object after the presentation object has been drawn to the offscreen drawing context.
19 . The system of claim 18 , wherein the instrumentation code assigns each drawing operation of the offscreen drawing context between the start marker and the end marker to the given presentation object.
20 . A non-transitory computer readable medium storing instructions that upon execution by one or more computers cause the one or more computers to perform operations comprising:
receiving a plurality of frame bundles for a user session with a native application at a client device, each frame bundle including data specifying:
for each of one or more points in time:
a presentation position of each presentation object used by the native application to generate a user interface of the native application at the point in time, each presentation object being an object that generates a visual representation of itself within a portion of the user interface; and
for one or more presentation objects, data representing one or more drawing operations performed to generate the visual representation of the presentation object at the client device; and
generating, based on the data specified by the plurality of frame bundles, playback data that presents playback of the user session including visual changes of the user interface that occurred during the user session, the generating including processing the data representing the one or more drawing operations to redraw the one or more presentation objects during playback of the user session.