CAPTURING AND PROCESSING INTERACTIONS WITH A USER INTERFACE OF A NATIVE APPLICATION

1 . A method comprising:

receiving a plurality of frame bundles for a user session with a native application, each frame bundle including data specifying:

for each of one or more points in time:

a presentation position of each presentation object used by the native application to generate a user interface of the native application at the point in time, each presentation object being an object that generates a visual representation of itself within a portion of the user interface; and

for one or more presentation objects, one or more drawing operations performed to generate the visual representation of the presentation object; and

generating, based on the data specified by the plurality of frame bundles, playback data that present visual changes of the user interface corresponding to the drawing operations performed to generate the visual representation of each presentation object.

2 . The method of claim 1 , wherein each presentation object comprises one of a view or a layer and wherein each frame bundle includes data specifying a hierarchical representation of each view or layer.

3 . The method of claim 1 , wherein generating the playback data comprises recreating the visual representation of each presentation object based on the drawing operations used to generate the presentation object.

4 . The method of claim 1 , wherein instrumentation code of the native application generates each frame bundle, including causing each layer of each presentation object to draw itself to an offscreen drawing context and storing data specifying drawing operations performed to draw each layer of each presentation object in the offscreen drawing context.

5 . The method of claim 4 , wherein the drawing context is a Portable Document Format (PDF) based drawing context and data specifying each drawing operation is stored in a PDF document.

6 . The method of claim 4 , wherein the instrumentation code causes, for each given presentation object, a drawing library of an operating system on which the native application runs to draw, to the offscreen drawing context, a start marker for the presentation object prior to the view being drawn to the offscreen drawing context and an end marker for the presentation object after the presentation object has been drawn to the offscreen drawing context.

7 . The method of claim 6 , wherein the instrumentation code assigns each drawing operation of the offscreen drawing context between the start marker and the end marker to the given presentation object.

8 . The method of claim 1 , wherein:

each frame bundle includes data specifying one or more user interface events that occurred during the presentation of one or more presentation objects; and

generating the playback data comprises recreating the one or more user interface events during presentation of the one or more presentation objects.

9 . A system, comprising:

a data store; and

one or more computers that interact with the data store and execute instructions that cause the one or more computers to perform operations comprising:

receiving a plurality of frame bundles for a user session with a native application, each frame bundle including data specifying:

for each of one or more points in time:

a presentation position of each presentation object used by the native application to generate a user interface of the native application at the point in time, each presentation object being an object that generates a visual representation of itself within a portion of the user interface; and

for one or more presentation objects, one or more drawing operations performed to generate the visual representation of the presentation object; and

generating, based on the data specified by the plurality of frame bundles, playback data that present visual changes of the user interface corresponding to the drawing operations performed to generate the visual representation of each presentation object.

10 . The system of claim 9 , wherein each presentation object comprises one of a view or a layer and wherein each frame bundle includes data specifying a hierarchical representation of each view or layer.

11 . The system of claim 9 , wherein generating the playback data comprises recreating the visual representation of each presentation object based on the drawing operations used to generate the presentation object.

12 . The system of claim 9 , wherein instrumentation code of the native application generates each frame bundle, including causing each layer of each presentation object to draw itself to an offscreen drawing context and storing data specifying drawing operations performed to draw each layer of each presentation object in the offscreen drawing context.

13 . The system of claim 12 , wherein the drawing context is a Portable Document Format (PDF) based drawing context and data specifying each drawing operation is stored in a PDF document.

14 . The system of claim 12 , wherein the instrumentation code causes, for each given presentation object, a drawing library of an operating system on which the native application runs to draw, to the offscreen drawing context, a start marker for the presentation object prior to the view being drawn to the offscreen drawing context and an end marker for the presentation object after the presentation object has been drawn to the offscreen drawing context.

15 . The system of claim 14 , wherein the instrumentation code assigns each drawing operation of the offscreen drawing context between the start marker and the end marker to the given presentation object.

16 . The system of claim 10 , wherein:

each frame bundle includes data specifying one or more user interface events that occurred during the presentation of one or more presentation objects; and

generating the playback data comprises recreating the one or more user interface events during presentation of the one or more presentation objects.

17 . A non-transitory computer readable medium storing instructions that upon execution by one or more computers cause the one or more computers to perform operations comprising:

receiving a plurality of frame bundles for a user session with a native application, each frame bundle including data specifying:

for each of one or more points in time:

a presentation position of each presentation object used by the native application to generate a user interface of the native application at the point in time, each presentation object being an object that generates a visual representation of itself within a portion of the user interface; and

for one or more presentation objects, one or more drawing operations performed to generate the visual representation of the presentation object; and

generating, based on the data specified by the plurality of frame bundles, playback data that present visual changes of the user interface corresponding to the drawing operations performed to generate the visual representation of each presentation object.

18 . The non-transitory computer readable medium of claim 17 , wherein each presentation object comprises one of a view or a layer and wherein each frame bundle includes data specifying a hierarchical representation of each view or layer.

19 . The non-transitory computer readable medium of claim 17 , wherein generating the playback data comprises recreating the visual representation of each presentation object based on the drawing operations used to generate the presentation object.

20 . The non-transitory computer readable medium of claim 17 , wherein instrumentation code of the native application generates each frame bundle, including causing each layer of each presentation object to draw itself to an offscreen drawing context and storing data specifying drawing operations performed to draw each layer of each presentation object in the offscreen drawing context.