Indexing access limited native applications

1 . A computer-implemented method performed by data processing apparatus comprising one or more computers in data communication, the method comprising:

receiving, from an authentication server, a granted consent permitting access to an environment instance of a native application;

instantiating, for an automated crawler, the environment instance of the native application with a set of robot account credentials for an access account, wherein the robot account credentials are generated for the access account on behalf of the automated crawler;

accessing the environment instance of the native application based at least in part on the native application being automatically signed in using the previously granted consent;

using the automated crawler to extract a plurality of images and environment instance data describing content of the environment instance, wherein the plurality of images depict a respective plurality of views of a rendered environment of the native application, wherein the environment instance data comprises a plurality of sets of textual data that are displayable on a user device when the user device displays the environment instance, and wherein each of the plurality of sets of textual data is associated with a corresponding view of the plurality of views depicted by the plurality of images; and

indexing the plurality of images and the environment instance data for the native application in an index that is searchable by a search engine.

2 . The method of claim 1 , wherein a first set of textual data of the plurality of sets of textual data is associated with a first view of the plurality of views, wherein the first set of textual data comprises a TextView object, and wherein the first view comprises an Image View object.

3 . The method of claim 2 , wherein the TextView object is designated as being invisible to users.

4 . The method of claim 1 , wherein, prior to instantiating the environment instance of the native application with the set of robot account credentials, the method comprises:

determining that the access account for indexing environment instances generated by the native application does not exist; and

generating the access account for indexing environment instances generated by the native application, the access account including the set of robot credentials.

5 . The method of claim 4 , wherein generating the access account for indexing environment instances generated by the native application comprises:

generating (i) a name, (ii) a password, and (iii) other account information corresponding with account information expected for particular users providing search queries.

6 . The method of claim 1 , wherein the environment instance comprises a multi-dimensional environment, and wherein the plurality of images are generated from a point of view of an artificial intelligence bot within the multi-dimensional environment.

7 . The method of claim 1 , wherein instantiating the environment instance of the native application with the set of robot account credentials comprises:

instantiating a virtual machine emulating an operating system of the user device with the set of robot account credentials; and

instantiating the environment instance of the native application within the virtual machine.

8 . The method of claim 7 , wherein instantiating a virtual machine emulating an operating system of a user device with the set of account credentials comprises:

instantiating the virtual machine emulating the operating system of the user device where the operating system indicates that the access account is signed in using the set of robot account credentials.

9 . The method of claim 1 , wherein the rendered environment comprises a three-dimensional openGL environment, and wherein using the automated crawler to extract the plurality of images comprises:

executing, by the automated crawler, a script comprising a predefined set of directional navigations; and

capturing the plurality of images while executing the script comprising the set of predefined directional navigations.

10 . The method of claim 1 , wherein the access account comprises a robot account, and

wherein instantiating, for the automated crawler, the environment instance of the native application comprises:

triggering the robot account to be generated; and

instantiating, for the automated crawler and in response to the triggering of the robot account, the environment instance of the native application with the set of robot account credentials for the robot account.

11 . The method of claim 1 , wherein the previously granted consent is established via an account authentication service for which an operating system indicates the access account is already signed in.

12 . The method of claim 1 , wherein the automated crawler is connected to the native application upon launch of the native application within a virtual machine emulating an operating system that indicates the access account is already signed in.

13 . A system, comprising: a data processing apparatus; and

software stored in non-transitory computer readable storage medium storing instructions executable by the data processing apparatus and that upon such execution cause the data processing apparatus to perform operations comprising:

receiving, from an authentication server, a granted consent permitting access to an environment instance of a native application;

instantiating, for an automated crawler, the environment instance of the native application with a set of robot account credentials for an access account, wherein the robot account credentials are generated for the access account on behalf of the automated crawler;

accessing, based at least in part on the native application being automatically signed in using the previously granted consent, the environment instance of the native application using the automated crawler to extract a plurality of images and environment instance data describing content of the environment instance, wherein the plurality of images depict a respective plurality of views of a rendered environment of the native application, wherein the environment instance data comprises a plurality of sets of textual data that are displayable on a user device when the user device displays the environment instance, and wherein each of the plurality of sets of textual data is associated with a corresponding view of the plurality of views depicted by the plurality of images; and

indexing the plurality of images and the environment instance data for the native application in an index that is searchable by a search engine.

14 . The system of claim 13 , wherein a first set of textual data of the plurality of sets of textual data is associated with a first view of the plurality of views, wherein the first set of textual data comprises a TextView object, and wherein the first view comprises an Image View object.

15 . The system of claim 14 , wherein the TextView object is designated as being invisible to users.

16 . The system of claim 13 , wherein, prior to instantiating the environment instance of the native application with the set of robot account credentials, the operations comprise:

determining that the access account for indexing environment instances generated by the native application does not exist; and

generating the access account for indexing environment instances generated by the native application, the access account including the set of robot credentials.

17 . The system of claim 16 , wherein generating the access account for indexing environment instances generated by the native application comprises:

generating (i) a name, (ii) a password, and (iii) other account information corresponding with account information expected for particular users providing search queries.

18 . The system of claim 13 , wherein

the environment instance comprises a multi-dimensional environment, and wherein the plurality of images are generated from a point of view of an artificial intelligence bot within the multi-dimensional environment.

19 . The system of claim 13 , wherein instantiating the environment instance of the native application with the set of robot account credentials comprises:

instantiating a virtual machine emulating an operating system of the user device with the set of robot account credentials; and

instantiating the environment instance of the native application within the virtual machine.

20 . A non-transitory computer readable memory storage system storing instructions executable by a data processing apparatus and that upon such execution cause the data processing apparatus to perform operations comprising:

receiving, from an authentication server, a granted consent permitting access to an environment instance of a native application;

instantiating, for an automated crawler, the environment instance of the native application with a set of robot account credentials for an access account, wherein the robot account credentials are generated for the access account on behalf of the automated crawler;

accessing the environment instance of the native application based at least in part on the native application being automatically signed in using the previously granted consent;

using the automated crawler to extract a plurality of images and environment instance data describing content of the environment instance, wherein the plurality of images depict a respective plurality of views of a rendered environment of the native application, wherein the environment instance data comprises a plurality of sets of textual data that are displayable on a user device when the user device displays the environment instance, and wherein each of the plurality of sets of textual data is associated with a corresponding view of the plurality of views depicted by the plurality of images; and

indexing the plurality of images and the environment instance data for the native application in an index that is searchable by a search engine.