Data indexing for search tool

1. A method of indexing data for a search service, wherein the method is performed using one or more processors, the method comprising:

providing an indexing application programming interface (API) for implementation in relation to one or more data resources of a data processing platform requiring data of the data resource to be indexed for searching using a search service of the data processing platform, the indexing API being configured, when implemented in relation to a data resource, to send data indicative of one or more definitions of data to be crawled from said data resource to an indexing system of the data processing platform, and to generate an event queue indicative of one or more events associated with said defined data that have occurred in relation to the data resource, wherein the providing of the indexing API comprises:

determining whether a schema corresponding to the data resource is available at the search service;

in response to determining that the schema is available:

retrieving the schema;

determining a first API compatible with the schema; and

retrieving the first API compatible with the schema;

in response to determining that the schema is unavailable:

creating a schema mapping, in the search service, corresponding to the data resource; and

retrieving a second API compatible with the schema mapping in the search service;

identifying one or more data resources that have implemented the indexing API;

querying the one or more identified data resources to determine one or more events indicated in respective buffers of the one or more identified data resources;

crawling the one or more identified data resources based on the determined one or more events to receive data for indexing; and

indexing the received data associated with said one or more events for access by the search service.

2. The method of claim 1 , wherein the indexing system is a centralised indexing system of a cloud-based data processing platform comprising one or more servers running the one or more data resources.

3. The method of claim 1 , wherein the indexing API is a web API.

4. The method of claim 1 , wherein the indexing API, when implemented by a data resource, is configured to cause creation of the event queue locally at said data resource.

5. The method of claim 1 , wherein querying the one or more identified data resources comprises identifying one or more new events in the one or more event queues which have occurred since a previous querying operation, and crawling the respective one or more data resources based only on the one or more new events.

6. The method of claim 5 , wherein the indexing API, when implemented by said one or more identified data resources, is further configured to cause the respective one or more event queues to empty subsequent to the crawling operation.

7. The method of claim 5 , wherein the indexing API, when implemented by said one or more data sources, is further configured to cause generation of an offset indicative of when the respective one or more event queues were previously queried.

8. The method of claim 1 , wherein the one or more event queues store event data indicative of a change of data.

9. The method of claim 8 , wherein the event data is indicative of one or more of new data, deleted data and modified data.

10. The method of claim 1 , wherein the one or more definitions of the data to be crawled permits interpretation of each event in the event queue such that data associated with the interpreted event is indexed based on the interpretation.

11. The method of claim 1 , wherein the method further comprises providing the search service specific to the data processing platform, the search service comprising a user interface for receiving a search query and for returning search results based on the search query and the data that has been indexed by the indexing operation.

12. The method of claim 11 , wherein the indexing API is operable, when implemented by the one or more data resources, to send a set of search restrictions defining limitations on the search results that can be returned to users responsive to a search query.

13. The method of claim 11 , wherein the search restrictions are based on an identifier of a user or user role associated with the search query.

14. The method of claim 13 , wherein the search restrictions define that search results relating to second data resources belonging to a particular organisation are only returned to users having a predetermined association with the particular organisation.

15. The method of claim 13 , wherein the search restrictions define one or more of: (i) a subset of the indexed data returned to the user in the search results (ii) whether the search results are accessible and (iii) the format of the search results.

16. The method of claim 1 , wherein the indexing API is operable, when implemented by the one or more data resources, to send a set of indexing preferences to the indexing system indicative of user-defined data sub-types to be crawled and indexed.

17. The method of claim 1 , wherein the one or more data resources comprise one or more of executable applications, data analysis applications, databases, metadata, documents.

18. The method of claim 1 , wherein the one or more events comprise a change in the schema of the data resource to a changed schema; and the providing of the indexing API comprises:

determining whether the changed schema is available at the search service;

in response to determining that the changed schema is available:

retrieving the changed schema;

determining a third API compatible with the changed schema; and

retrieving the third API compatible with the changed schema;

in response to determining that the changed schema is unavailable:

creating a second schema mapping, in the search service, corresponding to the data resource; and

retrieving a fourth API compatible with the second schema mapping in the search service.

19. A non-transitory computer-readable storage medium including instructions that, when executed by at least one processor of a computing system, cause the computing system to perform a method comprising:

providing an indexing application programming interface (API) for implementation in relation to one or more data resources of a data processing platform requiring data of the data resource to be indexed for searching using a search service of the data processing platform, the indexing API being configured, when implemented in relation to a data resource, to send data indicative of one or more definitions of data to be crawled from said data resource to an indexing system of the data processing platform, and to generate an event queue indicative of one or more events associated with said defined data that have occurred in relation to the data resource, wherein the providing of the indexing API comprises:

determining whether a schema corresponding to the data resource is available at the search service;

in response to determining that the schema is available:

retrieving the schema;

determining a first API compatible with the schema; and

retrieving the first API compatible with the schema;

in response to determining that the schema is unavailable:

creating a schema mapping, in the search service, corresponding to the data resource; and

retrieving a second API compatible with the schema mapping in the search service;

identifying one or more data resources that have implemented the indexing API;

querying the one or more identified data resources to determine one or more events indicated in respective buffers of the one or more identified data resources;

crawling the one or more identified data resources based on the determined one or more events to receive data for indexing; and

indexing the received data associated with said one or more events for access by the search service.

20. A system for indexing data, the system comprising:

one or more physical processors;

a memory storing instructions that, when executed by the one or more physical processors, cause the system to perform:

providing an indexing application programming interface (API) for implementation in relation to one or more data resources of a data processing platform requiring data of the data resource to be indexed for searching using a search service of the data processing platform, the indexing API being configured, when implemented in relation to a data resource, to send data indicative of one or more definitions of data to be crawled from said data resource to an indexing system of the data processing platform, and to generate an event queue indicative of one or more events associated with said defined data that have occurred in relation to the data resource, wherein the providing of the indexing API comprises:

determining whether a schema corresponding to the data resource is available at the search service;

in response to determining that the schema is available:

retrieving the schema;

determining a first API compatible with the schema; and

retrieving the first API compatible with the schema;

in response to determining that the schema is unavailable:

creating a schema mapping, in the search service, corresponding to the data resource; and

retrieving a second API compatible with the schema mapping in the search service;

identifying one or more data resources that have implemented the indexing API;

querying the one or more identified data resources to determine one or more events querying the one or more identified data resources to determine one or more events indicated in respective buffers of the one or more identified data resources;

crawling the one or more identified data resources based on the determined one or more events to receive data for indexing; and

indexing the received data associated with said one or more events for access by the search service.