Data indexing for search tool
Methods and systems for indexing data for a search tool are disclosed, wherein the method is performed using one or more processors. The methods and systems may comprise an operation of providing an indexing application programming interface (API) for implementation in relation to one or more data resources of a data processing platform requiring data of the data resource to be indexed for searching using a search tool of the data processing platform. The interface may be configured, when implemented in relation to a data resource, to send data indicative of one or more definitions of data to be crawled from said data resource to an indexing system of the data processing platform, and to generate an event queue indicative of one or more events associated with said defined data that have occurred in relation to the data resource.
1. A method of indexing data for a search service, wherein the method is performed using one or more processors, the method comprising:
determining a schema of a data resource to be crawled;
determining that the schema has changed;
following the change of the schema, determining whether the changed schema is available at a search component of a platform that crawls the data resource;
in response to determining that the changed schema is available at the compatible with a search component of the platform, retrieving a first application programming interface (API) appropriate for the changed schema, wherein the first API interfaces between the data resource and the search component;
in response to determining that the changed schema is unavailable at the search component of the platform:
creating, in the search component, a schema mapping corresponding to the data resource; and
retrieving a second API appropriate for the schema mapping, wherein the second API interfaces between the data resource and the search component;
implementing the search component to determine one or more events associated with the data resource;
crawling, using the search component, the data resource based on the determined one or more events to obtain data; and
indexing the obtained data associated with said one or more events for access by the search component.
2. The method of claim 1 , further comprising detecting a change in an access control privilege of the data resource; and the determining of the first schema is in response to detecting a change in the access control privilege.
3. The method of claim 2 , wherein the change in the access control privilege comprises changing a searching privilege of the data resource from unsearchable to searchable.
4. The method of claim 1 , further comprising:
prior to determining that the schema has changed:
retrieving a third API appropriate for the schema the search component;
implementing the search component to determine one or more second events associated with the data resource;
crawling, using the search component, the data resource based on the determined one or more second events to obtain second data; and
indexing the obtained second data associated with said one or more second events for access by the search component.
5. The method of claim 1 , wherein the one or more events have occurred since a previous querying operation and are determined based on an offset indicative of the previous querying operation; and the crawling the data resource is based only on the one or more events that have occurred since a previous querying operation.
6. The method of claim 1 , further comprising:
presenting the obtained data and metadata associated with the obtained data; and
selectively redacting the presentation of the obtained data based on a role of a viewer of the obtained data.
7. The method of claim 1 , wherein the one or more events comprise a change in, addition or removal of metadata.
8. The method of claim 1 , wherein the one or more events comprises a change in a file type of the data resource.
9. The method of claim 1 , wherein the one or more events comprises an additional schema in the data resource.
10. The method of claim 1 , further comprising populating an interface that includes details regarding the data resource, metadata, the schema or the change in the schema, access control properties, and indexing attributes.
11. A system comprising:
one or more physical processors;
a memory storing instructions that, when executed by the one or more physical processors, cause the system to perform:
determining a schema of a data resource to be crawled;
determining that the schema has changed;
following the change of the schema, determining whether the changed schema is available at a search component of a platform that crawls the data resource;
in response to determining that the changed schema is available at the search component of the platform, retrieving a first application programming interface (API) appropriate for the changed schema, wherein the first API interfaces between the data resource and the search component;
in response to determining that the changed schema is unavailable at the search component of the platform:
creating, in the search component, a schema mapping corresponding to the data resource; and
retrieving a second API appropriate for the schema mapping, wherein the second API interfaces between the data resource and the search component;
implementing the search component to determine one or more events associated with the data resource;
crawling, using the search component, the data resource based on the determined one or more events to obtain data; and
indexing the obtained data associated with said one or more events for access by the search component.
12. The system of claim 11 , wherein the instructions further cause the system to perform:
detecting a change in an access control privilege of the data resource; and the determining of the first schema is in response to detecting a change in the access control privilege.
13. The system of claim 12 , wherein the change in the access control privilege comprises changing a searching privilege of the data resource from unsearchable to searchable.
14. The system of claim 11 , wherein the instructions further cause the system to perform:
prior to determining that the schema has changed:
retrieving a third API appropriate for the schema to interface with the search component;
implementing the search component to determine one or more second events associated with the data resource;
crawling, using the search component, the data resource based on the determined one or more second events to obtain second data; and
indexing the obtained second data associated with said one or more second events for access by the search component.
15. The system of claim 11 , wherein the one or more events have occurred since a previous querying operation and are determined based on an offset indicative of the previous querying operation; and the crawling the data resource is based only on the one or more events that have occurred since a previous querying operation.
16. The system of claim 11 , wherein the instructions further cause the system to perform:
presenting the obtained data and metadata associated with the obtained data; and
selectively redacting the presentation of the obtained data based on a role of a viewer of the obtained data.
17. The system of claim 11 , wherein the one or more events comprise a change in, addition or removal of metadata.
18. The system of claim 11 , wherein the one or more events comprises a change in a file type of the data resource.
19. The system of claim 11 , wherein the one or more events comprises an additional schema in the data resource.
20. The system of claim 11 , wherein the instructions further cause the system to perform:
populating an interface that includes details regarding the data resource, metadata, the schema or the change in the schema, access control properties, and indexing attributes.