Method and apparatus for implementing a search index generator
The disclosure provides examples of methods, apparatuses, systems, and computer program products for implementing a search indexer. One example method includes receiving, by index workflow circuitry, a search index update indicator, causing, by the index workflow circuitry and in response to receiving the search index update indicator, index generation circuitry to retrieve data sets from a plurality of data sources, calculating a checksum for each of the data sets, comparing the checksum for each of the data sets to a set of search index metadata, in response to determining that at least one checksum for at least one of the data sets does not match a metadata checksum from the set of search index metadata, generating a set of search index data from the data set corresponding to the at least one checksum, generating a search index using at least the set of search index data, and transmitting the search index to a search service for use in responding to at least one search query.
1. A method for updating a search index, the method comprising:
receiving, by a processor, a search index update indicator, the search index update indicator generated in response to a data source being registered to or deleted from a set of data sources associated with the search index;
calculating, by the processor, a checksum for each data set of the set of the data sources;
comparing, by the processor, the checksum for each data set of the set of data sources to search index metadata;
in response to determining that a selected checksum for a selected data set of the set of data sources does not match the search index metadata, generating new index update data;
merging the new index update data with the search index to create an updated search index; and
transmitting, by the processor, the updated search index to a service for use in responding to at least one search query.
2. The method of claim 1 , wherein the search index update indicator is a periodic indicator.
3. The method of claim 1 , wherein the new index update data is generated by a MapReduce process.
4. The method of claim 1 , wherein the merging the new index update data with the search index comprises splitting the new index update data into multiple shards.
5. The method of claim 4 , wherein each of the multiple shards is merged concurrently into the search index.
6. The method of claim 5 , wherein each of the multiple shards is processed as a separate MapReduce process.
7. The method of claim 4 , wherein the index update data is split into multiple shards based on promotion location.
8. An apparatus for updating a search index, the apparatus comprising one or more processors and one or more memories for storing instructions that, with the one or more processors cause the apparatus to:
receive a search index update indicator, the search index update indicator generated in response to a data source being registered to or deleted from a set of data sources associated with the search index;
calculate a checksum for each data set of the set of the data sources;
compare the checksum for each data set of the set of data sources to search index metadata;
in response to determining that a selected checksum for a selected data set of the set of data sources does not match the search index metadata, generate new index update data;
merge the new index update data with the search index to create an updated search index; and
transmit the updated search index to a service for use in responding to at least one search query.
9. The apparatus of claim 8 , wherein the search index update indicator is a periodic indicator.
10. The apparatus of claim 8 , wherein the new index update data is generated by a MapReduce process.
11. The apparatus of claim 8 , wherein the merging the new index update data with the search index comprises splitting the new index update data into multiple shards.
12. The apparatus of claim 11 , wherein each of the multiple shards is merged concurrently into the search index.
13. The apparatus of claim 12 , wherein each of the multiple shards is processed as a separate MapReduce process.
14. The apparatus of claim 11 , wherein the index update data is split into multiple shards based on promotion location.
15. A non-transitory computer readable storage medium for updating a search index, the non-transitory computer readable storage medium comprising instructions that, when executed by a processor, configure an apparatus to:
receive a search index update indicator, the search index update indicator generated in response to a data source being registered to or deleted from a set of data sources associated with the search index;
calculate a checksum for each data set of the set of the data sources;
compare the checksum for each data set of the set of data sources to search index metadata;
in response to determining that a selected checksum for a selected data set of the set of data sources does not match the search index metadata, generate new index update data;
merge the new index update data with the search index to create an updated search index; and
transmit the updated search index to a service for use in responding to at least one search query.
16. The non-transitory computer readable storage medium of claim 15 , wherein the search index update indicator is a periodic indicator.
17. The non-transitory computer readable storage medium of claim 15 , wherein the new index update data is generated by a MapReduce process.
18. The non-transitory computer readable storage medium of claim 15 , wherein the merging the new index update data with the search index comprises splitting the new index update data into multiple shards.
19. The non-transitory computer readable storage medium of claim 18 , wherein each of the multiple shards is merged concurrently into the search index.
20. The non-transitory computer readable storage medium of claim 18 , wherein each of the multiple shards is processed as a separate MapReduce process.