Method and apparatus for implementing a search index generator
The disclosure provides examples of methods, apparatuses, systems, and computer program products for implementing a search indexer. One example method includes receiving, by index workflow circuitry, a search index update indicator, causing, by the index workflow circuitry and in response to receiving the search index update indicator, index generation circuitry to retrieve data sets from a plurality of data sources, calculating a checksum for each of the data sets, comparing the checksum for each of the data sets to a set of search index metadata, in response to determining that at least one checksum for at least one of the data sets does not match a metadata checksum from the set of search index metadata, generating a set of search index data from the data set corresponding to the at least one checksum, generating a search index using at least the set of search index data, and transmitting the search index to a search service for use in responding to at least one search query.
1. An apparatus for generating a search index, the apparatus comprising one or more processors and one or more memories for storing instructions that, with the one or more processors cause the apparatus to:
receive a search index update indicator, the search index update indicator generated in response to a change in one or more relationships between searchable items and search query criteria based at least in part on interactions with promotion content by consumer devices;
in response to receiving the search index update indicator, retrieve data sets from a plurality of data sources;
calculate a checksum for each of the data sets; and
transmit a search index to a service for use in responding to at least one search query, wherein the search index is generated using at least a set of search index data from a data set corresponding to at least one checksum.
2. The apparatus of claim 1 , wherein the search index update indicator is a periodic indicator.
3. The apparatus of claim 1 , wherein the one or more memories store instructions that, with the one or more processors, further cause the apparatus to: exclude a data set in response to determining that the at least one checksum matches at least one metadata checksum.
4. The apparatus of claim 1 , wherein the one or more memories store instructions that, with the one or more processors, further cause the apparatus to: split at least one data set into multiple shards.
5. The apparatus of claim 4 , wherein each of the multiple shards is processed concurrently.
6. The apparatus of claim 5 , wherein each of the multiple shards is processed as a separate MapReduce job.
7. The apparatus of claim 1 , wherein generating the search index comprises at least one MapReduce job.
8. A non-transitory computer readable storage medium for updating a search index, the non-transitory computer readable storage medium comprising instructions that, when executed by a processor, configure an apparatus to:
receive a search index update indicator, the search index update indicator generated in response to a change in one or more relationships between searchable items and search query criteria based at least in part on interactions with promotion content by consumer devices;
in response to receiving the search index update indicator, retrieve data sets from a plurality of data sources;
calculate a checksum for each of the data sets; and
transmit a search index to a service for use in responding to at least one search query, wherein the search index is generated using at least a set of search index data from a data set corresponding to at least one checksum.
9. The non-transitory computer readable storage medium of claim 8 , wherein the instructions further configure the apparatus to:
receive a set of search results from the search service;
determine a set of content corresponding to the set of search results; and
provide the set of content to at least one consumer.
10. The non-transitory computer readable storage medium of claim 8 , wherein at least two of the data sets are located at different data centers.
11. The non-transitory computer readable storage medium of claim 8 , wherein the search index comprises a set of JavaScript Object Notation pair values.
12. The non-transitory computer readable storage medium of claim 11 , wherein the pair values comprise a search term and a content identifier.
13. The non-transitory computer readable storage medium of claim 8 , wherein the search index update indicator is a new data set registering with the index workflow circuitry.
14. The non-transitory computer readable storage medium of claim 8 , wherein the search index update indicator is a periodic indicator.
15. The non-transitory computer readable storage medium of claim 8 , wherein the instructions further configure the apparatus to:
exclude a data set in response to determining that the at least one checksum matches at least one metadata checksum.
16. The non-transitory computer readable storage medium of claim 8 , wherein the instructions further configure the apparatus to:
split the at least one data set into multiple shards.
17. The non-transitory computer readable storage medium of claim 16 , wherein each of the multiple shards is processed concurrently.
18. The non-transitory computer readable storage medium of claim 17 , wherein each of the multiple shards is processed as a separate MapReduce job.
19. The non-transitory computer readable storage medium of claim 8 , wherein generating the search index comprises at least one MapReduce job.
20. A method for updating a search index, the method comprising:
receiving, by a processor, a search index update indicator, the search index update indicator generated in response to a change in one or more relationships between searchable items and search query criteria based at least in part on interactions with promotion content by consumer devices;
in response to receiving the search index update indicator, retrieving, by the processor and from a plurality of data sources, data sets;
calculating, by the processor, a checksum for each of the data sets; and
transmitting, by the processor, the search index to a service for use in responding to at least one search query, wherein the search index is generated using at least a set of search index data from a data set corresponding to at least one checksum.