Method and apparatus for implementing a search index generator
The disclosure provides examples of methods, apparatuses, systems, and computer program products for implementing a search indexer. One example method includes receiving, by index workflow circuitry, a search index update indicator, causing, by the index workflow circuitry and in response to receiving the search index update indicator, index generation circuitry to retrieve data sets from a plurality of data sources, calculating a checksum for each of the data sets, comparing the checksum for each of the data sets to a set of search index metadata, in response to determining that at least one checksum for at least one of the data sets does not match a metadata checksum from the set of search index metadata, generating a set of search index data from the data set corresponding to the at least one checksum, generating a search index using at least the set of search index data, and transmitting the search index to a search service for use in responding to at least one search query.
1 . A method for generating a search index, the method comprising:
receiving, by index workflow circuitry, a search index update indicator;
causing, by the index workflow circuitry and in response to receiving the search index update indicator, index generation circuitry to retrieve data sets from a plurality of data sources;
calculating a checksum for each of the data sets;
comparing the checksum for each of the data sets to a set of search index metadata;
in response to determining that at least one checksum for at least one of the data sets does not match a metadata checksum from the set of search index metadata, generating a set of search index data from the data set corresponding to the at least one checksum;
generating a search index using at least the set of search index data; and
transmitting the search index to a search service for use in responding to at least one search query.
2 . The method of claim 1 , further comprising:
receiving a set of search results from the search service;
determining a set of content corresponding to the set of search results; and
providing the set of content to at least one consumer.
3 . The method of claim 1 , wherein at least two of the data sets are located at different data centers.
4 . The method of claim 1 , wherein the search index comprises a set of JavaScript Object Notation pair values.
5 . The method of claim 4 , wherein the pair values comprise a search term and a content identifier.
6 . The method of claim 1 , wherein the search index update indicator is a new data set registering with the index workflow circuitry.
7 . The method of claim 1 , wherein the search index update indicator is a periodic indicator.
8 . The method of claim 1 , further comprising excluding a data set in response to determining that the at least one checksum matches at least one metadata checksum.
9 . The method of claim 1 , further comprising splitting the at least one data set into multiple shards.
10 . The method of claim 9 , wherein each of the multiple shards is processed concurrently.
11 . The method of claim 10 , wherein each of the multiple shards is processed as a separate MapReduce job.
12 . The method of claim 1 , wherein generating the search index comprises at least one MapReduce job.
13 . A non-transitory computer readable storage medium comprising instructions that, when executed by a processor, configure an apparatus to generate a search index, the instructions comprising:
program instructions for receiving a search index update indicator;
program instructions for causing, in response to receiving the search index update indicator, index generation circuitry to retrieve data sets from a plurality of data sources;
program instructions for calculating a checksum for each of the data sets;
program instructions for comparing the checksum for each of the data sets to a set of search index metadata;
program instructions for, in response to determining that at least one checksum for at least one of the data sets does not match a metadata checksum from the set of search index metadata, generating a set of search index data from the data set corresponding to the at least one checksum;
program instructions for generating a search index using at least the set of search index data; and
program instructions for transmitting the search index to a search service for use in responding to at least one search query.
14 . The computer readable storage medium of claim 13 , further comprising:
program instructions for receiving a set of search results from the search service;
program instructions for determining a set of content corresponding to the set of search results; and
program instructions for providing the set of content to at least one consumer.
15 . The computer readable storage medium of claim 13 , wherein at least two of the data sets are located at different data centers.
16 . The computer readable storage medium of claim 13 , wherein the search index comprises a set of JavaScript Object Notation pair values.
17 . The method of claim 16 , wherein the pair values comprise a search term and a content identifier.
18 . The computer readable storage medium of claim 13 , wherein the search index update indicator is a new data set registering with the index workflow circuitry.
19 . The computer readable storage medium of claim 13 , wherein the search index update indicator is a periodic indicator.
20 . An apparatus for generating a search index comprising index workflow circuitry, index generation circuitry, and communications circuitry, the apparatus configured to execute operations comprising:
receiving, by the index workflow circuitry, a search index update indicator,
causing, by the index workflow circuitry and in response to receiving the search index update indicator, index generation circuitry to retrieve data sets from a plurality of data sources,
calculating, by the index workflow circuitry, a checksum for each of the data sets,
comparing, by the index workflow circuitry, the checksum for each of the data sets to a set of search index metadata;
generating, by the index generation circuitry and in response to determining that at least one checksum for at least one of the data sets does not match a metadata checksum from the set of search index metadata, a set of search index data from the data set corresponding to the at least one checksum;
generating, by the index generation circuitry, a search index using at least the set of search index data; and
transmitting, by the communications circuitry, the search index to a search service for use in responding to at least one search query.