Systems and methods to build and utilize a search infrastructure
Methods and systems to build and utilize a search infrastructure are described. The system generates index information components in real-time based on a database that is time-stamped. The system updates index information at a plurality of query node servers based on the index information components. A query engine receives a search query from a client machine and identifies search results based on the query and the index information. The system communicates the search results, over the network, to the client machine.
1. A system comprising:
a map-reduce job module to generate index information components based on a database that is time-stamped, the index information components comprise a full-index and a plurality of mini-indexes, each of the plurality of mini-indexes having a mini-index identifier indicating a sequential order of the plurality of mini-indexes, the full-index comprises a snapshot of contents of the database, the plurality of mini-indexes respectively corresponding to a plurality of delta snapshots comprising updates to the database over time, each delta snapshot having a start-time and an end-time, wherein the end-time of a preceding delta snapshot corresponds to the start-time of a subsequent delta snapshot;
a query engine to update the index information components at a query node server based on the index information components, wherein the plurality of mini-indexes of the index information components is sequentially updated based on the mini-index identifier, the query engine to further receive a search query, and identify search results based on the search query and the index information components; and
a front-end server to communicate the search results.
2. The system of claim 1 , wherein the query engine updates the index information components at the query node server with the full-index and with the plurality of mini-indexes.
3. The system of claim 2 , wherein the plurality of mini-indexes includes a first mini-index and wherein the query engine updates the query node server based on the first mini-index.
4. The system of claim 3 , wherein the full-index is associated with full-index bill of material information and the first mini-index is associated with a first mini-index bill of material information, wherein the full-index bill of material information includes a full-index identifier and the first mini-index bill of material information includes a full-index identifier.
5. The system of claim 4 , wherein the query engine compares the full-index identifier in the full-index bill of material information with the full-index identifier in the mini-index bill of material information.
6. The system of claim 3 , wherein the plurality of mini-index indexes includes a second mini-index and a third mini-index.
7. The system of claim 6 , wherein the query engine blocks an update of the index information components in the query node server responsive to an identification of the second mini-index as arriving out of sequence.
8. The system of claim 6 , wherein the query engine blocks an update of the index information components in the query node server responsive to receipt of the second mini-index and identification of the first mini-index as not stored in mini-index storage information.
9. The system of claim 3 , wherein the query engine blocks an update of the index information components in the query node server responsive to an identification that the first mini-index is not compatible with the full-index that was utilized to update the index information components.
10. A method comprising:
generating index information components based on a database that is time-stamped, the index information components comprising a full-index and a plurality of mini-indexes, each of the plurality of mini-indexes having a mini-index identifier indicating a sequential order of the plurality of mini-indexes, the full-index comprises a snapshot of contents of the database, the plurality of mini-indexes respectively corresponding to a plurality of sequential delta snapshots comprising updates to the database, each sequential delta snapshot having a start-time and an end-time;
updating index information components at a query node server based on the index information components, wherein the plurality of mini-indexes of the index information components is sequentially updated based on the mini-index identifier;
receiving a search query and identifying search results based on the search query and the index information components; and
communicating the search results.
11. The method of claim 10 , wherein the updating the index information components includes updating index information at the query node server with the full-index and with the plurality of mini-indexes.
12. The method of claim 11 , wherein the plurality of mini-indexes includes a first mini-index and wherein the updating the index information components includes updating the query node server based on the first mini-index.
13. The method of claim 12 , wherein the full-index is associated with full-index bill of material information and the first mini-index is associated with a first mini-index bill of material information, wherein the full-index bill of material information includes a full-index identifier and the first mini-index bill of material information includes a full-index identifier.
14. The method of claim 13 , wherein the updating the index information comprises comparing the full-index identifier in the full-index bill of material information with the full-index identifier in the mini-index bill of material information.
15. The method of claim 12 , wherein the plurality of mini-index indexes includes a second mini-index and a third mini-index.
16. The method of claim 15 , wherein the updating the index information components comprises blocking the query node server from updating the index information components in the query node server responsive to identifying the second mini-index as arriving out of sequence.
17. The method of claim 15 , wherein the updating the index information components comprises blocking the query node server from updating the index information components in the query node server responsive to receiving the second mini-index and identifying the first mini-index as not being stored in mini-index storage information.
18. The method of claim 10 , wherein the updating the index information components comprises blocking the query node server from updating the index information components in the query node server responsive to identifying a mini-index is not compatible with the full-index that was utilized to update the index information components.
19. A non-transitory machine-readable medium comprising instructions, which when executed on a processor, cause the processor to perform operations comprising:
generating index information components based on a database that is time-stamped, the index information components comprising a full-index and a plurality of mini-indexes, each of the plurality of mini-indexes having a mini-index identifier indicating a sequential order of the plurality of mini-indexes, the full-index comprises a snapshot of contents of the database, the plurality of mini-indexes respectively corresponding to a plurality of delta snapshots comprising updates to the database, each delta snapshot associated with a period of time;
updating index information components at a query node server based on the index information components, wherein the plurality of mini-indexes of the index information components is sequentially updated based on the mini-index identifier;
receiving a search query and identifying search results based on the search query and the index information components; and
communicating the search results.
20. The non-transitory machine-readable medium of claim 19 , wherein the operations further comprise updating the index information components at the query node server with the full-index and with the plurality of mini-indexes.