Database live reindex
Systems and methods for live reindexing include receiving, at a dispatch server, a request to reindex a database, wherein, at the time the request to reindex is received, the dispatch server is configured to send queries for information in the database to a first index server, the first index server comprising a first index of information in the database. The systems and methods further include, responsive to receiving the request to reindex and while the database is live, generating a second index of the information in the database, storing the second index at a second index server, and configuring the dispatch server to use the second index server as a live index server and to no longer use the first index server as a live index server.
1. A computer-implemented method comprising:
configuring a dispatch server to use a first index server as a live index server, the first index server comprising a first index of information that is stored in a database and responding to queries on the database from the dispatch server;
receiving a request to reindex information in the database;
based on the receiving the request to reindex information in the database, initiating a generation of a second shadow index of information in the database while the dispatch server continues to use the first index server as a live index server;
while the dispatch server is quiesced without receiving queries, completing the generation of the second shadow index; and
based on the completing the generation of the second shadow index, configuring the dispatch server to use a second index server, that stores the second shadow index as a live index server.
2. The computer-implemented method of claim 1 , further comprising:
generating the second shadow index of information based on iterating through information in the database.
3. The computer-implemented method of claim 1 , wherein the completing the generation of the second shadow index is based on periodically iterating through information in the database until the second shadow index is up to date.
4. The computer-implemented method of claim 1 , wherein the first index server is no longer used as a live index server while the second shadow index is used as a live index server.
5. The computer-implemented method of claim 1 , wherein the request is associated with a new ontology for information in the database.
6. The computer-implemented method of claim 1 , further comprising:
prior to the request, the first index server is composed of a first number of shards; and
the request is associated with a change in the first number of shards of which the first index server to a second number of shards that is different than the first number of shards.
7. The computer-implemented method of claim 1 , wherein the second index server operates on one or more devices on which the first index server operates.
8. The computer-implemented method of claim 7 , further comprising:
writing a plurality of actions for generating the second shadow index to a commit log; and
performing the plurality of actions in the commit log when configuring the dispatch server to use the second index server as a live index server.
9. The computer-implemented method of claim 1 , further comprising:
generating the second shadow index based on a maximum batch size and a maximum thread count.
10. One or more non-transitory computer-readable media storing one or more programs, the one or more programs comprising instructions for:
configuring a dispatch server to use a first index server as a live index server, the first index server comprising a first index of information that is stored in a database and responding to queries on the database from the dispatch server;
receiving a request to reindex information in the database;
based on the receiving the request to reindex information in the database, initiating a generation of a second shadow index of information in the database while the dispatch server continues to use the first index server as a live index server;
while the dispatch server is quiesced without receiving queries, completing the generation of the second shadow index; and
based on the completing the generation of the second shadow index, configuring the dispatch server to use a second index server that stores the second shadow index as a live index server.
11. The one or more non-transitory computer-readable media of claim 10 , the one or more programs further comprising instructions for:
generating the second shadow index of information based on iterating through information in the database.
12. The one or more non-transitory computer-readable media of claim 10 , wherein the completing the generation of the second shadow index is based on periodically iterating through information in the database until the second shadow index is up to date.
13. The one or more non-transitory computer-readable media of claim 10 , wherein the first index server is no longer used as a live index server while the second shadow index is used as a live index server.
14. The one or more non-transitory computer-readable media of claim 10 , wherein the request is associated with a new ontology for information in the database.
15. The one or more non-transitory computer-readable media of claim 10 , the one or more programs further comprising instructions for:
prior to the request, the first index server is composed of a first number of shards; and
the request is associated with a change in the first number of shards of which the first index server to a second number of shards that is different than the first number of shards.
16. The one or more non-transitory computer-readable media of claim 10 , wherein the second index server operates on one or more devices on which the first index server operates.
17. The one or more non-transitory computer-readable media of claim 16 , wherein the one or more programs further comprising instructions for:
writing a plurality of actions for generating the second shadow index to a commit log, wherein the plurality of actions is not performed; and
performing the plurality of actions in the commit log when configuring the dispatch server to use the second index server as the live index server.
18. The one or more non-transitory computer-readable media of claim 10 , the one or more programs further comprising instructions for:
generating the second shadow index based on a maximum batch size and a maximum thread count.
19. A system comprising:
one or more processors;
a memory storing one or more programs configured for execution by the one or more processors, the one or more programs comprising instructions for:
configuring a dispatch server to use a first index server as a live index server, the first index server comprising a first index of information that is stored in a database and responding to queries on the database from the dispatch server;
receiving a request to reindex information in the database;
based on the receiving the request to reindex information in the database, initiating a generation of a second shadow index of information in the database while the dispatch server continues to use the first index server as a live index server;
while the dispatch server is quiesced without receiving queries, completing the generation of the second shadow index; and
based on the completing the generation of the second shadow index, configuring the dispatch server to use a second index server that stores the second shadow index as a live index server.
20. The system of claim 19 , the one or more programs further comprising instructions for:
generating the second shadow index of information based on iterating through information in the database.