Multi-user search system with methodology for bypassing instant indexing

1. A computing system comprising:

one or more processors;

storage media;

a token store storing a set of tokens for a current version of a document;

a tokenizer server configured to tokenize a new version of the document and to generate a set of tokens for the new version of the document;

an instant indexer server configured: (a) to determine tokens to index the document by, based on identified differences between the set of tokens for the new version of the document and the set of tokens for the current version of the document, (b) to determine whether the new version of the document is a bypass document version or non-bypass document version, and (c) to generate an index mutation for the new version of the document, the generated index mutation including: (c1) a specification of whether the generated index mutation is a bypass index mutation or a non-bypass index mutation based on whether the new version of the document is a bypass document version or non-bypass document version and (c2) one or more tokens reflecting the identified differences between the set of tokens for the new version of the document and the set of tokens for the current version of the document; and

an index mutation server configured: (d) if the generated index mutation is a non-bypass index mutation, to provide the generated index mutation to an index server that is configured to update, based on the provided index mutation, an index stored at the index server, and (e) if the generated index mutation is a bypass index mutation, not to provide the generated index mutation to an index server.

2. The system of claim 1 , wherein the instant indexer server is further configured to:

store the generated index mutation in an index mutation journal, if the generated index mutation is a non-bypass index mutation; and

not store the generated index mutation in the index mutation journal, if the generated index mutation is a bypass index mutation.

3. The system of claim 1 , wherein the instant indexer server is further configured to determine tokens to no longer use to index the document based on identified differences between the set of tokens for the new version of the document and the set of tokens for the current version of the document.

4. The system of claim 1 , wherein the index mutation server is further configured to receive a network request from the index server and to provide the generated index mutation from an index mutation journal to the index server in response to the network request, if the generated index mutation is a non-bypass index mutation.

5. The system of claim 1 , wherein the instant indexer server is further configured to append the generated index mutation to an index mutation journal, if the generated index mutation is a non-bypass index mutation.

6. The system of claim 1 , wherein the tokenizer is further configured to send a message to the instant indexer server responsive to tokenizing and generating the set of tokens for the new version of the document, the message comprising an identifier of the document, an identifier of the new version of the document, and a specification of whether the new version of the document is a bypass document version or a non-bypass document version.

7. The system of claim 1 , wherein the instant indexer server is further configured to determine a timestamp to associate with the generated index mutation in an index mutation journal, in the generated index mutation is a non-bypass index mutation.

8. The system of claim 1 , wherein the tokenizer is further configured to store the generated set of tokens for the new version of the document in the token store.

9. The system of claim 1 , wherein the instant indexer server is further configured to store the generated index mutation in an index mutation journal, if the generated index mutation is a non-bypass index mutation, and to store the set of tokens for the new version of the document in the token store responsive to storing the index mutation in the index mutation journal.

10. The system of claim 1 , wherein the index server is configured to update an in-memory delta index responsive to receiving the generated index mutation from the index mutation server.

11. The system of claim 1 , wherein the generated index mutation further comprises an identifier of the document.

12. A method comprising:

storing, in a token store, a set of tokens for a current version of a document;

tokenizing a new version of the document and generating a set of tokens for the new version of the document;

determining tokens to index the document by, based on identified differences between the set of tokens for the new version of the document and the set of tokens for the current version of the document;

determining whether the new version of the document is a bypass document version or non-bypass document version;

for the new version of the document, generating an index mutation including: (a) a specification of whether the generated index mutation is a bypass index mutation or a non-bypass index mutation based on whether the new version of the document is a bypass document version or non-bypass document version and (b) one or more tokens reflecting the identified differences between the set of tokens for the new version of the document and the set of tokens for the current version of the document;

if the generated index mutation is a non-bypass index mutation, providing the generated index mutation to an index server that is configured to update, based on the provided index mutation, an index stored at the index server; and

if the generated index mutation is a bypass index mutation, not providing the generated index mutation to the index server server;

wherein the method is performed by one or more computing devices.

13. The method of claim 12 , further comprising:

storing the generated index mutation in an index mutation journal, if the generated index mutation is a non-bypass index mutation; and

not storing the generated index mutation in the index mutation journal, if the generated index mutation is a bypass index mutation.

14. The method of claim 12 , further comprising:

determining tokens to no longer use to index the document based on identified differences between the set of tokens for the new version of the document and the set of tokens for the current version of the document.

15. The method of claim 12 , further comprising:

receiving a network request from the index server; and

providing the generated index mutation from an index mutation journal to the index server in response to the network request, if the generated index mutation is a non-bypass index mutation.

16. The method of claim 12 , further comprising:

appending the generated index mutation to an index mutation journal, if the generated index mutation is a non-bypass index mutation.

17. The method of claim 12 , further comprising:

adding a message to a message queue responsive to tokenizing and generating the set of tokens for the new version of the document, the message including an identifier of the document, an identifier of the new version of the document, and a specification of whether the new version of the document is a bypass document version or a non-bypass document version.

18. The method of claim 12 , Further comprising:

determining a time stamp to associate with the generated index mutation in an index mutation journal, if the generated index mutation is a non-bypass index mutation.

19. The method of claim 12 , further comprising:

storing the generated set of tokens for the new version of the document in the token store.

20. The method of claim 12 , further comprising:

storing the set of tokens for the new version of the document in the token store responsive to storing the index mutation in an index mutation journal, if the generated index mutation is a non-bypass index mutation.

21. The method of claim 12 , further comprising:

the index server updating an in-memory delta index responsive to obtaining the generated index mutation.

22. The method of claim 12 , wherein the generated index mutation further comprises an identifier of the document.

23. The method of claim 12 , further comprising:

storing, in an index mutation journal, the generated index mutation including a specification that the generated index mutation is a non-bypass index mutation, if the generated index mutation is a non-bypass index mutation; and

storing, in the index mutation journal, the generated index mutation including a specification that the generated index mutation is a bypass index mutation, if the generated index mutation is a bypass index mutation.

24. The method of claim 12 , further comprising:

determining, prior to providing the generated index mutation to the index server, that the generated index mutation is a non-bypass index mutation, if the generated index mutation includes a specification that the generated index mutation is a non-bypass index mutation; and

determining, prior to not providing the generated index mutation to the index server, that the generated index mutation is a bypass index mutation, if the generated index mutation includes a specification that the generated index mutation is a bypass index mutation.

25. The system of claim 1 , wherein the instant indexer server is further configured to:

store, in an index mutation journal, the generated index mutation including a specification that the generated index mutation is a non-bypass index mutation, if the generated index mutation is a non-bypass index mutation; and

store, in the index mutation journal, the generated index mutation including a specification that the generated index mutation is a bypass index mutation, if the generated index mutation is a bypass index mutation.

26. The system of claim 1 , wherein the index mutation server is further configured to:

determine, prior to providing the generated index mutation to the index server, that the generated index mutation is a non-bypass index mutation, if the generated index mutation includes a specification that the generated index mutation is a non-bypass index mutation; and

determine, prior to not providing the generated index mutation to the index server, that the generated index mutation is a bypass index mutation, if the generated index mutation includes a specification that the generated index mutation is a bypass index mutation.