Generating content snippets using a tokenspace repository

1. A query processing method, performed by a search engine server system, comprising:

receiving, from a client system distinct from the search engine server system, a search query having one or more query terms;

in response to receiving the search query:

identifying a set of search result documents in accordance with the search query;

generating a content snippet corresponding to content in a respective document of the identified set of search result documents, the content snippet associated with at least one query term of the one or more query terms in the search query; and

returning to the client system a response to the search query, the response including information identifying at least the respective search result document and including the content snippet of the respective search result document;

wherein generating the content snippet comprises:

receiving a set of first token identifiers, comprising compressed content of the respective search result document, from a compressed document repository;

performing a first decompression operation by applying one or more first mappings to the set of first token identifiers to provide a set of second token identifiers; and

performing a second decompression operation by applying a second mapping to the set of second token identifiers to recover uncompressed content comprising a portion of the respective document;

wherein the first decompression operation produces a respective second token identifier for each first token identifier, the set of second token identifiers has a same number of distinct token identifiers as the first set of token identifiers, and the second decompression operation produces a respective uncompressed content item for each second token identifier; and

wherein each first token identifier comprises an M bit integer value, and each second token identifier comprises an N bit integer value, N and M are positive integers and N is greater than M.

2. The method of claim 1 , further comprising deriving relevancy scores for at least a subset of the identified set of search result documents, and selecting the respective document based on having a high relevancy score.

3. The method of claim 2 , wherein the relevancy scores are based, at least in part on weights assigned to the query terms.

4. The method of claim 1 , wherein identifying a set of search result documents in accordance with the search query comprises:

identifying a preliminary set of documents in accordance with the search query;

generating a respective preliminary content snippet for a plurality of documents in the preliminary set of documents, each respective preliminary content snippet corresponding to content in the respective preliminary document, and associated with at least one query term of the one or more query terms in the search query;

producing a set of query expansion terms from the preliminary content snippets;

combining the one or more query terms with the query expansion terms to form an expanded search query; and

identifying a set of search result documents in accordance with the expanded search query.

5. A computer system for processing queries, comprising:

memory;

one or more processors; and

one or more programs stored in the memory and configured for execution by the one or more processors, the one or more programs including:

instructions for receiving, from a client system distinct from the search engine server system, a search query having one or more query terms;

instructions for identifying a set of search result documents in accordance with the search query;

instructions for generating a content snippet corresponding to content in a respective document of the identified set of search result documents, the content snippet associated with at least one query term of the one or more query terms in the search query; and

instructions for returning a response to the client system that includes information identifying at least the respective search result document and including the content snippet of the respective search result document;

wherein the instructions for generating the content snippet comprises:

instructions for receiving a set of first token identifiers, comprising compressed content of the respective search result document, from a compressed document repository;

instructions for performing a first decompression operation by applying one or more first mappings to the set of first token identifiers to provide a set of second token identifiers; and

instructions for performing a second decompression operation by applying a second mapping to the set of second token identifiers to recover uncompressed content comprising a portion of the respective document

wherein the first decompression operation produces a respective second token identifier for each first token identifier, the set of second token identifiers has a same number of distinct token identifiers as the first set of token identifiers, and the second decompression operation produces a respective uncompressed content item for each second token identifiers; and

wherein each first token identifier comprises an M bit integer value, and each second token identifier comprises an N bit integer value, N and M are positive integers and N is greater than M.

6. The computer system of claim 5 , further comprising instructions for deriving relevancy scores for at least a subset of the identified set of search result documents, and instructions for selecting the respective document based on having a high relevancy score.

7. The computer system of claim 6 , wherein the relevancy scores are based, at least in part on weights assigned to the query terms.

8. The computer system of claim 5 , wherein the instructions for identifying a set of search result documents in accordance with the search query comprises:

instructions for identifying a preliminary set of documents in accordance with the search query;

instructions for generating a respective preliminary content snippet for a plurality of documents in the preliminary set of documents, each respective preliminary content snippet corresponding to content in the respective preliminary document, and associated with at least one query term of the one or more query terms in the search query;

instructions for producing a set of query expansion terms from the preliminary content snippets;

instructions for combining the one or more query terms with the query expansion terms to form an expanded search query; and

instructions for identifying a set of search result documents in accordance with the expanded search query.

9. A non-transitory computer readable storage medium storing one or more programs to be executed by a computer system, the one or more programs comprising:

instructions for receiving, from a client system distinct from the search engine server system, a search query having one or more query terms;

instructions for identifying a set of documents in accordance with the search query;

instructions for generating a content snippet corresponding to content in a respective document of the identified set of search result documents, the content snippet associated with at least one query term of the one or more query terms in the search query; and

instructions for returning a response to the client system that includes information identifying at least the respective search result document and including the content snippet of the respective search result document;

wherein the instructions for generating the content snippet comprises:

instructions for receiving a set of first token identifiers, comprising compressed content of the respective search result document, from a compressed document repository;

instructions for performing a first decompression operation by applying one or more first mappings to the set of first token identifiers to provide a set of second token identifiers; and

instructions for performing a second decompression operation by applying a second mapping to the set of second token identifiers to recover uncompressed content comprising a portion of the respective document;

wherein the first decompression operation produces a respective second token identifier for each first token identifier, the set of second token identifiers has a same number of distinct token identifiers as the first set of token identifiers, and the second decompression operation produces a respective uncompressed content item for each second token identifier; and

wherein each first token identifier comprises an M bit integer value, and each second token identifier comprises an N bit integer value, N and M are positive integers and N is greater than M.

10. The non-transitory computer readable storage medium of claim 9 , further comprising instructions for deriving relevancy scores for at least a subset of the identified set of search result documents, and instructions for selecting the respective document based on having a high relevancy score.

11. The non-transitory computer readable storage medium of claim 10 , wherein the relevancy scores are based, at least in part on weights assigned to the query terms.

12. The non-transitory computer readable storage medium of claim 9 , wherein the instructions for identifying a set of search result documents in accordance with the search query comprises:

instructions for identifying a preliminary set of documents in accordance with the search query;

instructions for generating a respective preliminary content snippet for a plurality of documents in the preliminary set of documents, each respective preliminary content snippet corresponding to content in the respective preliminary document, and associated with at least one query term of the one or more query terms in the search query;

instructions for producing a set of query expansion terms from the preliminary content snippets;

instructions for combining the one or more query terms with the query expansion terms to form an expanded search query; and

instructions for identifying a set of search result documents in accordance with the expanded search query.