Hybrid processing of disjunctive and conjunctive conditions of a search query for a similarity search
View Patent ↗Aspects of the invention are configured to perform an operation comprising receiving a query specifying an AND condition and an OR condition, determining, based on an AND index structure, a set of documents, of a plurality of documents in a corpus, satisfying the AND condition of the query, computing a query similarity score for a first document in the set of documents, wherein the query similarity score is based on a first hash value computed for the OR condition of the query, a weight value for the OR condition, and a second hash value for the first document specified in an OR index, and returning an indication of the first document and the query similarity score as responsive to the query.
1. A system, comprising:
a processor; and
a memory containing a program which when executed by the processor performs an operation comprising:
receiving a query specifying an AND condition and an OR condition;
determining, based on an AND index structure, a set of documents, of a plurality of documents in a corpus, satisfying the AND condition of the query;
computing a query similarity score, based on an OR index, for a first document in the set of documents determined based on the AND index, comprising:
computing a respective hash value for each respective OR condition of the plurality of OR conditions;
computing a first hash value for the plurality of OR conditions;
determining a second hash value for the first document,
wherein the second hash value is specified in an OR index;
computing an overall similarity score for the first document relative to the plurality of OR conditions based on the first hash value for the plurality of OR conditions and the second hash value for the first document received from the OR index;
computing a respective OR similarity score for the first document relative to each respective OR condition of the plurality of OR conditions based on the second hash value for the first document received from the OR index and the respective hash value for the respective OR condition; and
adding, for each respective OR similarity score exceeding a predefined threshold, a weight associated with the respective OR condition to the overall similarity score; and
returning the overall similarity score as the query similarity score; and
returning an indication of the first document and the query similarity score as responsive to the query.
2. The system of claim 1 , wherein the AND index comprises a posting list configured to store a document identifier (ID) for each document including a respective feature, of a plurality of features, wherein the OR index comprises a respective hash value for each of the plurality of documents, wherein the second hash value and the hash values in the OR index are computed based on a locality-sensitive hashing function.
3. The system of claim 2 , wherein the query specifies a plurality of AND conditions, wherein the determined set of documents satisfy each of the plurality of AND conditions, wherein determining the set of documents comprises:
generating a search query including an indication of each of the plurality of AND conditions specified in the query;
processing the search query against the AND index; and
receiving, from the AND index, the set of documents comprising the document ID of each document in the set of documents.
4. The system of claim 1 , the operation further comprising prior to computing the similarity score for the first document:
receiving a document identifier (ID) for the first document from the OR index; and
determining that the document ID for the first document is included in the set of documents.
5. The system of claim 4 , the operation further comprising:
receiving a document identifier (ID) for a second document of the plurality of documents in the corpus from the OR index;
determining that the document ID for the second document is not included in the set of documents;
refraining from computing a query similarity score for the second document; and
refraining from returning the second document as responsive to the query.
6. The system of claim 1 , wherein the AND index and the OR index are generated during a preprocessing phase of the plurality of documents in the corpus.
7. A computer program product, comprising:
a non-transitory computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code executable by a processor to perform an operation comprising:
receiving a query specifying an AND condition and an OR condition;
determining, based on an AND index structure, a set of documents, of a plurality of documents in a corpus, satisfying the AND condition of the query;
computing a query similarity score, based on an OR index, for a first document in the set of documents determined based on the AND index, comprising:
computing a respective hash value for each respective OR condition of the plurality of OR conditions;
computing a first hash value for the plurality of OR conditions;
determining a second hash value for the first document, wherein the second hash value is specified in an OR index;
computing an overall similarity score for the first document relative to the plurality of OR conditions based on the first hash value for the plurality of OR conditions and the second hash value for the first document received from the OR index;
computing a respective OR similarity score for the first document relative to each respective OR condition of the plurality of OR conditions based on the second hash value for the first document received from the OR index and the respective hash value for the respective OR condition; and
adding, for each respective OR similarity score exceeding a predefined threshold, a weight associated with the respective OR condition to the overall similarity score; and
returning the overall similarity score as the query similarity score; and
returning an indication of the first document and the query similarity score as responsive to the query.
8. The computer program product of claim 7 , wherein the AND index comprises a posting list configured to store a document identifier (ID) for each document including a respective feature, of a plurality of features, wherein the OR index comprises a respective hash value for each of the plurality of documents, wherein the second hash value and the hash values in the OR index are computed based on a locality-sensitive hashing function.
9. The computer program product of claim 8 , wherein the query specifies a plurality of AND conditions, wherein the determined set of documents satisfy each of the plurality of AND conditions, wherein determining the set of documents comprises:
generating a search query including an indication of each of the plurality of AND conditions specified in the query;
processing the search query against the AND index; and
receiving, from the AND index, the set of documents comprising the document ID of each document in the set of documents.
10. The computer program product of claim 7 , the operation further comprising prior to computing the similarity score for the first document:
receiving a document identifier (ID) for the first document from the OR index; and
determining that the document ID for the first document is included in the set of documents.
11. The computer program product of claim 10 , wherein the AND index and the OR index are generated during a preprocessing phase of the plurality of documents in the corpus, wherein the operation further comprises:
receiving a document identifier (ID) for a second document of the plurality of documents in the corpus from the OR index;
determining that the document ID for the second document is not included in the set of documents;
refraining from computing a query similarity score for the second document; and
refraining from returning the second document as responsive to the query.