Methods and servers for ranking digital documents in response to a query
A method and server for ranking digital documents for a query are disclosed. T server hosts a search engine associated with an inverted index. The method includes accessing inverted index for retrieving query-independent data for a first document-term pair and a second document-term pair, where the query-independent data is indicative of (i) a term-specific occurrence of the first term in content of the document and (ii) a term-specific occurrence of the second term in the content of the document. The method includes generating a query-dependent feature using the query-independent data and which is indicative of a group occurrence of the first term with the second term in the content of the document. The method includes generating a ranking feature for the given document based on at least the first term, the second term, and the query-dependent feature, and ranking the given document based on at least the ranking feature.
1. A method of ranking digital documents in response to a query, the digital documents being potentially relevant to the query having a first term and a second term, the query having been submitted by a user of an electronic device communicatively coupled with a server hosting a search engine, the search engine being associated with an inverted index storing information associated with document-term (DT) pairs, the method executable by the server, the method comprising:
for a given document from a plurality of potentially relevant documents:
accessing the inverted index for retrieving query-independent data for a first DT pair and a second DT pair, the first DT pair having the given document and the first term, the second DT pair having the given document and the second term,
the query-independent data being indicative of (i) a term-specific occurrence of the first term in content associated with the given document and (ii) a term-specific occurrence of the second term in the content associated with the given document;
generating a query-dependent feature using the query-independent data retrieved for the first DT pair and the second DT pair,
the query-dependent feature being indicative of a group occurrence of the first term with the second term in the content associated with the given document;
generating a ranking feature for the given document based on at least the first term, the second term, and the query-dependent feature, the generating the ranking feature for the given document being performed by a Neural Network (NN); and
ranking, by the server, the given document from the plurality of potentially relevant documents based on at least the ranking feature;
wherein the method further comprises training the NN to generate the ranking feature, the training the NN comprising:
generating a training set for a training document-query (DQ) pair to be used during a given training iteration of the NN, the training DQ pair having a training query and a training document, the training document being associated with a label, the label being indicative of relevance of the training document to the training query, the generating comprising:
generating a plurality of training term embeddings based on respective terms from the training query;
accessing the inverted index associated with the search engine for retrieving a plurality of query-independent datasets associated with respective ones of a plurality of training DT pairs,
a given one of the plurality of training DT pairs including the training document and a respective one of the plurality of terms from the training query;
generating a plurality of training feature vectors for the plurality of training DT pairs using the plurality of query-independent datasets;
during a given training iteration of the NN:
inputting, into the NN, the plurality of training term embeddings and the plurality of training feature vectors for generating a predicted ranking feature for the training DQ pair; and
adjusting the NN based on a comparison between the label and the predicted ranking feature so that the NN generates for a given in-use DQ pair a respective predicted ranking feature that is indicative of relevance of a respective in-use document to a respective in-use query.
2. The method of claim 1 , wherein the query-independent data has been stored in the inverted index prior to receipt of the query from the electronic device, and wherein the query-dependent feature is generated after the receipt of the query from the electronic device.
3. The method of claim 1 , wherein the query-dependent feature is generated using the query-independent data in real-time during a document ranking procedure of the search engine.
4. The method of claim 1 , wherein the ranking is performed by a decision-tree-based Machine Learning Algorithm (MLA) configured to rank the plurality of potentially relevant documents based on their relevance to the query.
5. The method of claim 1 , wherein the method further comprises, for a given one of the plurality of terms, determining a similar term, and wherein:
when accessing the inverted index for retrieving the query-independent data, the retrieved query-independent data comprises query-independent data for a third DT pair, the third DT pair having the given document and the similar term.
6. The method of claim 1 , wherein the accessing the inverted index is for further retrieving a query-independent content-based data associated with the first DT pair and the second DT pair,
the query-independent content-based data being indicative of a textual context of the respective term within the content associated with the given document.
7. The method of claim 1 , wherein the term-specific occurrence of the first term comprises at least one of:
one or more positions of the first term in a title associated with given document;
one or more positions of the first term in a URL associated with the given document; and
one or more positions of the first term in a body of the given document.
8. The method of claim 1 , wherein the group occurrence of the first term with the second term in the content associated with the given document comprises at least one of:
a number of times the second term from the query is included in addition to the first term in a title associated with the given document;
a number of times the second term from the query is included in addition to the first term in a URL associated with the given document; and
a number of times the second term from the query is included in addition to the first term in a body of the given document.
9. A server for ranking digital documents in response to a query, the digital documents being potentially relevant to the query having a first term and a second term, the query having been submitted by a user of an electronic device communicatively coupled with the server hosting a search engine, the search engine being associated with an inverted index storing information associated with document-term (DT) pairs, the server configured to:
for a given document from a plurality of potentially relevant documents:
access the inverted index for retrieving query-independent data for a first DT pair and a second DT pair, the first DT pair having the given document and the first term, the second DT pair having the given document and the second term,
the query-independent data being indicative of (i) a term-specific occurrence of the first term in content associated with the given document and (ii) a term-specific occurrence of the second term in the content associated with the given document;
generate a query-dependent feature using the query-independent data retrieved for the first DT pair and the second DT pair,
the query-dependent feature being indicative of a group occurrence of the first term with the second term in the content associated with the given document;
generate a ranking feature for the given document based on at least the first term, the second term, and the query-dependent feature, the server employing a Neural Network (NN) for generating the ranking feature for the given document; and
rank the given document from the plurality of potentially relevant documents based on at least the ranking feature;
wherein the server is further configured to train the NN to generate the ranking feature, the server being configured to:
generate a training set for a training document-query (DQ) pair to be used during a given training iteration of the NN, the training DQ pair having a training query and a training document, the training document being associated with a label, the label being indicative of relevance of the training document to the training query, to generate the training set the server being configured to:
generate a plurality of training term embeddings based on respective terms from the training query;
access the inverted index associated with the search engine for retrieving a plurality of query-independent datasets associated with respective ones of a plurality of training DT pairs,
a given one of the plurality of training DT pairs including the training document and a respective one of the plurality of terms from the training query;
generate a plurality of training feature vectors for the plurality of training DT pairs using the plurality of query-independent datasets;
during a given training iteration of the NN:
input, by the server into the NN, the plurality of training term embeddings and the plurality of training feature vectors for generating a predicted ranking feature for the training DQ pair; and
adjust the NN based on a comparison between the label and the predicted ranking feature so that the NN generates for a given in-use DQ pair a respective predicted ranking feature that is indicative of relevance of a respective in-use document to a respective in-use query.
10. The server of claim 9 , wherein the query-independent data has been stored in the inverted index prior to receipt of the query from the electronic device, and wherein the query-dependent feature is generated after the receipt of the query from the electronic device.
11. The server of claim 9 , wherein the query-dependent feature is generated using the query-independent data in real-time during a document ranking procedure of the search engine.
12. The server of claim 9 , wherein the server is configured to rank by employing a decision-tree-based Machine Learning Algorithm (MLA) configured to rank the plurality of potentially relevant documents based on their relevance to the query.
13. The server of claim 9 , wherein the server is further configured to, for a given one of the plurality of terms, determine a similar term, and wherein:
when accessing the inverted index for retrieving the query-independent data, the retrieved query-independent data comprises query-independent data for a third DT pair, the third DT pair having the given document and the similar term.
14. The server of claim 9 , wherein the server is configured to access the inverted index for further retrieving a query-independent content-based data associated with the first DT pair and the second DT pair,
the query-independent content-based data being indicative of a textual context of the respective term within the content associated with the given document.
15. The server of claim 9 , wherein the term-specific occurrence of the first term comprises at least one of:
one or more positions of the first term in a title associated with given document;
one or more positions of the first term in a URL associated with the given document; and
one or more positions of the first term in a body of the given document.
16. The server of claim 9 , wherein the group occurrence of the first term with the second term in the content associated with the given document comprises at least one of:
a number of times the second term from the query is included in addition to the first term in a title associated with the given document;
a number of times the second term from the query is included in addition to the first term in a URL associated with the given document; and
a number of times the second term from the query is included in addition to the first term in a body of the given document.