Method and system for ranking plurality of digital documents
A method and server for ranking a plurality of documents are disclosed. The method includes determining, by the server, for each document from the plurality of documents a proximity value between a respective document vector and an other document vector. A given proximity value for a respective document being indicative of a similarity between (i) the respective document and (ii) at least one other one from the plurality of documents. The method also comprises using, by the server, the proximity values of the plurality of documents for ranking the plurality of documents.
1. A computer-implemented method of ranking a plurality of documents, the plurality of documents being associated with respective document data, the plurality of documents to be used as search results in response to a query submitted by a user, the query being associated with query data, the query having been submitted to a search engine hosted by a server, the method comprising:
generating, by the server executing a Machine Learning Algorithm (MLA), a plurality of document vectors for the plurality of documents and a query vector for the query,
a given document vector being generated based on the document data associated with a respective document from the plurality of documents and the query vector being generated based on the query data associated with the query,
the MLA having been trained:
based on a given training document-query pair associated with a respective relevance score,
the relevance score being indicative of a relevance of a training document in the given training pair to a training query in the given training pair,
to generate (i) a training document vector for the training document and (ii) a training query vector for the training query, such that a query-to-document (Q2D) proximity value between (i) the training document vector of the training document and (ii) the training query vector of the training query is representative of the relevance score,
executing a two-step ranking process including (i) a first ranking step based on Q2D proximity values between the query vector and respective ones amongst the plurality of document vectors, and (ii) a second step based on document-to-document (D2D) proximity values between document vectors,
the first step including:
determining, by the server, for each document from the plurality of documents, a corresponding Q2D proximity value between the query vector and the respective document vector, a given Q2D proximity value for a respective document being indicative of a similarity between (i) the query and (ii) the respective document,
determining, by the server, a reduced plurality of documents using the Q2D proximity values of the plurality of documents;
the second step including:
determining, by the server, for each document from the reduced plurality of documents a D2D proximity value between the respective document vector and an other document vector,
a given D2D proximity value for a respective document being indicative of a similarity between (i) the respective document and (ii) at least one other one from the reduced plurality of documents; and
ranking, by the server, documents from the reduced plurality of documents using the D2D proximity values of the reduced plurality of documents;
triggering, by the server, display of at least some of documents from the ranked reduced plurality of documents to the user.
2. The method of claim 1 , wherein the method further comprises:
generating, by the server, a reference vector for the plurality of documents as a combination of document vectors associated with the plurality of documents,
the reference vector is the other document vector,
the given D2D proximity value being indicative of the similarity between (i) the respective document and (ii) other ones from the plurality of documents.
3. The method of claim 2 , wherein the reference vector is one of:
an average vector generated based on the plurality of document vectors;
a mean vector generated based on the plurality of document vectors; and
a medoid vector generated based on the plurality of document vectors.
4. The method of claim 1 , wherein the plurality of documents is pre-selected from a pool of documents for ranking,
the pre-selection being executed by the server prior to the executing the two-step ranking process.
5. The method of claim 4 , wherein the pre-selection includes
determining, by the server, respective frequency parameters for documents from the pool of documents,
a given frequency parameter being indicative of a number of times query terms from the query are present in the respective document; and
selecting, by the server, a first set of documents from the pool of documents,
the first set of documents having documents associated with respective frequency parameters that are above a first threshold.
6. The method of claim 1 , wherein the server executes a ranking algorithm configured to rank the plurality of documents based on (i) the document data associated with the respective documents from the plurality of documents and (ii) additional document data associated with the plurality of documents.
7. The method of claim 6 , wherein the using the D2D proximity values of the plurality of documents for ranking the plurality of documents comprises:
using, by the server, the D2D proximity values as the additional document data for ranking the plurality of documents by the ranking algorithm.
8. The method of claim 1 , wherein document data comprises document data of a first type.
9. The method of claim 8 , wherein a first document from the plurality of documents has document data of the first type and a second document from the plurality of documents is missing document data of the first type, and wherein the method further comprises:
determining, by the server, a D2D proximity value between (i) the document vector of the first document and (ii) the document vector of the second document; and
in response to the D2D proximity value between (i) the document vector of the first document and (ii) the document vector of the second document being above a threshold, using, by the server, the document data of the first type of the first document as an estimation of the document data of the first type of the second document.
10. The method of claim 1 , wherein triggering display of at least some of documents from the ranked reduced plurality of documents comprises:
triggering, by the server, display of a Search Engine Results Page (SERP) on a device associated with the user,
the SERP including a plurality of search results, the plurality of documents being used as the plurality of search results, the plurality of search results being ranked in a same order as the ranked reduced plurality of documents.
11. The method of claim 1 , wherein the MLA is a Neural Network (NN), the NN comprises a document-dedicated portion and a query-dedicated portion,
(i) the document-dedicated portion being configured to generate the training document vector based on document data associated with the training document,
(ii) the query-dedicated portion being configured to generate the training query vector based on query data associated with the training query, and
the document-dedicated portion and the query-dedicated portion having been trained together such that the Q2D proximity value between (i) the training document vector and (ii) the training query vector is representative of the relevance score.
12. A server for ranking a plurality of documents, the plurality of documents being associated with respective document data, the plurality of documents to be used as search results in response to a query submitted by a user, the query being associated with query data, the query having been submitted to a search engine hosted by the server, the server comprising a hardware processor, the server being configured to:
generate a plurality of documents vectors for the plurality of documents and a query vector for the query by executing a Machine Learning Algorithm (MLA),
a given document vector being generated based on the document data associated with a respective document and the query vector being generated based on the query data associated with the query,
the MLA having been trained:
based on a given training document-query pair associated with a respective relevance score,
the relevance score being indicative of a relevance of a training document in the given training pair to a training query in the given training pair,
to generate (i) a training document vector for the training document and (ii) a training query vector for the training query, such that a query-to-document (Q2D) proximity value between (i) the training document vector of the training document and (ii) the training query vector of the training query is representative of the relevance score,
execute a two-step ranking process including (i) a first ranking step based on Q2D proximity values between the query vector and respective ones amongst the plurality of document vectors, and (ii) a second step based on document-to-document (D2D) proximity values between document vectors,
the server being configured to, during the first step:
determine, for each document from the plurality of documents, a corresponding Q2D proximity value between the query vector and the respective document vector, a given Q2D proximity value for a respective document being indicative of a similarity between (i) the query and (ii) the respective document,
determine a reduced plurality of documents using the Q2D proximity values of the plurality of documents;
the server being configured to, during the second step:
determine, for each document from the reduced plurality of documents, a D2D proximity value between the respective document vector and an other document vector,
a given D2D proximity value for a respective document being indicative of a similarity between (i) the respective document and (ii) at least one other one from the reduced plurality of documents; and
rank documents from the reduced plurality of documents using the D2D proximity values of the reduced plurality of documents; use the proximity values of the plurality of documents for ranking the plurality of documents;
trigger display of at least some of documents from the ranked reduced plurality of documents to the user.
13. The server of claim 12 , wherein the server is further configured to:
generate a reference vector for the plurality of documents as a combination of document vectors associated with the plurality of documents,
the reference vector is the other document vector,
the given D2D proximity value being indicative of the similarity between (i) the respective document and (ii) other ones from the plurality of documents.
14. The server of claim 13 , wherein the reference vector is one of:
an average vector generated based on the plurality of document vectors;
a mean vector generated based on the plurality of document vectors; and
a medoid vector generated based on the plurality of document vectors.
15. The server of claim 12 , wherein the plurality of documents is pre-selected from a pool of documents for ranking,
the pre-selection being executed by the server prior to the executing the two-step ranking process.
16. The server of claim 15 , wherein the pre-selection includes the server configured to:
determine respective frequency parameters for documents from the pool of documents,
a given frequency parameter being indicative of a number of times query terms from the query are present in the respective document; and
select a first set of documents from the pool of documents,
the first set of documents having documents associated with respective frequency parameters that are above a first threshold.