Granted Patent B2

US 11,681,713 · App. 16/277,088 · Granted Jun 20, 2023

Inventors: Valery Ivanovich Duzhik (Minsk, BY); Andrey Dmitrievich Drozdovsky (Moscow, RU); Oleg Pavlovich Naydin (Moscow, RU)

Assignee: YANDEX EUROPE AG

G06F16/24578G06F16/953

Abstract

Claims (73)

1. A computer-implemented method for ranking search results using a first machine learning algorithm (MLA), the method being executed by a server hosting a second MLA, the second MLA having been trained to determine query similarity, the method comprising:

receiving, by the second MLA, a new query;

retrieving, by the second MLA from a search log database, a plurality of search queries having been submitted on a search engine server;

computing, by the second MLA, a first similarity parameter between the new query and each query of the plurality of search queries, the first similarity parameter being determined based on one or more textual features of the new query and each query of the plurality of search queries;

selecting, by the second MLA, from the plurality of search queries, a first past query associated with a highest first similarity parameter;

retrieving, by the second MLA, a set of search results associated with the first past query, the set of search results including a first search result associated with:

a first annotation including:

a second past query having been used to access the first search result on the search engine server, the second past query being different from the new query and the first past query;

a first user-interaction parameter indicative of user behavior with the first search result in response to the second past query having been used to access the first search result on the search engine server;

a second annotation including:

a third past query having been used to access the first search result on the search engine server, the third past query being different from the new query, the first past query and the second past query;

a second user-interaction parameter indicative of user behavior with the first search result in response to the third past query having been used to access the first search result on the search engine server;

computing, by the second MLA, a second similarity parameter for each of the second past query and third past query, the second similarity parameter being computed by:

determining a third similarity parameter between the new query and each of the second past query and third past query, the third similarity parameter being determined based on one or more textual features of the new query and each of the second past query and third past query;

multiplying the third similarity parameter associated with the second past query with the first user-interaction parameter; and

multiplying the third similarity parameter associated with the third past query with the second user-interaction parameter;

in response to the second similarity parameter associated with the second past query being higher than the second similarity parameter associated with the third past query, using the second similarity parameter associated with the second past query as a ranking feature by the first MLA for ranking the associated set of search results as search results for the new query.

2. The method of claim 1 , wherein the method further comprises:

during a training phase:

retrieving from the search log database, the plurality of search queries having been submitted on a search engine server, each of the plurality of search queries being associated with the respective set of search results, each respective search result of the respective set of search results being associated with a respective user-interaction parameter;

computing, for each query of the plurality of search queries, a respective query vector based on the respective user-interaction parameter associated with each search result of the respective set of search results;

computing, for each possible pair of queries within the plurality of search queries, based on the respective query vectors of each query of the pair of queries, a respective training similarity parameter, the respective training similarity parameter being indicative of a degree of similarity between queries in the pair of queries;

generating a set of training objects, each training object of the set of training objects including an indication of a respective one of each of the possible pair of queries and the respective training similarity parameter; and

training the second MLA on the set of training objects to determine a fourth similarity parameter of a new pair of queries, at least one query of the new pair of queries not being included in the set of training objects.

3. The method of claim 2 , wherein the first user-interaction parameter being indicative of user behavior with the first search result by at least one user after having submitted the second past query to the search engine server.

4. The method of claim 2 , wherein, during the training phase, the respective training similarity parameter is computed by using one of: a scalar multiplication or a cosine similarity.

5. The method of claim 4 , wherein the training the second MLA on the set of training objects to determine the fourth similarity parameter of the new pair of queries is based on at least one respective textual feature of each query of the new pair of queries.

6. The method of claim 5 , wherein the second MLA is a neural network.

7. The method of claim 6 , wherein the first MLA is a decision tree based algorithm.

8. The method of claim 7 , wherein the first MLA uses a plurality of additional features for ranking, and wherein the second similarity parameter is added to the plurality of additional features for ranking.

9. The method of claim 2 , wherein the user-interaction parameter is at least one of: a dwell time and a click-through rate (CTR).

10. The method of claim 3 , wherein the method further comprises, prior to the receiving of the new query:

generating a plurality of annotations including the first annotation and second annotation; and

storing the plurality of annotations in a storage of the server.

11. The method of claim 10 , wherein the generating the plurality of annotations is executed by a third trained MLA.

12. A system for ranking search results using a first machine learning algorithm (MLA), the system executable by a second MLA on the system, the second MLA having been trained to determine query similarity, the system comprising:

a processor;

a non-transitory computer-readable medium comprising instructions;

the processor, upon executing the instructions, being configured to:

receive, by the second MLA, a new query;

retrieve, by the second MLA from a search log database, a plurality of search queries having been submitted on a search engine server;

compute, by the second MLA, a first similarity parameter between the new query and each query of the plurality of search queries, the first similarity parameter being determined based on one or more textual features of the new query and each query of the plurality of search queries;

select, by the second MLA, from the plurality of search queries, a first past query associated with a highest first similarity parameter;

retrieve, by the second MLA, a set of search results associated with the first past query, the set of search results including a first search result associated with:

a first annotation including:

a second past query having been used to access the first search result on the search engine server, the second past query being different from the new query and first past query; and

a first user-interaction parameter indicative of user behavior with the first search result in response to the second past query having been used to access the first search result on the search engine server;

a second annotation including:

a third past query having been used to access the first search result on the search engine server, the third past query being different from the new query, the first past query and the second past query; and

a second user-interaction parameter indicative of user behavior with the first search result in response to the third past query having been used to access the first search result on the search engine server;

compute, by the second MLA, a second similarity parameter for each of the second past query and third past query, the second similarity parameter being computed by the processor configured to:

determine a third similarity parameter between the new query and each of the second past query and third past query, the third similarity parameter being determined based on one or more textual features of the new query and each of the second past query and third past query;

multiply the third similarity parameter associated with the second past query with the first user-interaction parameter;

multiply the third similarity parameter associated with the third past query with the second user-interaction parameter; and

in response to the second similarity parameter associated with the second past query being higher than the second similarity parameter associated with the third past query, use the second similarity parameter associated with the second past query as a ranking feature by the first MLA for ranking the associated set of search results as search results for the new query.

13. The system of claim 12 , wherein the processor is further configured to:

during a training phase:

retrieve from the search log database, the plurality of search queries having been submitted on a search engine server, each of the plurality of search queries being associated with the respective set of search results, each respective search result of the respective set of search results being associated with a respective user-interaction parameter;

compute, for each query of the plurality of search queries, a respective query vector based on the respective user-interaction parameter associated with each search result of the respective set of search results;

compute, for each possible pair of queries within the plurality of search queries, based on the respective query vectors of each query of the pair of queries, a respective training similarity parameter, the respective training similarity parameter being indicative of a degree of similarity between queries in the pair of queries;

generate a set of training objects, each training object of the set of training objects including an indication of a respective one of each of the possible pair of queries and the respective training similarity parameter; and

train the second MLA on the set of training objects to determine a fourth similarity parameter of a new pair of queries, at least one query of the new pair of queries not being included in the set of training objects.

14. The system of claim 12 , wherein the first user-interaction parameter being indicative of user behavior with the first search result by at least one user after having submitted the second past query to the search engine server.

15. The system of claim 13 , wherein, during the training phase, the respective training similarity parameter is computed by using one of: a scalar multiplication or a cosine similarity.

16. The system of claim 15 , wherein the training the second MLA on the set of training objects to determine the fourth similarity parameter of the new pair of queries is based on at least one respective textual feature of each query of the new pair of queries.

17. The system of claim 16 , wherein the second MLA is a neural network.

18. The system of claim 17 , wherein the first MLA is a decision tree based algorithm.

19. The system of claim 18 , wherein the first MLA uses a plurality of additional features for ranking, and wherein the second similarity parameter is added to the plurality of additional features for ranking.

20. The system of claim 13 , wherein the user-interaction parameter is at least one of: a dwell time and a click-through rate (CTR).

21. The system of claim 14 , wherein the processor is further configured to, prior to the receiving of the new query:

generate a plurality of annotations including the first annotation and second annotation; and

store the plurality of annotations in a storage of the system.

22. The system of claim 21 , wherein the generating the plurality of annotations is executed by a third trained MLA.

Assignments (9)

Priority Claims (1)

Continuity (1)

Cited By (1)

US 12,619,618