Method of and system for building search index using machine learning algorithm

1. A computer-implemented method for selecting documents for inclusion into a search index of a search engine, the search engine implemented by a server, the server executing a machine learning algorithm (MLA),

the MLA having been trained to rank documents based on size and user interactions with the documents,

the method executable by the server, the method comprising:

acquiring, by the server, a plurality of digital documents, each respective document of the plurality of digital documents having a respective size;

determining, by the server, for each respective document of the plurality of digital documents, a respective value parameter based on user interactions with the respective document, the respective value parameter being indicative of a usefulness of the respective document to the search engine as a search result document;

ranking, by the MLA, the plurality of digital documents to obtain a ranked list of documents, the ranking being based on:

a respective quality value parameter of the respective document, the respective quality value parameter being based on the respective value parameter and the respective size of the respective document,

the ranking being performed such that a cumulative quality value parameter is maximized based on:

the quality value parameter of a given digital document of the ranked list of documents weighted by a sum of the sizes of a given subset of documents of the plurality of digital documents, wherein the given subset of documents consists of documents having a respective value parameter below the value parameter of the given digital document in the plurality of digital documents according to:

Q

m

=

∑

i

N

⁢

(

q

i

·

∑

j

M

⁢

s

j

)

,

p

j

<

p

i

wherein:

Q is the cumulative quality value parameter;

q i is the respective value parameter of the respective i th document in the plurality of digital documents;

s j is the respective size of the respective j th document in the subset of documents;

p i is the respective quality value parameter of the i th document in the plurality of digital documents;

p j is the respective quality value parameter of the j th document in the plurality of digital documents;

N is a number of documents in the plurality of digital documents; and

M is a number of documents in the given subset of documents;

selecting, by the MLA, a subset of documents from the ranked list of documents based on the quality value parameter of each respective document of the subset of documents; and

storing, by the server, the subset of documents in the search index.

2. The method of claim 1 , wherein the given subset consists of all documents having a respective value parameter below the value parameter of the given digital document.

3. The method of claim 2 , wherein the user interactions are at least one of: predicted user interactions with the respective document, and past user interactions with the respective document.

4. The method of claim 3 , wherein a rank of a respective document in the ranked list of documents is based at least partially on a ratio of the respective value parameter to the respective size of the respective document.

5. The method of claim 4 , wherein the selecting is further based on:

a sum of the respective sizes of the documents in the subset of documents being below a predetermined threshold size.

6. The method of claim 5 , wherein the predetermined threshold size is an available storage size of the search index of the search engine.

7. The method of claim 6 , wherein the subset of documents selected from the ranked list of documents comprises a number of O top ranked documents that fit within the search index of the search engine.

8. The method of claim 1 , wherein the MLA has been trained to rank documents using a LambdaMART algorithm.

9. The method of claim 1 , wherein the MLA has been trained to rank documents using a listwise ranking algorithm.

10. The method of claim 1 , wherein the ranking comprises maximizing the cumulative quality value parameter based on:

Q

m

=

∑

i

N

⁢

(

w

i

·

q

i

·

∑

j

M

⁢

w

j

·

s

j

)

,

p

j

<

p

i

wherein:

Q m is the cumulative quality value parameter;

w i is a respective weight associated with the respective value parameter;

q i is the respective value parameter of the respective i th in the plurality of digital documents;

w j is a respective weight associated with the respective size;

s j is the respective size of the respective j th document in the subset of documents;

p i is the respective quality value parameter of the i th document in the plurality of digital documents;

p j is the respective quality value parameter of the j th document in the plurality of digital documents;

N is a number of documents in the plurality of digital documents; and

M is a number of documents in the given subset of documents.

11. The method of claim 1 , wherein the computing the respective value parameter of the respective document is performed using a second MLA executed by the server.

12. A system for selecting documents for inclusion into a search index of a search engine, the search engine implemented by the system, the system executing a machine learning algorithm (MLA),

the MLA having been trained to rank documents based on size and user interactions with the documents,

the system comprising:

a processor;

a non-transitory computer-readable medium comprising instructions, the processor,

upon executing the instructions, being configured to:

acquire a plurality of digital documents, each respective document of the plurality of digital documents having a respective size;

determine for each respective document of the plurality of digital documents, a respective value parameter based on user interactions with the respective document, the respective value parameter being indicative of a usefulness of the respective document to the search engine as a search result document;

rank, by the MLA, the plurality of digital documents to obtain a ranked list of documents, the ranking being based on:

a respective quality value parameter of the respective document, the respective quality value parameter being based on the respective value parameter and the respective size of the respective document,

the ranking being performed such that a cumulative quality value parameter is maximized based on:

 the quality value parameter of a given digital document of the ranked list of documents weighted by a sum of the sizes of a given subset of documents of the plurality of digital documents, wherein the given subset of documents consists of documents having a respective value parameter below the value parameter of the given digital document in the plurality of digital documents according to:

Q

m

=

∑

i

N

⁢

(

q

i

·

∑

j

M

⁢

s

j

)

,

p

j

<

p

i

wherein:

 Q is the cumulative quality value parameter;

 q i is the respective value parameter of the respective i th document in the plurality of digital documents;

 s j is the respective size of the respective j th document in the subset of documents;

 p i is the respective quality value parameter of the i th document in the plurality of digital documents;

 p j is the respective quality value parameter of the j th document in the plurality of digital documents;

 N is a number of documents in the plurality of digital documents; and

 M is a number of documents in the given subset of documents;

select, by the MLA, a subset of documents from the ranked list of documents based on the quality value parameter of each respective document of the subset of documents; and

store the subset of documents in the search index.

13. The system of claim 12 , wherein the given subset consists of all documents having a respective value parameter below the value parameter of the given digital document.

14. The system of claim 13 , wherein the user interactions are at least one of: predicted user interactions with the respective document, and past user interactions with the respective document.

15. The system of claim 14 , wherein a rank of a respective document in the ranked list of documents is based at least partially on a ratio of the respective value parameter to the respective size of the respective document.

16. The system of claim 15 , wherein the selecting is further based on:

a sum of the respective sizes of the documents in the subset of documents being below a predetermined threshold size.

17. The system of claim 16 , wherein the predetermined threshold size is an available storage size of the search index of the search engine.

18. The system of claim 17 , wherein the subset of documents selected from the ranked list of documents comprises a number of O top ranked documents that fit within the search index of the search engine.

19. The system of claim 12 , wherein the MLA has been trained to rank documents using a LambdaMART algorithm.

20. The system of claim 12 , wherein the MLA has been trained to rank documents using a listwise ranking algorithm.

21. The system of claim 12 , wherein the ranking comprises maximizing the cumulative quality value parameter based on:

Q

m

=

∑

i

N

⁢

(

w

i

·

q

i

·

∑

j

M

⁢

w

j

·

s

j

)

,

p

j

<

p

i

wherein:

Q m is the cumulative quality value parameter;

w i is a respective weight associated with the respective value parameter;

q i is the respective value parameter of the respective i th in the plurality of digital documents;

w j is a respective weight associated with the respective size;

s j is the respective size of the respective j th document in the subset of documents;

p i is the respective quality value parameter of the i th document in the plurality of digital documents;

p j is the respective quality value parameter of the j th document in the plurality of digital documents;

N is a number of documents in the plurality of digital documents; and

M is a number of documents in the given subset of documents.

22. The system of claim 12 , wherein the computing the respective value parameter of the respective document is performed using a second MLA executed by the system.