Granted Patent B2

US 12,730,788 · App. 18/629,674 · Granted Sep 8, 2026

Inventors: Onur Gungor (Sunnyvale, CA); Tri Cao (San Bruno, CA); Vineet Abhishek (Mountain View, CA)

Assignee: Walmart Apollo, LLC

G06F16/217G06F11/3409G06F16/24578G06F16/9538

Abstract

Claims (68)

1 . A system comprising:

one or more processors; and

one or more non-transitory computer-readable media storing computing instructions that, when executed on the one or more processors, cause the one or more processors to perform operations comprising:

generating, using at least two different search algorithms, combined search results based on a search query from an electronic device of a user, each search result of the combined search results having a respective rank based on at least one of the at least two different search algorithms;

storing interactions by at least one user with the search results of the combined search results, wherein an interaction with a respective search result of the combined search results indicates which of the at least two different search algorithms returned the respective search result;

determining a set of search results from the combined search results based on the stored interactions, wherein each result within the set of search results was respectively interacted with by the at least one user;

determining a traffic impact for at least one of the at least two different search algorithms based on the set of search results, wherein the traffic impact is based on:

a number of search results, in the set of search results, that were interacted with by the at least one user and were returned by both a first search algorithm and a second search algorithm of the at least two different search algorithms,

a number of search results, in the set of search results, that were interacted with by the at least one user and were returned by the first search algorithm but not the second search algorithm, and

a number of search results, in the set of search results, that were interacted with by the at least one user and were returned by the second search algorithm but not the first algorithm; and

implementing the first search algorithm or the second search algorithm on a website based on the determined traffic impact.

2 . The system of claim 1 , wherein generating the combined search results comprises:

interleaving search results of the second search algorithm and search results of the first search algorithm.

3 . The system of claim 1 , wherein generating the combined search results comprises:

including a copy of a search result returned by both the first search algorithm and the second search algorithm.

4 . The system of claim 1 , wherein the respective interaction with each result within the set of search results comprises at least one of:

a click through of the result;

an item view of the result;

an add to cart of the result; or

a purchase of the result.

5 . The system of claim 1 , wherein the operations further comprise:

hashing a user ID of the user; and

determining that the user is eligible for a search algorithm test based upon the user ID, as hashed.

6 . The system of claim 1 , wherein each search result within the set of search results has an identical rank in both search results of the first search algorithm and search results of the second search algorithm.

7 . The system of claim 1 , wherein the operations further comprise:

generating a report comprising the traffic impact, wherein generating the report is based at least in part on the respective interaction with each result within the set of search results, and is further based at least in part on at least one of:

determining a latency of the first search algorithm;

determining a latency of the second search algorithm;

adding the latency of the first search algorithm and the latency of the second search algorithm to the report;

determining a recall size of the first search algorithm;

determining a recall size of the second search algorithm; or

adding the recall size of the first search algorithm and the recall size of the second search algorithm to the report.

8 . The system of claim 7 , wherein generating the report further comprises:

plotting a graph comprising:

an axis for a lift of the second search algorithm over the first search algorithm for each respective rank in the combined search results; and

an axis for the traffic impact for each result rank in the set of search results; and

adding the graph to the report.

9 . A method being implemented via execution of computing instructions configured to run at one or more processors and configured to be stored at one or more non-transitory computer-readable media, the method comprising:

generating, using at least two different search algorithms, combined search results based on a search query from an electronic device of a user, each search result of the combined search results having a respective rank based on at least one of the at least two different search algorithms;

storing interactions by at least one user with the search results of the combined search results, wherein an interaction with a respective search result of the combined search results indicates which of the at least two different search algorithms returned the respective search result;

determining a set of search results from the combined search results based on the stored interactions, wherein each result within the set of search results was respectively interacted with by the at least one user;

determining a traffic impact for at least one of the at least two different search algorithms based on the set of search results, wherein the traffic impact is based on:

a number of search results, in the set of search results, that were interacted with by the at least one user and were returned by both a first search algorithm and a second search algorithm of the at least two different search algorithms,

a number of search results, in the set of search results, that were interacted with by the at least one user and were returned by the first search algorithm but not the second search algorithm, and

a number of search results, in the set of search results, that were interacted with by the at least one user and were returned by the second search algorithm but not the first search algorithm; and

implementing the first search algorithm or the second search algorithm on a website based on the determined traffic impact.

10 . The method of claim 9 , wherein generating the combined search results comprises:

interleaving search results of the second search algorithm and search results of the first search algorithm.

11 . The method of claim 9 , wherein generating the combined search results comprises:

including a copy of a search result returned by both the first search algorithm and the second search algorithm.

12 . The method of claim 9 , wherein the respective interaction with each result within the set of search results comprises at least one of:

a click through of the result;

an item view of the result;

an add to cart of the result; or

a purchase of the result.

13 . The method of claim 9 further comprising:

hashing a user ID of the user; and

determining that the user is eligible for a search algorithm test based upon the user ID, as hashed.

14 . The method of claim 9 , wherein each search result within the set of search results has an identical rank in both search results of the first search algorithm and search results of the second search algorithm.

15 . The method of claim 9 further comprising:

generating a report comprising the traffic impact, wherein generating the report is based at least in part on the respective interaction with each result within the set of search results, and is further based at least in part on at least one of:

determining a latency of the first search algorithm;

determining a latency of the second search algorithm;

adding the latency of the first search algorithm and the latency of the second search algorithm to the report;

determining a recall size of the first search algorithm;

determining a recall size of the second search algorithm; or

adding the recall size of the first search algorithm and the recall size of the second search algorithm to the report.

16 . The method of claim 15 , wherein generating the report further comprises: plotting a graph comprising: an axis for a lift of the second search algorithm over the first search algorithm for each respective rank in the combined search results; and an axis for the traffic impact for each result rank in the set of search results; and adding the graph to the report.

Assignments (1)

Continuity (3)

References Cited (108)