Dynamic augmenting relevance rankings using data from external ratings sources
In general, embodiments of the present invention provide systems, methods and computer readable media for dynamically augmenting relevance rankings using data from external ratings sources.
1. A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to implement operations, the operations comprising:
receiving, using a processor and from an aggregator node via a first network connection, a multi-query search request, wherein the multi-query search request is received by the aggregator node via a second network connection and via a relevance service application programming interface (API) associated with a relevance service framework that comprises a distributed search server;
retrieving, using the processor and from a memory, an ordered list of trilateral inventory objects of a plurality of trilateral inventory objects, wherein each trilateral inventory object of the ordered list of trilateral inventory objects is associated with a respective relevance score, and wherein the ordered list of trilateral inventory objects is ordered based on the relevance scores, wherein each trilateral inventory object of the ordered list of trilateral inventory objects is described by a set of trilateral inventory object attributes that change on a real-time basis, including a trilateral inventory object category, customer ratings from an external ratings source, and a conversion rate;
retrieving, using the processor and from a memory, an ordered list of trilateral inventory objects, wherein each trilateral inventory object of the ordered list of trilateral inventory objects is associated with a respective relevance score, and wherein the ordered list of trilateral inventory objects is ordered based on the relevance scores, wherein each trilateral inventory object of the ordered list of trilateral inventory objects is described by a set of trilateral inventory object attributes including a trilateral inventory object category, customer ratings from an external ratings source, and a conversion rate;
for each trilateral inventory object of the ordered list of trilateral inventory objects, calculating, using the processor and at least a distributed search node plugin of a distributed search server configured for relevance scoring of the trilateral inventory objects for a consumer identifier associated with a consumer device, an updated relevance score using a performance score and an external rating score, wherein calculating the updated relevance score comprises:
for each trilateral inventory object category,
calculating, using the processor, an average category conversion rate using the respective conversion rates associated with each of the trilateral inventory objects associated with the trilateral inventory object category;
dividing, using the processor, a star rating range associated with the external ratings source into a pre-determined set of subranges;
for each subrange of the pre-determined set of subranges,
upon determining, using the processor, whether there is a sufficient amount of impression data collected from trilateral inventory objects having star ratings within the subrange, calculating, using the processor, an average conversion rate for the trilateral inventory objects having star ratings within the subrange; and
for each trilateral inventory object category,
calculating, using the processor, a boost scaling factor for the subrange using the average conversion rate for the subrange, the average conversion rate for the trilateral inventory object category, and a ratings/conversion rate correlation coefficient for trilateral inventory objects within the trilateral inventory object category; and
setting, using the processor, the boost scaling factor for the subrange to 1;
providing dynamically augmented relevance rankings for the plurality of trilateral inventory objects described by the set of trilateral inventory object attributes that change on a real-time basis by:
reordering, using the processor, the ordered list of trilateral inventory objects based on the updated relevance scores to produce a reordered list of trilateral inventory objects;
selecting, using the processor, a top N ranked number of trilateral inventory objects from the reordered list of trilateral inventory objects to render more relevant trilateral inventory objects, as compared to the ordered list of trilateral inventory objects, for the consumer identifier via an electronic interface of the consumer device;
configuring the top N ranked number of trilateral inventory objects for real-time rendering via the electronic interface of the consumer device;
providing, via the second network connection, the top N ranked number of trilateral inventory objects to the relevance service API; and
transmitting, via the relevance service API and using a third network connection, real-time data associated with the top N ranked number of trilateral inventory objects to the consumer device to facilitate the real-time rendering of the top N ranked number of trilateral inventory objects via the electronic interface of the consumer device, wherein the top N ranked number of trilateral inventory objects are accessible via the electronic interface of the consumer device; and
in response to receiving, via the relevance service API and using the third network connection, a user acceptance indication provided via user input to the electronic interface within a given time period that begins at the real-time rendering of the top N ranked number of trilateral inventory objects via the electronic interface, initiating, using the processor, redemption of a good, service or experience associated with at least one trilateral inventory object from the top N ranked number of trilateral inventory objects.
2. The system of claim 1 , wherein the pre-determined set of subranges is configurable.
3. The system of claim 1 , wherein calculating an updated relevance score is performed offline, and wherein the operations further comprise:
for each trilateral inventory object category, generating, using the processor, a table comprising a plurality of table entries, wherein each table entry of the plurality of table entries comprises a key value pair in which a key represents the subrange and a value represents the boost scaling factor.
4. The system of claim 3 , the operations further comprising:
for each table, calculating, using the processor, z-scores using the boost scaling factor; and
determining, using the processor, whether to consolidate a first table and a second table into a third table using the z-scores calculated for the first table and the second table.
5. The system of claim 1 , wherein calculating the average category conversion rate using the respective conversion rates associated with each of the trilateral inventory objects associated with the trilateral inventory object category comprises:
for each trilateral inventory object, calculating, using the processor, an average star rating using a total of votes for each of the star ratings;
sorting, using the processor, the trilateral inventory objects based on their respective average star ratings; and
calculating, using the processor, an external rating score for each trilateral inventory object using a cumulative distribution function applied to the sorted group of trilateral inventory objects.
6. A computer program product, stored on a non-transitory computer readable medium, comprising instructions that when executed on one or more computers cause the one or more computers to perform operations, the operations comprising:
receiving, using a processor and from an aggregator node via a first network connection, a multi-query search request, wherein the multi-query search request is received by the aggregator node via a second network connection and via a relevance service application programming interface (API) associated with a relevance service framework that comprises a distributed search server;
retrieving, using the processor and from a memory, an ordered list of trilateral inventory objects of a plurality of trilateral inventory objects, wherein each trilateral inventory object of the ordered list of trilateral inventory objects is associated with a respective relevance score, and wherein the ordered list of trilateral inventory objects is ordered based on the relevance scores, and wherein each trilateral inventory object of the ordered list of trilateral inventory objects is described by a set of trilateral inventory object attributes that change on a real-time basis, including a trilateral inventory object category, customer ratings from an external ratings source, and a conversion rate;
for each trilateral inventory object of the ordered list of trilateral inventory objects,
calculating, using the processor and at least a distributed search node plugin of the distributed search server configured for relevance scoring of the trilateral inventory objects for a consumer identifier associated with a consumer device, its average external rating;
in an instance in which the trilateral inventory object is determined to be associated with enough evidence, calculating, using the processor, its online boost factor based at least in part on an offline boost scaling factor and a tuning parameter, wherein calculating the offline boost scaling factor comprises:
for each trilateral inventory object category,
calculating, using the processor, an average category conversion rate using the respective conversion rates associated with each of the trilateral inventory objects associated with the trilateral inventory object category;
dividing, using the processor, a star rating range associated with the external ratings source into a pre-determined set of subranges; and
for each of the pre-determined set of subranges,
upon determining, using the processor, that there is a sufficient amount of impression data collected from trilateral inventory objects having star ratings within the subrange, calculating, using the processor, an average conversion rate for the trilateral inventory objects having star ratings within the subrange; and
for each trilateral inventory object category, calculating, using the processor, a boost scaling factor for the subrange using the average conversion rate for the subrange, the average conversion rate for the trilateral inventory object category, and a ratings/conversion rate correlation coefficient for trilateral inventory objects within the trilateral inventory object category; and
setting, using the processor, the boost scaling factor for the subrange to 1;
in an instance in which the trilateral inventory object is determined not to have enough evidence, setting, using the processor, the value of the online boost factor to 1;
providing dynamically augmented relevance rankings for the plurality of trilateral inventory objects described by the set of trilateral inventory object attributes that change on a real-time basis by:
updating, using the processor, each trilateral inventory object relevance score based on its respective online boost scaling factor; and
reordering, using the processor, the ordered list of trilateral inventory objects based on the updated relevance scores to produce a reordered list of trilateral inventory objects;
selecting a top N ranked number of trilateral inventory objects from the reordered list of trilateral inventory objects to render more relevant trilateral inventory objects, as compared to the ordered list of trilateral inventory objects, for the consumer identifier via an electronic interface of the consumer device;
configuring the top N ranked number of trilateral inventory objects for real-time rendering via the electronic interface of the consumer device;
providing, via the second network connection, the top N ranked number of trilateral inventory objects to the relevance service API; and
transmitting, via the relevance service API and using a third network connection, real-time data associated with the top N ranked number of trilateral inventory objects to the consumer device to facilitate the real-time rendering of the top N ranked number of trilateral inventory objects via the electronic interface of the consumer device, wherein the top N ranked number of trilateral inventory objects are accessible via the electronic interface of the consumer device; and
in response to receiving, via the relevance service API and using the third network connection, a user acceptance indication provided via user input to the electronic interface within a given time period that begins at the real-time rendering of the top N ranked number of trilateral inventory objects via the electronic interface, initiating, using the processor, redemption of a good, service or experience associated with at least one trilateral inventory object from the top N ranked number of trilateral inventory objects.
7. The computer program product of claim 6 , wherein the offline boost scaling factor is retrieved from a set of boost tables.
8. The computer program product of claim 6 , wherein the pre-determined set of subranges is configurable.
9. A computer-implemented method, comprising:
receiving, using a processor and from an aggregator node via a first network connection, a multi-query search request, wherein the multi-query search request is received by the aggregator node via a second network connection and via a relevance service application programming interface (API) associated with a relevance service framework that comprises a distributed search server;
retrieving, using the processor and from a memory, an ordered list of trilateral inventory objects of a plurality of trilateral inventory objects, wherein each trilateral inventory object of the ordered list of trilateral inventory objects is associated with a respective relevance score, and wherein the ordered list of trilateral inventory objects is ordered based on the relevance scores, wherein each trilateral inventory object of the ordered list of trilateral inventory objects is described by a set of trilateral inventory object attributes that change on a real-time basis, including a trilateral inventory object category, customer ratings from an external ratings source, and a conversion rate;
for each trilateral inventory object of the ordered list of trilateral inventory objects, calculating, using the processor and at least a distributed search node plugin of the distributed search server configured for relevance scoring of the trilateral inventory objects for a consumer identifier associated with a consumer device, an updated relevance score using a performance score and an external rating score, wherein calculating the updated relevance score comprises:
for each trilateral inventory object category,
calculating, using the processor, an average category conversion rate using the respective conversion rates associated with each of the trilateral inventory objects associated with the trilateral inventory object category;
dividing, using the processor, a star rating range associated with the external ratings source into a pre-determined set of subranges;
for each subrange of the pre-determined set of subranges,
upon determining, using the processor, whether there is a sufficient amount of impression data collected from trilateral inventory objects having star ratings within the subrange, calculating, using the processor, an average conversion rate for the trilateral inventory objects having star ratings within the subrange; and
for each trilateral inventory object category,
calculating, using the processor, a boost scaling factor for the subrange using the average conversion rate for the subrange, the average conversion rate for the trilateral inventory object category, and a ratings/conversion rate correlation coefficient for trilateral inventory objects within the trilateral inventory object category; and
setting, using the processor, the boost scaling factor for the subrange to 1;
providing dynamically augmented relevance rankings for the plurality of trilateral inventory objects described by the set of trilateral inventory object attributes that change on a real-time basis by:
reordering, using the processor, the ordered list of trilateral inventory objects based on the updated relevance scores to produce a reordered list of trilateral inventory objects;
selecting, using the processor, a top N ranked number of trilateral inventory objects from the reordered list of trilateral inventory objects to render more relevant trilateral inventory objects, as compared to the ordered list of trilateral inventory objects, for the consumer identifier via an electronic interface of the consumer device;
configuring the top N ranked number of trilateral inventory objects for real-time rendering via the electronic interface of the consumer device;
providing, via the second network connection, the top N ranked number of trilateral inventory objects to the relevance service API; and
transmitting, via the relevance service API and using a third network connection, real-time data associated with the top N ranked number of trilateral inventory objects to the consumer device to facilitate the real-time rendering of the top N ranked number of trilateral inventory objects via the electronic interface of the consumer device, wherein the top N ranked number of trilateral inventory objects are accessible via the electronic interface of the consumer device; and
in response to receiving, via the relevance service API and using the third network connection, a user acceptance indication provided via user input to the electronic interface within a given time period that begins at the real-time rendering of the top N ranked number of trilateral inventory objects via the electronic interface, initiating, using the processor, redemption of a good, service or experience associated with at least one trilateral inventory object from the top N ranked number of trilateral inventory objects.
10. The computer-implemented method of claim 9 , wherein the pre-determined set of subranges is configurable.
11. The computer-implemented method of claim 9 , wherein calculating an updated relevance score is performed offline, and wherein the computer-implemented method further comprises:
for each trilateral inventory object category, generating, using the processor, a table comprising a plurality of table entries, wherein each table entry of the plurality of table entries comprises a key value pair in which a key represents the subrange and a value represents the boost scaling factor.
12. The computer-implemented method of claim 11 , further comprising:
for each table, calculating, using the processor, z-scores using the boost scaling factor; and
determining, using the processor, whether to consolidate a first table and a second table into a third table using the z-scores calculated for the first table and the second table.
13. The computer-implemented method of claim 9 , wherein calculating the average category conversion rate using the respective conversion rates associated with each of the trilateral inventory objects associated with the trilateral inventory object category comprises:
for each trilateral inventory object, calculating, using the processor, an average star rating using a total of votes for each of the set of star ratings;
sorting, using the processor, the trilateral inventory objects based on their respective average star ratings; and
calculating, using the processor, an external rating score for each trilateral inventory object using a cumulative distribution function applied to the sorted group of trilateral inventory objects.