Evaluating performance of recommender system

1. A method for evaluating a performance of a recommender system, comprising:

identifying, at a processor of the recommender system, a session of M content items consumed by a user, wherein the session is a sequence of the M content items defined such that, for each of a first (M-1) content items in the session, a delay time between a completion of consumption of one of the first (M-1) content items and a commencement of consumption of a directly next content item in the session is less than a predetermined duration of time, M being greater than two;

ranking, at the processor, the M content items in the session based on a measure associated with a first (M-2) content items in the session, a content item, of the content items, having a corresponding vector representation, the measure being a vector representation of a sum of vector representations for the first (M-2) content items, a rank based on a degree of a match, for the each content item, between the vector representation of the sum of the vector representations for the first (M-2) content items and the corresponding vector representation;

producing, at the processor, a predicted rank of an (M-1)th content item in the session based on a last (M-2) content items in the session;

generating, at the processor, a session metric, the session metric being a quotient of the predicted rank of the (M-1)th content item divided by a number N of content items in a content item-base, N being greater than or equal to M;

using, at the processor, the session metric to quantify, for each recommender sub-system of the recommender system, a quality of a recommendation produced by a recommender sub-system;

adjusting, at the processor and based on the session metric, a weight applied to an output of the recommender sub-system;

repeating, at the processor and for at least one other session, the identifying, the ranking, the producing, the generating, the using, and the adjusting at least until the session metric is at a minimal value between 0 and 1, wherein the minimal value being between 0 and 0.5 is indicative that the predicted rank of the (M-1)th content item is near a top of a rank of the M content items, the minimal value realized without any user personalization information, and the session metric being used by the processor to weight combined recommendations from the recommender sub-systems; and

producing, by the processor and after the minimal value has been realized, an output of the recommender system, the output of the recommender system being a recommended content item.

2. The method of claim 1 , wherein the content items are songs.

3. The method of claim 1 , wherein the adjusting the weight applied to the output of the recommender sub-system is in response to the session metric being less than a threshold.

4. The method of claim 1 , further comprising receiving a content item consumption history of the user.

5. The method of claim 1 , wherein the processor weights combined recommendations from at least one selected from the group consisting of: dynamic sub-systems, offline sub-systems, and pooling sub-systems.

6. A non-transitory computer-readable medium storing computer code for controlling a processor of a recommender system to cause the processor to evaluate a performance of the recommender system, the computer code including instructions to:

identify a session of M content items consumed by a user, wherein the session is a sequence of the M content items defined such that, for each of a first (M-1) content items in the session, a delay time between a completion of consumption of one of the first (M-1) content items and a commencement of consumption of a directly next content item in the session is less than a predetermined duration of time, M being greater than two;

rank the M content items in the session based on a measure associated with a first (M-2) content items in the session, a content item, of the content items, having a corresponding vector representation, the measure being a vector representation of a sum of vector representations for the first (M-2) content items, a rank based on a degree of a match, for the each content item, between the vector representation of the sum of the vector representations for the first (M-2) content items and the corresponding vector representation;

producing a predicted rank of an (M-1)th content item in the session based on a last (M-2) content items in the session;

generate a session metric, the session metric being a quotient of the predicted rank of the (M-1)th content item divided by a number N of content items in a content item-base, N being greater than or equal to M;

use the session metric to quantify, for each recommender sub-system of the recommender system, a quality of a recommendation produced by a recommender sub-system;

adjust, based on the session metric, a weight applied to an output of the recommender sub-system;

repeat, for at least one other session, the instructions to identify, rank, produce, generate, use, and adjust at least until the session metric is at a minimal value between 0 and 1, wherein the minimal value being between 0 and 0.5 is indicative that the predicted rank of the (M-1)th content item is near a top of a rank of the M content items, the minimal value realized without any user personalization information, and the session metric being used by the processor to weight combined recommendations from the recommender sub-systems; and

produce, after the minimal value has been realized, an output of the recommender system, the output of the recommender system being a recommended content item.

7. A system for evaluating a performance of a recommender system, comprising:

a memory configured to store a content item consumption history of a user; and

a processor of the recommender system configured to:

identify a session of M content items consumed by the user, wherein the session is a sequence of the M content items,

rank the M content items in the session, based on a measure associated with a first (M-2) content items in the session, a content item, of the content items, having a corresponding vector representation, the measure being a vector representation of a sum of vector representations for the first (M-2) content items, a rank based on a degree of a match, for the each content item, between the vector representation of the sum of the vector representations for the first (M-2) content items and the corresponding vector representation,

produce a predicted rank of an (M-1)th content item, based on a last (M-2) content items in the session,

generate a session metric, the session metric being a quotient of the predicted rank of the (M-1)th content item divided by a number of content items in a content item-base, N being greater than or equal to M,

use the session metric to the recommender system to quantify, for each recommender sub-system of the recommender system, a quality of a recommendation produced by a recommender sub-system,

adjust, based on the session metric, a weight applied to an output of the recommender sub-system,

repeat, for at least one other session, operations to identify, rank, produce, generate, use, and adjust at least until the session metric is at a minimal value between 0 and 1, wherein the minimal value being between 0 and 0.5 is indicative that the predicted rank of the (M-1)th content item is near a top of a rank of the M content items, the minimal value realized without any user personalization information, wherein the session is defined such that, for each of the first (M-1) content items in the session, a delay time between a completion of consumption of one of the first (M-1) content items and a commencement of consumption of a directly next content item in the session is less than a predetermined duration of time, M being greater than two, and the session metric is used by the processor to weight combined recommendations from the recommender sub-systems; and

produce, after the minimal value has been realized, an output of the recommender system, the output of the recommender system being a recommended content item.