METHOD OF TESTING ONLINE RECOMMENDER SYSTEM
A recommender system can be analyzed to determine various performance characteristics of an online content service provider. The recommender system is analyzed using a predetermined policy, to determine if it satisfies such policy, and/or has other measurable intended and/or unintended biases. The policy can include such parameters as whether a particular profile is presented with certain particular types of items by the recommender system. The reliability of search engines can also be tested using a similar approach.
1 - 20 . (canceled)
21 . An automated method of measuring behavior of a recommender system, which recommender includes one or more programs executing on a computing machine and is used for recommending items of interest to online users, the method comprising the steps of:
(a) setting up a target preference to be used by the recommender system including when providing recommendation results for items to a user;
(b) causing the recommender system to interact with at least one proxy account and so as to generate a plurality of separate recommendations for a plurality of corresponding items for a user associated with said proxy account;
(c) verifying whether the recommender system exhibits said target preference by examining said separate recommendations.
22 . The method of claim 21 , wherein said target preference is identified as part of a contractual arrangement between the online content service provider and a content provider which provides items to the online content service provider for distribution.
23 . The method of claim 22 , wherein said target preference is specified as an absolute number of recommendations, and/or a percentage of recommendations to be provided to subscribers.
24 . The method of claim 23 wherein said target preference is limited to a particular time period.
25 . The method of claim 21 , wherein said preference is measured as part of an electronic audit of the performance of the online content service provider.
26 - 30 . (canceled)
31 . The method of claim 21 , wherein said target preference is given to a set of preference items which are favored over other comparable items when deciding what to present to said user as part of said recommendation results.
32 . The method of claim 21 wherein said recommender system is part of a search engine.
33 . The method of claim 21 wherein said proxy account includes a benchmark profile of a user.
34 . The method of claim 32 wherein said benchmark profile includes demographic data including at least one of an age, gender, income or location of the user.
35 . The method of claim 21 wherein said proxy account includes dummy standard data representing a target user profile.
36 . The method of claim 21 generating a report identifying a degree to which said plurality of corresponding items includes a programmed bias by the recommender system towards particular types of items.
37 . The method of claim 21 generating a report identifying a degree to which said plurality of corresponding items includes a programmed bias by the recommender system towards items originating from a particular source.
38 . The method of claim 21 including a step: generating a report identifying a degree to which the recommender system is associating a particular user profile with a particular item.
39 . The method of claim 21 wherein step (c) is performed at least in part automatically by analyzing logs of actual interactions by users with the recommender system.
40 . The method of claim 21 including a step of generating a report identifying a performance of the recommender system in achieving said target preference.
41 . The method of claim 21 during step (b) said interacting is done automatically by a client computing device separate from said recommender system.
42 . The method of claim 21 wherein the recommender system is a first recommender system, and steps (a) through (c) are repeated for a second separate recommender system to identify said target preference for a set of items offered in common with the first recommender system.
43 . The method of claim 21 wherein steps (a) through (c) are repeated for a plurality of proxy accounts to identify a target preference, wherein said plurality of proxy accounts include different user profile data.
44 . An automated method of measuring behavior of a recommender system operating at an online vendor website, which recommender includes one or more programs executing on a computing machine and is used for recommending items of interest to an online user, the method comprising the steps of:
(a) setting up a target preference to be used by the recommender system including when providing recommendation results for items to a user in response to a query for an item;
wherein said target preference is set by the online vendor for a set of preference items;
(b) testing the recommender system by causing it to interact with at least one proxy account associated with a test user and generate a plurality of separate recommendations for a plurality of corresponding items in response to requests for items by said test user;
(c) generating a report identifying whether the recommender system is accurately presenting said set of preference items as part of said plurality of separate recommendations in accordance with said target preference.
45 . The method of claim 44 , wherein said target preference is identified as part of a contractual arrangement between the online content service provider and a content provider which provides items to the online content service provider for distribution.
46 . The method of claim 44 , wherein said target preference is specified as an absolute number of recommendations, and/or a percentage of recommendations to be provided to subscribers.
47 . The method of claim 44 , wherein said target preference is limited to a particular time period.
48 . The method of claim 44 , wherein said preference is measured as part of an electronic audit of the performance of the online content service provider.
49 . The method of claim 44 , wherein said target preference is given to a set of preference items which are favored over other comparable items when deciding what to present to said user as part of said recommendation results.
50 . The method of claim 44 wherein said recommender system is part of a search engine.
51 . The method of claim 44 wherein said proxy account includes a benchmark profile of a user.
52 . The method of claim 51 wherein said benchmark profile includes demographic data including at least one of an age, gender, income or location of the user.
53 . The method of claim 44 wherein said proxy account includes dummy data representing a target user profile.
54 . The method of claim 44 generating a report identifying a degree to which said plurality of corresponding items includes a programmed bias by the recommender system towards particular types of items.
55 . The method of claim 44 generating a report identifying a degree to which said plurality of corresponding items includes a programmed bias by the recommender system towards items originating from a particular source.
56 . The method of claim 44 including a step: generating a report identifying a degree to which the recommender system is associating a particular user profile with a particular item.
57 . The method of claim 44 wherein step (c) is performed at least in part automatically by analyzing logs of actual interactions by users with the recommender system.
58 . The method of claim 44 including a step of generating a report identifying a performance of the recommender system in achieving said target preference.
59 . The method of claim 44 wherein during step (b) said testing is done automatically by a client computing device separate from said recommender system.
60 . The method of claim 44 wherein the recommender system is a first recommender system, and steps (a) through (c) are repeated for a second separate recommender system to identify said target preference for a set of items offered in common with the first recommender system.
61 . The method of claim 44 wherein steps (a) through (c) are repeated for a plurality of proxy accounts to identify a target preference, wherein said plurality of proxy accounts include different user profile data.