Systems and methods for resource-efficient data collection for multi-stage ranking systems
Systems, methods, and non-transitory computer-readable media can receive a set of candidate training items for training an early stage model in a multi-stage recall optimization model, wherein the multi-stage recall optimization model comprises the early stage model and a target model. A random subset of the candidate training items is selected from the set of candidate training items. For each training item in the subset of candidate training items, a score is determined based on the target model. Each training item in the subset of candidate training items is labeled with a label based on a probability of the training item being a top-K of the set of candidate training items had the set of candidate training items been scored based on the target model.
1. A computer-implemented method comprising:
receiving, by a computing system, a set of candidate training items for training an early stage model in a multi-stage recall optimization model, wherein the multi-stage recall optimization model comprises the early stage model and a target model;
selecting, by the computing system, a random subset of candidate training items from the set of candidate training items to train the early stage model;
determining, by the computing system, for each training item in the subset of candidate training items, a score based on the target model; and
labeling, by the computing system, each training item in the subset of candidate training items with a label based on a probability of the training item being in a top-K of the set of candidate training items had the set of candidate training items been scored based on the target model.
2. The computer-implemented method of claim 1 , wherein the label is a non-binary label.
3. The computer-implemented method of claim 2 , wherein the label is a value between 0 and 1.
4. The computer-implemented method of claim 1 , wherein the probability of each training item being in the top-K of the set of candidate training items is determined based on a hypergeometric distribution.
5. The computer-implemented method of claim 4 , wherein the label for a training item is equal to the probability of the training item being in the top-K of the set of candidate training items had the set of candidate training items been scored based on the target model.
6. The computer-implemented method of claim 1 , further comprising training the early stage model based on the labels for the subset of candidate training items.
7. The computer-implemented method of claim 1 , wherein the target model is trained based on a labeled set of training data, wherein the labeled set of training data is labeled based on user feedback information.
8. The computer-implemented method of claim 1 , further comprising ranking the subset of candidate training items based on the scores.
9. The computer-implemented method of claim 1 , further comprising:
providing a set of candidate items to the multi-stage recall optimization model; and
identifying a set of the top-K items from the set of candidate items based on the multi-stage recall optimization model.
10. The computer-implemented method of claim 9 , wherein identifying the set of the top-K items from the set of candidate items based on the multi-stage recall optimization model comprises:
identifying a first subset of the set of candidate items based on the early stage model;
providing the first subset of the set of candidate items to the target model; and
identifying the set of top-K items from the set of candidate items based on the target model.
11. A system comprising:
at least one processor; and
a memory storing instructions that, when executed by the at least one processor, cause the system to perform a method comprising:
receiving a set of candidate training items for training an early stage model in a multi-stage recall optimization model, wherein the multi-stage recall optimization model comprises the early stage model and a target model;
selecting a random subset of candidate training items from the set of candidate training items to train the early stage model;
determining, for each training item in the subset of candidate training items, a score based on the target model; and
labeling each training item in the subset of candidate training items with a label based on a probability of the training item being in a top-K of the set of candidate training items had the set of candidate training items been scored based on the target model.
12. The system of claim 11 , wherein the label is a non-binary label.
13. The system of claim 12 , wherein the label is a value between 0 and 1.
14. The system of claim 11 , wherein the probability of each training item being in the top-K of the set of candidate training items is determined based on a hypergeometric distribution.
15. The system of claim 14 , wherein the label for a training item is equal to the probability of the training item being in the top-K of the set of candidate training items had the set of candidate training items been scored based on the target model.
16. A non-transitory computer-readable storage medium including instructions that, when executed by at least one processor of a computing system, cause the computing system to perform a method comprising:
receiving a set of candidate training items for training an early stage model in a multi-stage recall optimization model, wherein the multi-stage recall optimization model comprises the early stage model and a target model;
selecting a random subset of candidate training items from the set of candidate training items to train the early stage model;
determining, for each training item in the subset of candidate training items, a score based on the target model; and
labeling each training item in the subset of candidate training items with a label based on a probability of the training item being in a top-K of the set of candidate training items had the set of candidate training items been scored based on the target model.
17. The non-transitory computer-readable storage medium of claim 16 , wherein the label is a non-binary label.
18. The non-transitory computer-readable storage medium of claim 17 , wherein the label is a value between 0 and 1.
19. The non-transitory computer-readable storage medium of claim 16 , wherein the probability of each training item being in the top-K of the set of candidate training items is determined based on a hypergeometric distribution.
20. The non-transitory computer-readable storage medium of claim 19 , wherein the label for a training item is equal to the probability of the training item being in the top-K of the set of candidate training items had the set of candidate training items been scored based on the target model.