IP Library Granted Patent US 10,796,244
Granted Patent B2
US 10,796,244 · App. 15/442,142 · Granted Oct 6, 2020

Method and apparatus for labeling training samples

Inventors: Huige Cheng (Beijing, CN); Yaozong Mao (Beijing, CN)
Assignee: BAIDU ONLINE NETWORK TECHNOLOGY (BEIJING) CO., LTD
G06N20/00G06N7/005
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,796,244
App. No.
15/442,142
Granted
Oct 6, 2020
Kind
B2
Abstract

Provided in the present invention are a method and apparatus for labeling training samples. In the embodiments of the present invention, two mutually independent classifiers, i.e. a first classifier and a second classifier, are used to perform collaborative forecasting on M unlabeled first training samples to obtain some of the labeled first training samples, without the need for the participation of operators; the operation is simple and the accuracy is high, thereby improving the efficiency and reliability of labeling training samples.

Claims (57)

1. An apparatus for labeling training samples, comprising:

one or more processors; and

a memory having one or more programs stored thereon to be executed by said one or more processors, the programs including instruction for:

inputting M unlabeled first training samples into a first classifier to obtain a first forecasting result of each first training sample in the M first training samples, M being an integer greater than or equal to 1;

selecting N first training samples as second training samples from the M first training samples according to the first forecasting result of each first training sample, N being an integer greater than or equal to 1 and less than or equal to M;

inputting the N second training samples into a second classifier to obtain a second forecasting result of each second training sample in the N second training samples, the first classifier and the second classifier being independent of each other;

selecting P second training samples from said N second training samples according to the second forecasting result of each second training sample, P being an integer greater than or equal to 1 and less than or equal to N;

selecting Q first training samples from other first training samples according to first forecasting results of the other first training samples in the M first training samples apart from the N second training samples and the value of P, Q being an integer greater than or equal to 1 and less than or equal to a difference of M−N; and

generating P labeled second training samples according to second forecasting results of the P second training samples and each of the second training samples; and

generating Q labeled first training samples according to first forecasting results of the Q first training samples and each of the first training samples therein.

2. The apparatus of claim 1 ,

wherein the programs include instruction for:

obtaining a first probability that said first training samples indicated by the first forecasting result are of a designated type; and

selecting, from the M first training samples, the N first training samples of which the first probability satisfies a pre-set first training condition as the second training samples; or

wherein the programs include instruction for:

obtaining a second probability that the second training samples indicated by the second forecasting result are of the designated type; and

selecting, from the N second training samples, the P second training samples of which the second probability satisfies a pre-set second training condition.

3. The apparatus of claim 2 , wherein the designated type comprises a positive-example type, a counter-example type, or a combination thereof.

4. The apparatus of claim 2 ,

wherein the first training condition comprises a probability that the first training samples indicated by the first forecasting result are of the designated type is greater than or equal to a first threshold value and is less than or equal to a second threshold value; or

wherein the second training condition comprises a designated number with a minimum probability that the second training samples indicated by the second forecasting result are of the designated type.

5. The apparatus of claim 1 , wherein the programs include instruction for:

selecting, from the other first training samples, P first training samples of which a third probability that the first training samples indicated by the first forecasting result are of a designated type satisfies a pre-set third training condition; and

selecting, from the other first training samples, Q−P first training samples of which the third probability satisfies a pre-set fourth training condition.

6. The apparatus of claim 5 ,

wherein the third training condition comprises a designated number with a minimum probability that the first training samples indicated by the first forecasting result are of a designated type; or

wherein the fourth training condition comprises a designated number with a maximum probability that the first training samples indicated by the first forecasting result are of a designated type.

7. The apparatus of claim 1 , wherein a ratio of Q−P to 2P is a golden ratio.

8. A method for labeling training samples, comprising a labeling process including:

selecting, from M first training samples each associated with a first forecasting result, N first training samples as N second training samples based upon the first forecasting result of the each of the M first training samples, M being a first predetermined integer greater than or equal to 1, N being a second predetermined integer between 1 and M inclusive;

selecting P second training samples from the N second training samples based upon a second forecasting result of each of the N second training samples, P being a third predetermined integer between 1 and N inclusive;

selecting Q first training samples from other first training samples based upon first forecasting results of the other first training samples in the M first training samples apart from the N second training samples and a value of P, Q being a fourth predetermined integer between 1 and M−N inclusive;

generating P labeled second training samples based upon second forecasting results of the P second training samples and each of the second training samples therein; and

generating Q labeled first training samples based upon first forecasting results of the Q first training samples and each of the first training samples therein.

9. The method of claim 8 , wherein the labelling process further includes inputting the M first training samples into a first classifier to obtain the first forecasting result of the each of the M first training samples.

10. The method of claim 9 , wherein said inputting the M first training samples comprises inputting the M first training samples as unlabeled first training samples.

11. The method of claim 9 , wherein the first classifier is based on a training sample set, the labeling process further including adding the P labeled second training samples and the Q labeled first training samples to the training sample set.

12. The method of claim 11 , further comprising repeatedly executing the labeling process until a classification accuracy rate based on the first classifier is greater than or equal to a pre-set accuracy rate threshold value.

13. The method of claim 11 , further comprising repeatedly executing the labeling process until a number of the first training samples contained in the training sample set is greater than or equal to a pre-set number threshold value.

14. The method of claim 8 , wherein the labelling process further includes inputting the N second training samples into a second classifier to obtain the second forecasting result of the each of the N second training samples.

15. The method of claim 14 , wherein the second classifier is independent from a first classifier used for obtaining the first forecasting result of the each of the M first training samples.

16. An apparatus for labeling training samples, comprising:

one or more processors; and

a memory having one or more programs stored thereon to be executed by said one or more processors, the programs including instruction for a labeling process including:

instruction for selecting, from M first training samples each associated with a first forecasting result, N first training samples as N second training samples based upon the first forecasting result of the each of the M first training samples, M being a first predetermined integer greater than or equal to 1, N being a second predetermined integer between 1 and M inclusive;

instruction for selecting P second training samples from the N second training samples based upon a second forecasting result of each of the N second training samples, P being a third predetermined integer between 1 and N inclusive;

instruction for selecting Q first training samples from other first training samples based upon first forecasting results of the other first training samples in the M first training samples apart from the N second training samples and a value of P, Q being a fourth predetermined integer between 1 and M−N inclusive;

instruction for generating P labeled second training samples based upon second forecasting results of the P second training samples and each of the second training samples therein; and

instruction for generating Q labeled first training samples based upon first forecasting results of the Q first training samples and each of the first training samples therein.

17. The apparatus of claim 16 , wherein the instruction for the labeling process includes instruction for inputting the M first training samples into a first classifier to obtain the first forecasting result of the each of the M first training samples, wherein the M first training samples are unlabeled.

18. The apparatus of claim 17 ,

wherein the first classifier is based on a training sample set, wherein the instruction for the labeling process includes instruction for adding the P labeled second training samples and the Q labeled first training samples to the training sample set, and

wherein the programs include instruction for repeatedly executing the instruction for the labeling process until a classification accuracy rate based on the first classifier is greater than or equal to a pre-set accuracy rate threshold value.

19. The apparatus of claim 17 ,

wherein the first classifier is based on a training sample set, wherein the instruction for the labeling process includes instruction for adding the P labeled second training samples and the Q labeled first training samples to the training sample set, and

wherein the programs include instruction for repeatedly executing the instruction for the labeling process until a number of the first training samples contained in the training sample set is greater than or equal to a pre-set number threshold value.

20. The apparatus of claim 16 , wherein the instruction for the labeling process includes instruction for inputting the N second training samples into a second classifier to obtain the second forecasting result of the each of the N second training samples, wherein the second classifier is independent from a first classifier used for obtaining the first forecasting result of the each of the M first training samples.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 24, 2017
From: CHENG, HUIGE; MAO, YAOZONG
To: BAIDU ONLINE NETWORK TECHNOLOGY (BEIJING) CO., LTD
Reel/Frame 041374/0177 →
Priority Claims (1)
CN 2014 1 0433020 · Aug 28, 2014 · national
Continuity (2)
Continuation 14586902 · Dec 30, 2014
Related Publication 20170161645A1 · Jun 8, 2017
Cited By (1)
US 12,525,337