Generating training data for machine learning
A computer-implemented method includes receiving a rule, wherein the rule includes at least one token, and receiving at least two dictionaries, wherein the at least two dictionaries include at least one general language dictionary and at least one domain-specific dictionary for a domain. The computer-implemented method further includes, for each of the at least one token, selecting at least one word at random from at least one of the at least two dictionaries and adding the at least one word to a test data line, such that the test data line includes a candidate statement conforming to the rule. The computer-implemented method further includes filtering the candidate statement based on a domain-specific model for the domain and including the candidate statement in training data provided to a machine learning model. A corresponding computer program product and computer system are also disclosed.
1. A computer-implemented method, comprising:
receiving a rule, wherein said rule comprises at least one token;
receiving at least two dictionaries, wherein said at least two dictionaries comprise at least one general language dictionary and at least one domain-specific dictionary for a domain;
for each of said at least one token, selecting at least one word at random from at least one of said at least two dictionaries and adding said at least one word to a test data line, such that said test data line comprises a candidate statement conforming to said rule;
filtering said candidate statement based on a domain-specific model for said domain; and
including said candidate statement in training data provided to a machine learning model.
2. The computer-implemented method of claim 1 , further comprising inserting at least one additional word randomly selected from at least one of said at least two dictionaries into said test data line.
3. The computer-implemented method of claim 1 , wherein filtering said candidate statement comprises discarding said candidate statement, if said candidate statement fails to meet a definition of semantically correct candidate statements for said domain, according to said domain-specific model.
4. The computer-implemented method of claim 3 , wherein said domain-specific model is based on a general corpus within said domain.
5. The computer-implemented method of claim 3 , wherein said domain-specific model is based on a corpus that excludes user-specific information.
6. The computer-implemented method of claim 3 , wherein said domain-specific model is an n-gram model of domain-specific statements.
7. The computer-implemented method of claim 3 , wherein said domain is medical diagnosis.
8. The computer-implemented method of claim 7 , wherein said domain-specific model is based on a general medical corpus.
9. The computer-implemented method of claim 7 , wherein said domain-specific model is based on a corpus that excludes medical patient records.
10. The computer-implemented method of claim 7 , wherein said domain-specific model is an n-gram model of medical diagnosis statements.
11. The computer-implemented method of claim 1 , wherein said rule is expressed using regular expressions.
12. The computer-implemented method of claim 1 , wherein said rule is expressed as a state machine.
13. The computer-implemented method of claim 1 , wherein said rule encodes engineered knowledge of a human expert.