Method and apparatus of providing semi-automated classifier adaptation for natural language processing
View Patent ↗Example embodiments of the present invention may include a method that provides transcribing spoken utterances occurring during a call and assigning each of the spoken utterances with a corresponding set of first classifications. The method may also include determining a confidence rating associated with each of the spoken utterances and the assigned set of first classifications, and performing at least one of reclassifying the spoken utterances with new classifications based on at least one additional classification operation, and adding the assigned first classifications and the corresponding plurality of spoken utterances to a training data set.
1. A method, comprising:
transcribing via a processor at least a portion of a plurality of spoken utterances occurring during a call;
assigning each of the plurality of spoken utterances with a corresponding set of first classifications;
determining a confidence rating associated with each of the plurality of spoken utterances and the assigned set of first classifications; and
performing at least one of:
reclassifying the plurality of spoken utterances with new classifications based on at least one additional classification operation, and
adding the assigned first classifications and the corresponding plurality of spoken utterances to a training data set.
2. The method of claim 1 , wherein the plurality of spoken utterances are user spoken utterances responsive to automated questions presented to the user during the call.
3. The method of claim 1 , wherein the transcribed at least a portion of the plurality of spoken utterances comprises at least one of rejected spoken utterances, low confidence rated spoken utterances and disconfirmed spoken utterances.
4. The method of claim 1 , wherein if the determining the confidence rating associated with each of the plurality of spoken utterances produces confidence ratings that are above a predefined threshold then adding the labeled plurality of spoken utterances to the training data set.
5. The method of claim 1 , wherein if determining the confidence rating associated with each of the plurality of spoken utterances produces confidence ratings that are below a predefined threshold then reclassifying the plurality of spoken utterances with new classifications based on at least two different classification operations.
6. The method of claim 5 , wherein the at least two different classification operations comprise at least one of generating a list of the three highest classification confidence levels among the plurality of utterances, performing a Naïve-Bayes classification operation on the plurality of spoken utterances, and performing a decision tree classification on the plurality of spoken utterances.
7. The method of claim 1 , further comprising:
updating a production classifier used to perform classification operations by adding the new classifications to a database; and
repeating the transcribing, via the processor, of the at least a portion of the plurality of spoken utterances occurring during the call;
repeating the assigning of each of the plurality of spoken utterances with the corresponding set of first classifications;
repeating the determining of the confidence rating associated with each of the plurality of spoken utterances and the assigned set of first classifications; and
repeating the performing of the at least one of the
reclassifying the plurality of spoken utterances with the new classifications based on the at least one additional classification operation, and
adding the assigned first classifications and the corresponding plurality of spoken utterances to the training data set.
8. An apparatus, comprising:
a receiver configured to receive a plurality of spoken utterances and record them in a database memory; and
a processor configured to
transcribe at least a portion of the plurality of spoken utterances occurring during a call,
assign each of the plurality of spoken utterances with a corresponding set of first classifications,
determine a confidence rating associated with each of the plurality of spoken utterances and the assigned set of first classifications, and
perform at least one of
reclassify the plurality of spoken utterances with new classifications based on at least one additional classification operation, and
add the assigned first classifications and the corresponding plurality of spoken utterances to a training data set.
9. The apparatus of claim 8 , wherein the plurality of spoken utterances are user spoken utterances responsive to automated questions presented to the user during the call.
10. The apparatus of claim 8 , wherein the transcribed at least a portion of the plurality of spoken utterances comprises at least one of rejected spoken utterances, low confidence rated spoken utterances and disconfirmed spoken utterances.
11. The apparatus of claim 8 , wherein if the determined confidence ratings associated with each of the plurality of spoken utterances produces confidence ratings that are above a predefined threshold then the processor is configured to add the labeled plurality of spoken utterances to the training data set.
12. The apparatus of claim 8 , wherein if the determined confidence rating associated with each of the plurality of spoken utterances produces confidence ratings that are below a predefined threshold then the processor is configured to reclassify the plurality of spoken utterances with new classifications based on at least two different classification operations.
13. The apparatus of claim 12 , wherein the at least two different classification operations comprise the process performing at least one of generating a list of the three highest classification confidence levels among the plurality of utterances, performing a Naïve-Bayes classification operation on the plurality of spoken utterances, and performing a decision tree classification on the plurality of spoken utterances.
14. A non-transitory computer readable storage medium configured to store instructions that when executed cause a processor to perform:
transcribing via a processor at least a portion of a plurality of spoken utterances occurring during a call;
assigning each of the plurality of spoken utterances with a corresponding set of first classifications;
determining a confidence rating associated with each of the plurality of spoken utterances and the assigned set of first classifications; and
performing at least one of
reclassifying the plurality of spoken utterances with new classifications based on at least one additional classification operation, and
adding the assigned first classifications and the corresponding plurality of spoken utterances to a training data set.
15. The non-transitory computer readable storage medium of claim 14 , wherein the plurality of spoken utterances are user spoken utterances responsive to automated questions presented to the user during the call.
16. The non-transitory computer readable storage medium of claim 14 , wherein the transcribed at least a portion of the plurality of spoken utterances comprises at least one of rejected spoken utterances, low confidence rated spoken utterances and disconfirmed spoken utterances.
17. The non-transitory computer readable storage medium of claim 14 , wherein if the determining the confidence rating associated with each of the plurality of spoken utterances produces confidence ratings that are above a predefined threshold then adding the labeled plurality of spoken utterances to the training data set.
18. The non-transitory computer readable storage medium of claim 14 , wherein if the determining the confidence rating associated with each of the plurality of spoken utterances produces confidence ratings that are below a predefined threshold then reclassifying the plurality of spoken utterances with new classifications based on at least two different classification operations.
19. The non-transitory computer readable storage medium of claim 18 , wherein the at least two different classification operations comprise at least one of generating a list of the three highest classification confidence levels among the plurality of utterances, performing a Naïve-Bayes classification operation on the plurality of spoken utterances, and performing a decision tree classification on the plurality of spoken utterances.
20. The non-transitory computer readable storage medium of claim 14 , wherein the processor is further configured to perform:
updating a production classifier used to perform classification operations by adding the new classifications to a training database; and
repeating the transcribing, via the processor, of the at least a portion of the plurality of spoken utterances occurring during the call;
repeating the assigning of each of the plurality of spoken utterances with the corresponding set of first classifications;
repeating the determining of the confidence rating associated with each of the plurality of spoken utterances and the assigned set of first classifications; and
repeating the performing of the at least one of the
reclassifying the plurality of spoken utterances with the new classifications based on the at least one additional classification operation, and
adding the assigned first classifications and the corresponding plurality of spoken utterances to the training data set.