Method and apparatus of providing semi-automated classifier adaptation for natural language processing

1. A method, comprising:

assigning, by a processor, each of a plurality of spoken utterances occurring during a call with a corresponding set of classifications;

determining, by the processor, a confidence rating associated with each of the plurality of spoken utterances and the assigned set of classifications; and

performing, by the processor, at least one of:

reclassifying the plurality of spoken utterances with new classifications based on at least one additional classification operation, and

adding the assigned classifications and the corresponding plurality of spoken utterances to a training data set.

2. The method of claim 1 , wherein the plurality of spoken utterances are user spoken utterances responsive to automated questions presented to the user during the call.

3. The method of claim 1 , comprising transcribing, via the processor, a plurality of spoken utterances occurring during the call, wherein the transcribed plurality of spoken utterances comprises at least one of rejected spoken utterances, low confidence rated spoken utterances and disconfirmed spoken utterances.

4. The method of claim 1 , wherein if the determining the confidence rating associated with each of the plurality of spoken utterances produces confidence ratings that are above a predefined threshold then adding the labeled plurality of spoken utterances to the training data set.

5. The method of claim 1 , wherein if determining the confidence rating associated with each of the plurality of spoken utterances produces confidence ratings that are below a predefined threshold then reclassifying the plurality of spoken utterances with new classifications based on at least two different classification operations.

6. The method of claim 5 , wherein the at least two different classification operations comprise at least one of generating a list of the three highest classification confidence levels among the plurality of utterances, performing a Naïve-Bayes classification operation on the plurality of spoken utterances, and performing a decision tree classification on the plurality of spoken utterances.

7. The method of claim 1 , further comprising:

updating a production classifier used to perform classification operations by adding the new classifications to a database; and

repeating the assigning of each of the plurality of spoken utterances with the corresponding set of classifications;

repeating the determining of the confidence rating associated with each of the plurality of spoken utterances and the assigned set of classifications; and

repeating the performing of the at least one of the

reclassifying the plurality of spoken utterances with the new classifications based on the at least one additional classification operation, and

adding the assigned classifications and the corresponding plurality of spoken utterances to the training data set.

8. An apparatus, comprising:

a receiver configured to receive a plurality of spoken utterances that occur during a call and record them in a database memory; and

a processor configured to:

assign each of the plurality of spoken utterances with a corresponding set of classifications,

determine a confidence rating associated with each of the plurality of spoken utterances and the assigned set of classifications, and

perform at least one of:

reclassify the plurality of spoken utterances with new classifications based on at least one additional classification operation, and

add the assigned classifications and the corresponding plurality of spoken utterances to a training data set.

9. The apparatus of claim 8 , wherein the plurality of spoken utterances are user spoken utterances responsive to automated questions presented to the user during the call.

10. The apparatus of claim 8 , wherein the processor is configured to transcribe the plurality of spoken utterances that occur during the call, wherein the transcribed plurality of spoken utterances comprises at least one of rejected spoken utterances, low confidence rated spoken utterances and disconfirmed spoken utterances.

11. The apparatus of claim 8 , wherein if the determined confidence ratings associated with each of the plurality of spoken utterances produces confidence ratings that are above a predefined threshold then the processor is configured to add the labeled plurality of spoken utterances to the training data set.

12. The apparatus of claim 8 , wherein if the determined confidence rating associated with each of the plurality of spoken utterances produces confidence ratings that are below a predefined threshold then the processor is configured to reclassify the plurality of spoken utterances with new classifications based on at least two different classification operations.

13. The apparatus of claim 12 , wherein the at least two different classification operations comprise the process performing at least one of generating a list of the three highest classification confidence levels among the plurality of utterances, performing a Naïve-Bayes classification operation on the plurality of spoken utterances, and performing a decision tree classification on the plurality of spoken utterances.

14. A non-transitory computer readable storage medium configured to store instructions that when executed cause a processor to perform:

assigning each of a plurality of spoken utterances occurring during a call with a corresponding set of classifications;

determining a confidence rating associated with each of the plurality of spoken utterances and the assigned set of classifications; and

performing at least one of

reclassifying the plurality of spoken utterances with new classifications based on at least one additional classification operation, and

adding the assigned classifications and the corresponding plurality of spoken utterances to a training data set.

15. The non-transitory computer readable storage medium of claim 14 , wherein the plurality of spoken utterances are user spoken utterances responsive to automated questions presented to the user during the call.

16. The non-transitory computer readable storage medium of claim 14 , configured to store instructions that when executed cause the processor to perform transcribing the plurality of spoken utterances occurring during the call, wherein the transcribed spoken utterances comprises at least one of rejected spoken utterances, low confidence rated spoken utterances and disconfirmed spoken utterances.

17. The non-transitory computer readable storage medium of claim 14 , wherein if the determining the confidence rating associated with each of the plurality of spoken utterances produces confidence ratings that are above a predefined threshold then adding the labeled plurality of spoken utterances to the training data set.

18. The non-transitory computer readable storage medium of claim 14 , wherein if the determining the confidence rating associated with each of the plurality of spoken utterances produces confidence ratings that are below a predefined threshold then reclassifying the plurality of spoken utterances with new classifications based on at least two different classification operations.

19. The non-transitory computer readable storage medium of claim 18 , wherein the at least two different classification operations comprise at least one of generating a list of the three highest classification confidence levels among the plurality of utterances, performing a Naïve-Bayes classification operation on the plurality of spoken utterances, and performing a decision tree classification on the plurality of spoken utterances.

20. The non-transitory computer readable storage medium of claim 14 , wherein the processor is further configured to perform:

updating a production classifier used to perform classification operations by adding the new classifications to a training database; and

repeating the assigning of each of the plurality of spoken utterances with the corresponding set of classifications;

repeating the determining of the confidence rating associated with each of the plurality of spoken utterances and the assigned set of classifications; and

repeating the performing of the at least one of the

reclassifying the plurality of spoken utterances with the new classifications based on the at least one additional classification operation, and

adding the assigned classifications and the corresponding plurality of spoken utterances to the training data set.