Methods and systems for assessing and improving the performance of a speech recognition system

1. A method for assessing a performance of a speech recognition system, comprising:

using at least one processor, determining a grade, corresponding to either recognition of instances of a word or recognition of instances of various words among a set of words, wherein the grade indicates a level of the performance of the system; and

the grade is based on a recognition rate determined from a plurality of recognition observations and at least one recognition factor.

2. The method of claim 1 , wherein the grade is a letter, score, number, color, bar having a certain length, or symbol, indicating a rating.

3. The method of claim 1 , the grade is based on a numerical score for the recognition of a word.

4. The method of claim 3 , further comprising:

determining the numerical score;

comparing the numerical score to a set of predetermined scores that indicate possible performance levels; and

assigning to the numerical score the grade.

5. The method of claim 1 , wherein the recognition rate is an error rate based on estimated errors estimated from evaluating system behavior.

6. The method of claim 1 , wherein the recognition rate is an accuracy rate based on estimated correct observations by the system estimated from evaluating system behavior.

7. The method of claim 5 , wherein the recognition rate is determined, in part, using:

a count of occurrences for which instances of the word or instances of the various words are recognized within a certain confidence factor range.

8. The method of claim 5 , wherein the recognition rate is determined, in part, using:

a count of occurrences for which the system rejects a first hypothesis of a first utterance, followed by a user repeating the first utterance in a second utterance, followed by the system accepting a second hypothesis of the second utterance.

9. The method of claim 5 , wherein the recognition rate is determined, in part, using:

a count of occurrences when instances of the word or one of the various words are recognized incorrectly in the system's hypothesis for a first utterance, followed by a user repeating the first utterance in a second utterance, followed by the system recognizing and accepting the hypothesis for the second utterance.

10. The method of claim 5 , wherein the recognition rate is determined, in part, using:

a count of occurrences when a hypothesis generated by the system does not match an expected response word-for-word, but the hypothesis mostly matches the expected response word-for-word.

11. The method of claim 5 , wherein the recognition rate is determined, in part, using:

a count of occurrences when a user provides feedback to the system.

12. The method of claim 1 , wherein the recognition factor is a number of words in a vocabulary of the system.

13. The method of claim 1 , wherein the recognition factor is an average number of words per utterance for utterances used in the determination of the grade.

14. The method of claim 1 , wherein the word or the various words is a digit or digits.

15. The method of claim 1 , wherein the recognition factor is an environmental factor.

16. The method of claim 15 , wherein the environmental factor is a measurement of the user's speech level.

17. The method of claim 15 , wherein the environmental factor is a background noise level.

18. The method of claim 1 , wherein the recognition factor is a measure of difficulty of recognition of the word or various words.

19. The method of claim 1 , wherein the grade is a measure of performance of a system or systems used by one or more users.

20. The method of claim 1 , wherein the recognition factor is based on performance of a system or systems used by one or more users.

21. The method of claim 1 , further comprising:

determining if the grade for a word or various words from a set of words fails to meet a predetermined standard and improvement is therefore needed; and

if the improvement is needed, determining appropriate corrective action.

22. The method of claim 21 , further comprising:

creating an alert to perform the corrective action through an alert mechanism.

23. The method of claim 1 , further comprising:

determining if the grade of the system is not as good as other like-generated grades of other like systems used by other users, wherein the other like systems are operated in an environment similar to the system's environment and improvement is therefore achievable; and

if the improvement is achievable, determining appropriate corrective action.

24. The method of claim 21 , further comprising creating an alert to communicate the corrective action to the user.

25. The method of claim 1 , further comprising:

generating a report of the performance of the system; and

displaying the report.

26. The method of claim 25 , wherein the report includes information about performances of multiple systems used by multiple users.

27. The method of claim 25 , further comprising displaying the report in a location other than on the speech recognition system.

28. The method of claim 1 , wherein the method is performed as the system is being used by a user.

29. The method of claim 1 , wherein the method is performed after the system is used by a user for a period of time, using information from a log of system activity.

30. An apparatus for assessing a performance of a speech recognition system, comprising:

a processor that is configured to determine a grade, corresponding to either recognition of instances of a word or recognition of instances of various words among a set of words, wherein the grade indicates a level of the performance of the system; and

the grade is based on a recognition rate determined from a plurality of recognition observations and at least one recognition factor.

31. The apparatus of claim 30 , further comprising:

a report generator that generates a report of the performance of the system; and

a display for displaying the report.

32. The apparatus of claim 31 , wherein the display is located at the system.

33. The apparatus of claim 31 , wherein the report includes information about performances of multiple systems used by multiple users.

34. The apparatus of claim 31 , wherein the display is at a location other than where the system and the other like systems are located.

35. The apparatus of claim 31 , wherein the display is a management console.

36. The apparatus of claim 31 , wherein the display is a web browser.

37. The apparatus of claim 30 , wherein the system is located on a portable terminal.

38. A method for assessing a performance of a speech recognition system, comprising:

using at least one processor, determining a grade, corresponding to either recognition of instances of a word or recognition of instances of various words among a set of words, wherein the grade indicates a level of the performance of the system; and

the grade is based on a count of a plurality of observations with errors or a count of a plurality of correct observations and at least one recognition factor.

39. An apparatus for assessing a performance of a speech recognition system, comprising:

a processor that is configured to determine a grade, corresponding to either recognition of instances of a word or recognition of instances of various words among a set of words, wherein the grade indicates a level of the performance of the system; and

the grade is based on a count of a plurality of observations with errors or a count of a plurality of correct observations and at least one recognition factor.

40. The method of claim 5 , wherein the error rate comprises:

a substitution rate at which an instance of the word or one of the various words is incorrectly recognized as another word, when a hypothesis generated by the system is compared to an expected response.