Augmentation and calibration of output from non-deterministic text generators by modeling its characteristics in specific environments
View Patent ↗Outputs of an automatic probabilistic event detection system, such as a fact extraction system, a speech-to-text engine or an automatic character recognition system, are matched with comparable results produced manually or by a different system. This comparison allows statistical modeling of the run-time behavior of the event detection system. This model can subsequently be used to give supplemental or replacement data for an output sequence of the system. In particular, the model can effectively calibrate the system for use with data of a particular statistical nature.
1. A computer-implemented method of processing outputs of an automatic system for probabilistic detection of events, comprising:
collecting statistics at a training system executing on a computing device, the statistics related to observed outputs of the automatic system comprising:
providing at least one input sequence to the automatic system, the input sequence associated with a transcript;
observing an output sequence of a text generating system executing on a computing device of the automatic system generated in response to the provided at least one input sequence; and
comparing the output sequence with the transcript; and
using the statistics at the training system to process automatically an original output sequence of the automatic system and produce an alternate output sequence and a confidence assessment regarding parts of at least one of the original output sequence and the alternate output sequence, the process including at least one of automatically supplementing and replacing at least part of the original output sequence with the alternate output sequence in accordance with the confidence assessment.
2. A computer-implemented method as recited in claim 1 , wherein at least part of the alternate output sequence contains information that can be used by systems that can use the at least part of the original output sequence directly.
3. A computer-implemented method as recited in claim 2 , wherein data in the alternate output sequence includes confidence assessments regarding parts of at least one of the original and alternate output sequences, where the confidence assessments supplement data in the original output sequence.
4. A computer-implemented method as recited in claim 1 , wherein the alternate output sequence includes information of a plurality of alternatives that can replace at least part of the original output sequence that can be used by systems that can use the at least part of the original output sequence directly.
5. A computer-implemented method as recited in claim 4 , wherein data in the alternate output sequence includes confidence assessments regarding parts of the alternatives, where the confidence assessments supplement data in the original output sequence.
6. A computer-implemented method as recited in claim 4 , wherein data in the alternate output sequence includes confidence assessments regarding parts of the alternatives, where the confidence assessments replace at least part of the original output sequence.
7. A computer-implemented method as recited in claim 1 , wherein the detected events involve word recognition.
8. A computer-implemented method as recited in claim 7 , wherein the automatic system is an automatic speech recognition system.
9. A computer-implemented method as recited in claim 8 ,
wherein the automatic speech recognition system operates on low-grade audio signals having word recognition precision below 50 percent; and
wherein said method further comprises utilizing human transcription of the low-grade audio signals as a source for data relating to the statistics being collected.
10. A computer-implemented method as recited in claim 7 , wherein the automatic probabilistic event detection system is an automatic character recognition system.
11. A computer-implemented method as recited in claim 7 , wherein the alternate output sequence includes at least one of
an alternate recognition score for at least one of the words,
at least one alternate word that may have been one detectable event that transpired,
the at least one alternate word along with a recognition score for the at least one alternate word,
at least one alternate sequence of words that may have been another detectable event that transpired,
the at least one alternate sequence of words along with a recognition score for at least one word that is part of the at least one alternate sequence of words,
an indication that no detectable event has transpired,
a word lattice describing a plurality of alternatives for detectable word sequences, and
the word lattice along with a recognition score for at least one among
at least one word in the detectable word sequences,
at least one path in the word lattice, and
at least one edge in the word lattice.
12. A computer-implemented method as recited in claim 1 , wherein said using comprises:
building a first model modeling behavior of the automatic system as a process with at least one inner state, which may be unrelated to inner states of the automatic system, and inferring the at least one inner state of the process from the observed outputs of the automatic system;
building a second model, based on the statistics obtained by said collecting, to infer data to at least one of supplement and replace at least part of the original output sequence from the at least one inner state of the process in the first model;
combining the first and second models to form a function for converting the original output sequence into the alternate output sequence; and
using the function on the original output sequence of the automatic system to create the alternate output sequence.
13. A computer-implemented method as recited in claim 12 , further comprising repeating said using of the function on different original output sequences of the automatic system to create additional alternate output sequences.
14. A computer-implemented method as recited in claim 12 , wherein the process in said first model is one of a Generalized Hidden Markov process and a special case of a Generalized Hidden Markov process.
15. A computer-implemented method as recited in claim 12 ,
wherein the second model is a parametric model, and
wherein said building of the second model uses at least one direct parametric estimation technique for inferring from at least one of the inner states.
16. A computer-implemented method as recited in claim 15 , wherein the at least one direct parametric estimation technique includes at least one of maximal likelihood estimation and entropy maximization.
17. A computer-implemented method as recited in claim 12 , wherein for at least one of the inner states said building of the second model uses at least one estimation technique utilizing information estimated for other inner states.
18. A computer-implemented method as recited in claim 17 , wherein the at least one estimation technique utilizes at least one of a mixture model and kernel-based learning.
19. A computer-implemented method as recited in claim 12 , wherein said building of the first and second models assumes the inner states of the process to be fully determined by the observed outputs during at least one point in time.
20. A computer-implemented method as recited in claim 19 , wherein said building of the first and second models assumes the inner states of the process during at least one point in time to be fully determined by a subset of the observed outputs that includes at least an identity of at least one event detected by the automatic system.
21. A computer-implemented method as recited in claim 12 , wherein said building of at least one of the first and second models uses at least one discretization function.
22. A computer-implemented method as recited in claim 12 , wherein at least one of said building and combining uses Bayesian methods.
23. A computer-implemented method as recited in claim 1 , further comprising repeating said collecting on several statistically different training materials.
24. A computer-implemented method as recited in claim 23 , wherein said collecting uses samples of statistically different sets of materials as initial training material.
25. A computer-implemented method as recited in claim 23 , further comprising identifying parameters that remain invariant between the statistically different sets of materials.
26. A computer-implemented method as recited in claim 25 , wherein said identifying improves estimation of at least one of the parameters.
27. A computer-implemented method as recited in claim 25 , wherein said identifying is used to enable training when available statistically self-similar sets of materials are too small to allow effective training.
28. A computer-implemented method as recited in claim 25 , wherein said identifying is used to increase effectiveness of further training on material that is not statistically similar to initial training material.
29. A computer-implemented method as recited in claim 1 , wherein material used for said collecting is statistically similar to material used during said using.
30. At least one computer readable medium storing instructions for controlling at least one computer system to perform a method of processing outputs of an automatic system for probabilistic detection of events, comprising:
collecting statistics related to observed outputs of the automatic system comprising:
providing at least one input sequence to the automatic system, the input sequence associated with a transcript;
observing an output sequence of the automatic system generated in response to the provided at least one input sequence; and
comparing the output sequence with the transcript; and
using the statistics to process automatically an original output sequence of the automatic system and produce an alternate output sequence and a confidence assessment regarding parts of at least one of the original output sequence and the alternate output sequence, the process including at least one of automatically supplementing and replacing at least part of the original output sequence with the alternate output sequence in accordance with the confidence assessment.
31. At least one computer readable medium as recited in claim 30 , wherein at least part of the alternate output sequence contains information that can be used by systems that can use the at least part of the Original output sequence directly.
32. At least one computer readable medium as recited in claim 31 , wherein data in the alternate output sequence includes confidence assessments regarding parts of at least one of the Original and alternate output sequences, where the confidence assessments supplement data in the Original output sequence.
33. At least one computer readable medium as recited in claim 30 , wherein the alternate output sequence includes information of a plurality of alternatives that can replace at least part of the Original output sequence that can be used by systems that can use the at least part of the Original output sequence directly.
34. At least one computer readable medium as recited in claim 33 , wherein data in the alternate output sequence includes confidence assessments regarding parts of the alternatives, where the confidence assessments supplement data in the Original output sequence.
35. At least one computer readable medium as recited in claim 33 , wherein data in the alternate output sequence includes confidence assessments regarding parts of the alternatives, where the confidence assessments replace at least part of the Original output sequence.
36. At least one computer readable medium as recited in claim 30 , wherein the detected events involve word recognition.
37. At least one computer readable medium as recited in claim 36 , wherein the automatic system is an automatic speech recognition system.
38. At least one computer readable medium as recited in claim 37 ,
wherein the automatic speech recognition system operates on low-grade audio signals having word recognition precision below 50 percent; and
wherein said method further comprises utilizing human transcription of the low-grade audio signals as a source for data relating to the statistics being collected.
39. At least one computer readable medium as recited in claim 36 , wherein the automatic probabilistic event detection system is an automatic character recognition system.
40. At least one computer readable medium as recited in claim 36 , wherein the alternate output sequence includes at least one of
an alternate recognition score for at least one of the words,
at least one alternate word that may have been one detectable event that transpired,
the at least one alternate word along with a recognition score for the at least one alternate word,
at least one alternate sequence of words that may have been another detectable event that transpired,
the at least one alternate sequence of words along with a recognition score for at least one word that is part of the at least one alternate sequence of words,
an indication that no detectable event has transpired,
a word lattice describing a plurality of alternatives for detectable word sequences, and
the word lattice along with a recognition score for at least one among
at least one word in the detectable word sequences,
at least one path in the word lattice, and
at least one edge in the word lattice.
41. At least one computer readable medium as recited in claim 30 , wherein said using comprises:
building a first model modeling behavior of the automatic system as a process with at least one inner state, which may be unrelated to inner states of the automatic system, and inferring the at least one inner state of the process from the observed outputs of the automatic system;
building a second model, based on the statistics obtained by said collecting, to infer data to at least one of supplement and replace at least part of the Original output sequence from the at least one inner state of the process in the first model;
combining the first and second models to form a function for converting the Original output sequence into the alternate output sequence; and
using the function on the Original output sequence of the automatic system to create the alternate output sequence.
42. At least one computer readable medium as recited in claim 41 , further comprising repeating said using of the function on different Original output sequences of the automatic system to create additional alternate output sequences.
43. At least one computer readable medium as recited in claim 41 , wherein the process in said first model is one of a Generalized Hidden Markov process and a special case of a Generalized Hidden Markov process.
44. At least one computer readable medium as recited in claim 41 ,
wherein the second model is a parametric model, and
wherein said building of the second model uses at least one direct parametric estimation technique for inferring from at least one of the inner states.
45. At least one computer readable medium as recited in claim 44 , wherein the at least one direct parametric estimation technique includes at least one of maximal likelihood estimation and entropy maximization.
46. At least one computer readable medium as recited in claim 41 , wherein for at least one of the inner states said building of the second model uses at least one estimation technique utilizing information estimated for other inner states.
47. At least one computer readable medium as recited in claim 46 , wherein the at least one estimation technique utilizes at least one of a mixture model and kernel-based learning.
48. At least one computer readable medium as recited in claim 41 , wherein said building of the first and second models assumes the inner states of the process to be fully determined by the observed outputs during at least one point in time.
49. At least one computer readable medium as recited in claim 48 , wherein said building of the first and second models assumes the inner states of the process during at least one point in time to be fully determined by a subset of the observed outputs that includes at least an identity of at least one event detected by the automatic system.
50. At least one computer readable medium as recited in claim 41 , wherein said building of at least one of the first and second models uses at least one discretization function.
51. At least one computer readable medium as recited in claim 41 , wherein at least one of said building and combining uses Bayesian methods.
52. At least one computer readable medium as recited in claim 30 , further comprising repeating said collecting on several statistically different training materials.
53. At least one computer readable medium as recited in claim 52 , wherein said collecting uses samples of statistically different sets of materials as initial training material.
54. At least one computer readable medium as recited in claim 53 , further comprising identifying parameters that remain invariant between the statistically different sets of materials.
55. At least one computer readable medium as recited in claim 54 , wherein said identifying improves estimation of at least one of the parameters.
56. At least one computer readable medium as recited in claim 54 , wherein said identifying is used to enable training when available statistically self-similar sets of materials are too small to allow effective training.
57. At least one computer readable medium as recited in claim 54 , wherein said identifying is used to increase effectiveness of further training on material that is not statistically similar to initial training material.
58. At least one computer readable medium as recited in claim 30 , wherein material used for said collecting is statistically similar to material used during said using.
59. An apparatus for processing outputs of an automatic system for probabilistic detection of events, comprising:
collection means for collecting statistics related to observed outputs of the automatic system comprising:
providing means for providing at least one input sequence to the automatic system, the input sequence associated with a transcript;
observing means for observing an output sequence of the automatic system generated in response to the provided at least one input sequence; and
comparing means for comparing the output sequence with the transcript; and
processing means for using the statistics to process automatically an Original output sequence of the automatic system and produce an alternate output sequence and a confidence assessment regarding parts of at least one of the Original output sequence and the alternate output sequence, the process including at least one of automatically supplementing and replacing at least part of the original Output sequence with the alternate output sequence in accordance with the confidence assessment.
60. An apparatus as recited in claim 59 , wherein the alternate output sequence includes information of a plurality of alternatives that can replace at least part of the Original output sequence that can be used by systems that can use the at least part of the Original output sequence directly.
61. An apparatus as recited in claim 60 , wherein data in the alternate output sequence includes confidence assessments regarding parts of the alternatives, where the confidence assessments supplement data in the Original output sequence.
62. An apparatus as recited in claim 59 , wherein the detected events involve word recognition.
63. An apparatus as recited in claim 62 , wherein the automatic system is an automatic speech recognition system.
64. An apparatus as recited in claim 62 , wherein the alternate output sequence includes at least one of
an alternate recognition score for at least one of the words,
at least one alternate word that may have been one detectable event that transpired,
the at least one alternate word along with a recognition score for the at least one alternate word,
at least one alternate sequence of words that may have been another detectable event that transpired,
the at least one alternate sequence of words along with a recognition score for at least one word that is part of the at least one alternate sequence of words,
an indication that no detectable event has transpired,
a word lattice describing a plurality of alternatives for detectable word sequences, and
the word lattice along with a recognition score for at least one among
at least one word in the detectable word sequences,
at least one path in the word lattice, and
at least one edge in the word lattice.
65. An apparatus as recited in claim 59 , wherein said processing means comprises:
first model means for building a first model modeling behavior of the automatic system as a process with at least one inner state, which may be unrelated to inner states of the automatic system, and inferring the at least one inner state of the process from the observed outputs of the automatic system;
second model means for building a second model, based on the statistics obtained by said collection means, to infer data to at least one of supplement and replace at least part of the Original output sequence from the at least one inner state of the process in the first model;
combination means for combining the first and second models to form a function for converting the Original output sequence into the alternate output sequence; and
function means for applying the function to the Original output sequence of the automatic system to create the alternate output sequence.
66. An apparatus as recited in claim 65 , wherein said function means applies the function on different Original output sequences of the automatic system to create additional alternate output sequences.
67. An apparatus as recited in claim 65 , wherein the process in said first model is one of a Generalized Hidden Markov process and a special case of a Generalized Hidden Markov process.
68. An apparatus as recited in claim 65 ,
wherein the second model is a parametric model, and
wherein said second model means uses at least one direct parametric estimation technique for inferring from at least one of the inner states.
69. An apparatus as recited in claim 65 , wherein said second model means, for at least one of the inner states, uses at least one estimation technique utilizing information estimated for other inner states.
70. An apparatus as recited in claim 69 , wherein the at least one estimation technique utilizes at least one of a mixture model and kernel-based learning.
71. A system for processing outputs of an automatic system for probabilistic detection of events, comprising:
an interface to receive observed outputs from the automatic system; and
at least one processor programmed to:
collect statistics related to the observed outputs of the automatic system by:
providing at least one input sequence to the automatic system, the input sequence associated with a transcript;
observing an output sequence of the automatic system generated in response to the provided at least one input sequence; and
comparing the output sequence with the transcript; and
use the statistics to automatically produce an alternate output sequence and a confidence assessment regarding parts of at least one of the Original output sequence and the alternate output sequence, thereafter at least one of automatically supplementing and replacing at least part of the Original output sequence of the automatic system with the alternate output sequence in accordance with the confidence assessment.
72. A system as recited in claim 71 , wherein at least part of the alternate output sequence includes information of a plurality of alternatives that can replace at least part of the Original output sequence that can be used by systems that can use the at least part of the Original output sequence directly.
73. A system as recited in claim 71 , wherein data in the alternate output sequence includes confidence assessments regarding parts of the alternatives, where the confidence assessments supplement data in the Original output sequence.
74. A system as recited in claim 71 , wherein the detected events involve word recognition.
75. A system as recited in claim 74 , wherein the automatic system is an automatic speech recognition system.
76. A system as recited in claim 75 ,
wherein the automatic speech recognition system operates on low-grade audio signals having word recognition precision below 50 percent; and
wherein said interface further receives human transcription of the low-grade audio signals for use by said processor as data relating to the statistics being collected.
77. A system as recited in claim 74 , wherein the alternate output sequence includes at least one of
an alternate recognition score for at least one of the words,
at least one alternate word that may have been one detectable event that transpired,
the at least one alternate word along with a recognition score for the at least one alternate word,
at least one alternate sequence of words that may have been another detectable event that transpired,
the at least one alternate sequence of words along with a recognition score for at least one word that is part of the at least one alternate sequence of words,
an indication that no detectable event has transpired,
a word lattice describing a plurality of alternatives for detectable word sequences, and
the word lattice along with a recognition score for at least one among
at least one word in the detectable word sequences,
at least one path in the word lattice, and
at least one edge in the word lattice.
78. A system as recited in claim 71 , wherein said processor is programmed to build a first model modeling behavior of the automatic system as a process with at least one inner state, which may be unrelated to inner states of the automatic system, and inferring the at least one inner state of the process from the observed outputs of the automatic system, to build a second model, based on the statistics obtained, to infer data to at least one of supplement and replace at least part of the Original output sequence from the at least one inner state of the process in the first model, to combine the first and second models to form a function for converting the Original output sequence into the alternate output sequence, and to apply the function to the Original output sequence of the automatic system to create the alternate output sequence.
79. A system as recited in claim 78 , wherein said processor applies the function on different Original output sequences of the automatic system to create additional alternate output sequences.
80. A system as recited in claim 78 , wherein the process in said first model is one of a Generalized Hidden Markov process and a special case of a Generalized Hidden Markov process.
81. A system as recited in claim 78 ,
wherein the second model is a parametric model, and
wherein said processor builds the second model using at least one direct parametric estimation technique for inferring from at least one of the inner states.
82. A system as recited in claim 78 , wherein said processor, for at least one of the inner states, uses at least one estimation technique utilizing information estimated for other inner states.
83. A system as recited in claim 82 , wherein said processor is programmed to utilize at least one of a mixture model and kernel-based learning as the at least one estimation technique.