Automated Extraction of Semantic Content and Generation of a Structured Document from Speech
Techniques are disclosed for automatically generating structured documents based on speech, including identification of relevant concepts and their interpretation. In one embodiment, a structured document generator uses an integrated process to generate a structured textual document (such as a structured textual medical report) based on a spoken audio stream. The spoken audio stream may be recognized using a language model which includes a plurality of sub-models arranged in a hierarchical structure. Each of the sub-models may correspond to a concept that is expected to appear in the spoken audio stream. Different portions of the spoken audio stream may be recognized using different sub-models. The resulting structured textual document may have a hierarchical structure that corresponds to the hierarchical structure of the language sub-models that were used to generate the structured textual document.
1 . A computer-implemented method comprising steps of:
(A) identifying a probabilistic language model including a plurality of probabilistic language models associated with a plurality of concepts logically organized in a first hierarchy;
(B) using a speech recognition decoder to apply the probabilistic language model to a spoken audio stream to produce a document including content organized into a plurality of sub-structures logically organized in a second hierarchy having a logical structure defined by a path through the first hierarchy.
2 . The method of claim 1 , wherein the step (B) comprises a step of traversing the path through the first hierarchy to produce the document.
3 . The method of claim 1 , wherein the step (B) comprises a step of:
(B) (1) for each of a plurality of segments S of a spoken audio stream, performing steps of:
(a) recognizing segment S with at least two of the plurality of probabilistic language models to identify at least two candidate contents for segment S;
(b) selecting one of the at least two candidate contents as final content corresponding to segment S; and
(c) inserting the final content for segment S into the document sub-structure associated with the probabilistic language model which produced the candidate content selected in step (B) (1) (b).
4 . The method of claim 1 , wherein the plurality of probabilistic language models includes at least one n-gram language model.
5 . The method of claim 1 , wherein the plurality of probabilistic language models includes at least one finite state language model.
6 . The method of claim 5 , wherein the plurality of probabilistic language models includes at least one n-gram language model.
7 . The method of claim 1 , wherein the plurality of sub-structures includes a sub-structure representing a semantic concept.
8 . The method of claim 7 , wherein the semantic concept comprises a date.
9 . The method of claim 7 , wherein the semantic concept comprises a medication.
10 . The method of claim 7 , wherein the semantic concept is represented in the document in a computer-readable form.
11 . The method of claim 1 , further comprising a step of:
(C) rendering the document to produce a rendition indicating the structure of the document.
12 . The method of claim 1 , wherein the step (B) comprises steps of:
(B) (1) identifying a path through the first hierarchy;
(B) (2) generating a document having a structure corresponding to the path identified in step (B) (1).
13 . The method of claim 12 , wherein the step (B) (1) comprises a step of identifying a path through the first hierarchy which, when applied by a speech recognition decoder to recognize the spoken audio stream, produces an optimal recognition result with respect to the first hierarchy of the plurality of probabilistic language models.
14 . The method of claim 1 , wherein the speech recognition decoder includes a plurality of speech recognition decoders, and wherein the step (B) includes steps of:
(B) (1) identifying a segment of the spoken audio stream;
(B) (2) identifying one of the plurality of probabilistic language models;
(B) (3) identifying one of the plurality of speech recognition decoders having an association with the identified one of the plurality of probabilistic language models; and
(B) (4) using the identified speech recognition decoder to apply the identified probabilistic language model to the identified segment to produce content.
15 . The method of claim 14 , wherein the identified one of the plurality of probabilistic language models comprises an n-gram language model, and wherein the identified speech recognition decoder comprises an n-gram speech recognition decoder.
16 . The method of claim 14 , wherein the identified one of the plurality of probabilistic language models comprises a context-free grammar, and wherein the identified speech recognition decoder comprises a context-free grammar speech recognition decoder.
17 . The method of claim 1 , wherein step (B) comprises steps of:
(B) (1) identifying a mapping between the plurality of probabilistic language models and a plurality of segments in the audio stream;
(B) (2) for each of the plurality of segments, performing steps of:
(B) (2) (a) identifying a corresponding one of the plurality of probabilistic language models using the mapping;
(B) (2) (b) identifying one of the plurality of sub-structures associated with the identified probabilistic language model;
(B) (3) (b) using the speech recognition decoder to recognize segment using the identified one of the probabilistic language models thereby to produce content in the identified sub-structure.
18 . The method of claim 17 , wherein the steps (B) (1) and (B) (2) are performed at least in part concurrently.
19 . The method of claim 1 , wherein the step (B) comprises steps of:
(B) (1) identifying a portion of the spoken audio stream representing semantic information; and
(B) (2) storing a representation of the semantic information in the document in a machine-readable form.
20 . An apparatus comprising:
identification means for identifying a probabilistic language model including a plurality of probabilistic language models associated with a plurality of concepts logically organized in a first hierarchy; and
document production means for using a speech recognition decoder to apply the probabilistic language model to a spoken audio stream to produce a document including content organized into a plurality of sub-structures logically organized in a second hierarchy having a logical structure defined by a path through the first hierarchy.
21 . The apparatus of claim 20 , wherein the document production means comprises means for traversing the path through the first hierarchy to produce the document.
22 . The apparatus of claim 20 , wherein the document production means comprises:
iteration means comprising, for each of a plurality of segments S of a spoken audio stream:
means for recognizing segment S with at least two of the plurality of probabilistic language models to identify at least two candidate contents for segment S;
first selection means for selecting one of the at least two candidate contents as final content corresponding to segment S; and
insertion means for inserting the final content for segment S into the document sub-structure associated with the probabilistic language model which produced the candidate content selected by the first selection means.
23 . The apparatus of claim 20 , wherein the plurality of probabilistic language models includes at least one n-gram language model.
24 . The apparatus of claim 20 , wherein the plurality of probabilistic language models includes at least one finite state language model.
25 . The apparatus of claim 24 , wherein the plurality of probabilistic language models includes at least one n-gram language model.
26 . The apparatus of claim 20 , wherein the plurality of sub-structures includes a sub-structure representing a semantic concept.
27 . The apparatus of claim 26 , wherein the semantic concept comprises a date.
28 . The apparatus of claim 26 , wherein the semantic concept comprises a medication.
29 . The apparatus of claim 26 , wherein the semantic concept is represented in the document in a computer-readable form.
30 . The apparatus of claim 20 , further comprising:
means for rendering the document to produce a rendition indicating the structure of the document.
31 . The apparatus of claim 20 , wherein the document production means comprises:
second identification means for identifying a path through the first hierarchy; and
means for generating a document having a structure corresponding to the path identified by the second identification means.
32 . The apparatus of claim 31 , wherein the second identification means comprises means for identifying a path through the first hierarchy which, when applied by a speech recognition decoder to recognize the spoken audio stream, produces an optimal recognition result with respect to the first hierarchy of the plurality of probabilistic language models.
33 . The apparatus of claim 20 , wherein the speech recognition decoder includes a plurality of speech recognition decoders, and wherein the document production means includes:
means for identifying a segment of the spoken audio stream;
means for identifying one of the plurality of probabilistic language models;
means for identifying one of the plurality of speech recognition decoders having an association with the identified one of the plurality of probabilistic language models; and
means for using the identified speech recognition decoder to apply the identified probabilistic language model to the identified segment to produce content.
34 . The apparatus of claim 33 , wherein the identified one of the plurality of probabilistic language models comprises an n-gram language model, and wherein the identified speech recognition decoder comprises an n-gram speech recognition decoder.
35 . The apparatus of claim 33 , wherein the identified one of the plurality of probabilistic language models comprises a context-free grammar, and wherein the identified speech recognition decoder comprises a context-free grammar speech recognition decoder.
36 . The apparatus of claim 20 , wherein the document production means comprises:
second identification means for identifying a mapping between the plurality of probabilistic language models and a plurality of segments in the audio stream;
iteration means comprising, for each of the plurality of segments:
means for identifying a corresponding one of the plurality of probabilistic language models using the mapping;
means for identifying one of the plurality of sub-structures associated with the identified probabilistic language model; and
means for using the speech recognition decoder to recognize segment using the identified one of the probabilistic language models thereby to produce content in the identified sub-structure.
37 . The apparatus of claim 36 , wherein the second identification means and the iteration means are configured to operate at least in part concurrently.
38 . The apparatus of claim 20 , wherein the document production means comprises:
means for identifying a portion of the spoken audio stream representing semantic information; and
means for storing a representation of the semantic information in the document in a machine-readable form.