Granted Patent B2

US 12,254,870 · App. 18/033,243 · Granted Mar 18, 2025

Inventors: Julie A. Van Dyke (New Haven, CT); Michael Gorman (Edina, MN); Mark Lacek (Minneapolis, MN)

Assignee: Cascade Reading, Inc.

G10L15/1822G10L13/02G10L15/02G10L15/05G10L25/90G10L25/93

Abstract

Claims (75)

1. A method for generating a text arrangement from acoustic input, comprising:

obtaining an audio sample that includes multiple speech segments providing words from human speech;

identifying acoustic properties of the words from the audio sample using an acoustic analyzer provided by an audio processing service, wherein the acoustic properties are identified from an acoustic signal of the audio sample, and wherein the acoustic analyzer extracts at least one waveform from the acoustic signal;

computing respective measurements of the acoustic properties for each of the words, using the acoustic analyzer, based on direct measurement of the acoustic signal or comparisons within the acoustic signal captured from the at least one waveform;

generating an acoustic language model in a computer-readable data structure to represent a linguistic relationship of the words, based on the acoustic properties of the words from the audio sample, wherein the linguistic relationship of the words is determined from the respective measurements of the acoustic properties; and

outputting data to arrange the words into a cascade format, based on the linguistic relationship, wherein the cascade format establishes horizontal displacement and vertical displacement among the words.

2. The method of claim 1 , wherein the linguistic relationship of the words is determined by applying at least one acoustically-defined cuing rule to respective words, based on the acoustic properties of the respective words.

3. The method of claim 1 , wherein the acoustic properties of the words include at least one of: pitch values; intensity; duration; or pauses between the words.

4. The method of claim 3 , wherein a respective property of the acoustic properties is used to determine constituency and dependency of respective words in at least one sentence of the words.

5. The method of claim 1 , further comprising:

obtaining a text sample that includes a text string of the words;

aligning the text string to the at least one waveform; and

identifying the words from the audio sample based on the aligning.

6. The method of claim 5 , further comprising:

performing speech recognition on the audio sample to generate the text sample.

7. The method of claim 6 , wherein the speech recognition identifies boundaries of multiple sentences among the words.

8. The method of claim 5 , wherein the linguistic relationship of the words is determined using constituency parsing and dependency parsing of the text string.

9. The method of claim 8 , wherein the constituency parsing generates constituency data for the text string that defines constituents of the text string based on an X-bar schema, and

wherein to arrange the words into the cascade format includes use of the constituency data.

10. The method of claim 8 , wherein the dependency parsing generates dependency data for the text string that defines dependencies of the text string based on sentence structure, and

wherein to arrange the words into the cascade format includes use of the dependency data.

11. The method of claim 1 , wherein computing the respective measurements of the acoustic properties for each of the words includes computing measurement values for pitch, slope, normalized duration, and normalized intensity for each of the words, wherein the computed measurement values are derived from the acoustic properties of the words; and

wherein the linguistic relationship among each of the words is determined using the computed measurement values for each of the words.

12. The method of claim 11 , wherein computing the respective measurements of the acoustic properties for each of the words includes computing measurement values for pause durations between words, pitch slope changes, word elongations, and intensity changes, between at least two of the words; and

wherein the linguistic relationship among each of the words is determined using the computed measurement values between the at least two of the words.

13. The method of claim 1 , further comprising:

displaying feedback for the audio sample, based on comparison of the cascade format to a target cascade format, the target cascade format generated from linguistically-driven text formatting.

14. The method of claim 1 , further comprising:

causing a display of the words in the cascade format, wherein the horizontal displacement is provided in the display from the use of indentation, and wherein the vertical displacement is provided in the display from the use of line breaks.

15. The method of claim 1 , further comprising:

generating a new audio sample, based on the cascade format, using a voice synthesis engine.

16. A non-transitory computer-readable storage medium comprising instructions, wherein the instructions, when executed by a computing device, cause the computing device to perform operations comprising:

obtaining an audio sample that includes multiple speech segments providing words from human speech;

identifying acoustic properties of the words from the audio sample using an acoustic analyzer provided by an audio processing service, wherein the acoustic properties are identified from an acoustic signal of the audio sample, and wherein the acoustic analyzer extracts at least one waveform from the acoustic signal;

computing respective measurements of the acoustic properties for each of the words, using the acoustic analyzer, based on direct measurement of the acoustic signal or comparisons within the acoustic signal captured from the at least one waveform;

generating an acoustic language model in a computer-readable data structure to represent a linguistic relationship of the words, based on the acoustic properties of the words from the audio sample, wherein the linguistic relationship of the words is determined from the respective measurements of the acoustic properties; and

outputting data to arrange the words into a cascade format, based on the linguistic relationship, wherein the cascade format establishes horizontal displacement and vertical displacement among the words.

17. The non-transitory computer-readable storage medium of claim 16 , the operations further comprising:

obtaining a text sample that includes a text string of the words;

aligning the text string to the at least one waveform; and

identifying the words from the audio sample based on the aligning.

18. The non-transitory computer-readable storage medium of claim 17 , the operations further comprising:

performing speech recognition on the audio sample to generate the text sample;

wherein the speech recognition identifies boundaries of multiple sentences among the words.

19. The non-transitory computer-readable storage medium of claim 17 ,

wherein the linguistic relationship of the words is determined using constituency parsing and dependency parsing of the text string;

wherein the constituency parsing generates constituency data for the text string that defines constituents of the text string based on an X-bar schema, and wherein to arrange the words into the cascade format includes use of the constituency data;

wherein the dependency parsing generates dependency data for the text string that defines dependencies of the text string based on sentence structure, and

wherein to arrange the words into the cascade format includes use of the dependency data.

20. The non-transitory computer-readable storage medium of claim 16 , wherein computing the respective measurements of the acoustic properties for each of the words includes computing measurement values for pitch, slope, normalized duration, and normalized intensity for each of the words, wherein the computed measurement values are derived from the acoustic properties of the words;

wherein the acoustic properties of the words include at least one of: pitch values; intensity; duration; or pauses between the words; and

wherein the linguistic relationship among each of the words is determined using the computed measurement values for each of the words.

21. A device, comprising:

processing circuitry; and

a memory device including instructions embodied thereon, wherein the instructions, which when executed by the processing circuitry, configure the processing circuitry to:

obtain an audio sample that includes multiple speech segments providing words from human speech;

identify acoustic properties of the words from the audio sample using an acoustic analyzer provided by an audio processing service, wherein the acoustic properties are identified from least one waveform an acoustic signal of the audio sample, and wherein the acoustic analyzer extracts at least one waveform from the acoustic signal;

compute respective measurements of the acoustic properties for each of the words, using the acoustic analyzer, based on direct measurement of the acoustic signal or comparisons within the acoustic signal captured from the at least one waveform;

generate an acoustic language model in a computer-readable data structure to represent a linguistic relationship of the words, based on the acoustic properties of the words from the audio sample, wherein the linguistic relationship of the words is determined from the respective measurements of the acoustic properties; and

output data to arrange the words into a cascade format, based on the linguistic relationship, wherein the cascade format establishes horizontal displacement and vertical displacement among the words.

22. The device of claim 21 , wherein the instructions further configure the processing circuitry to:

obtain a text sample that includes a text string of the words;

align the text string to the at least one waveform; and

identify the words from the audio sample based on the aligning.

23. The device of claim 22 , wherein the instructions further configure the processing circuitry to:

perform speech recognition on the audio sample to generate the text sample;

wherein the speech recognition identifies boundaries of multiple sentences among the words.

24. The device of claim 22 ,

wherein the linguistic relationship of the words is determined using constituency parsing and dependency parsing of the text string;

wherein the constituency parsing generates constituency data for the text string that defines constituents of the text string based on an X-bar schema, and wherein to arrange the words into the cascade format includes use of the constituency data;

wherein the dependency parsing generates dependency data for the text string that defines dependencies of the text string based on sentence structure, and

wherein to arrange the words into the cascade format includes use of the dependency data.

25. The device of claim 21 , wherein to compute the respective measurements of the acoustic properties for each of the words includes to compute measurement values for pitch, slope, normalized duration, and normalized intensity for each of the words, wherein the computed measurement values are derived from the acoustic properties of the words;

wherein the acoustic properties of the words include at least one of: pitch values; intensity; duration; or pauses between the words; and

wherein the linguistic relationship among each of the words is determined using the computed measurement values for each of the words.

Assignments (1)

Continuity (2)

References Cited (209)

Cited By (1)

US 12,456,002