Granted Patent B2

US 12,300,227 · App. 17/724,320 · Granted May 13, 2025

Inventors: Jackson Liscombe (New Marlborough, MA); Hardik Kothare (Burlingame, CA); Doug Habberstad (Savannah, GA); Andrew Cornish (Gore, NZ); Oliver Roesler (Weyhe, DE); Michael Neumann (Waiblingen, DE); David Pautler (San Francisco, CA); David Suendermann-Oeft (San Francisco, CA); Vikram Ramanarayanan (San Francisco, CA)

Assignee: Modality.AI

G10L15/22G10L25/66G10L25/84G10L25/93

Abstract

Claims (22)

1. A method of customizing a computer-generated dialog session for a user having a speech pathology, comprising:

eliciting speech from a user and corresponding facial movements from a user from a plurality of tasks during the computer-generated dialog session;

identifying, by a computer device, (a) a span of speech and (b) a span of non-speech in an audio stream of the user's speech, wherein the identification of the span of speech and span of non-speech comprises:

calculating, by the computing device, a value of a frame within the stream based on the logarithm of the root mean squared of the energy of the frame;

comparing, by the computing device, the value to a threshold;

marking, by the computing device, the frame as one of:

speech if the value meets the threshold; and

non-speech if the value is below the threshold;

altering, by the computing device, a plurality of the following configurable parameters: (a) a threshold minimum signal strength of the user's speech (dB) to consider as the start of the span of the user's speech; (b) an adjustment factor by which signal strengths of background noise increases between consecutive spans of the user's speech; (c) a threshold between signal strength during the span of speech and a signal strength during the span of non-speech; and

presenting, by the computing device, the customized dialog session by applying one of the plurality of altered parameters;

wherein the plurality of tasks comprise of: an open-ended question, a sustained vowel phonation, oral diadochokinesis alternating motion rate or a repetition of a pre-selected syllable, a speech intelligibility test sentence, a spontaneous speech while describing a pre-selected picture, or any combination thereof.

2. The method of claim 1 , further comprising

identifying, by the computing device, a start of speech in the audio stream as a function of the span of speech continuing for a first threshold period of time;

identifying, by the computing device, an end of speech in the audio stream as a function of the span of non-speech continuing for a second threshold period of time.

3. The method of claim 1 , further comprising beginning, by the computing device, the dialog session with a microphone check for speech and background noise.

4. The method of claim 1 , further identifying, by the computing device, a user as having a particular speech pathology, and using the identification to set a plurality of the configurable parameters.

5. The method of claim 4 , further comprising using, by the computing device, multiple questionnaires to identify progression of the particular speech pathology in the user.

6. The method of claim 1 , further comprising customizing, by the computing device, the configurable parameters as a function of a plurality of the following linguistic features: prosody, voice quality, articulation, acoustics, respiration, and cognitive/mental/emotional state.

7. The method of claim 1 , wherein the step of identifying the spans of speech and non-speech further comprises iterative re-estimation of speech sounds and background noise.

8. The method of claim 1 , wherein the step of identifying the spans of speech and non-speech further comprises determining, by the computing device, spans of speech to be those in which an average signal level of speech sounds (dB) exceeds an average signal level of the background noise (dB) by a threshold amount.

9. The method of claim 1 , further comprising calculating, by the computing device, a weighted penalty of proportions of false positive and false negative times, when compared to a hand annotation of actual speech in the audio stream.

10. The method of claim 1 , further comprising using a questionnaire to ascertain scores for at least three different domains affected by the speech pathology, bulbar, limb, or respiratory.

Assignments (1)

Continuity (2)

References Cited (56)