Multi-party conversation analyzer and logger
A multi-party conversation analyzer and logger uses a variety of techniques including spectrographic voice analysis, absolute loudness measurements, directional microphones, and telephonic directional separation to determine the number of parties who take part in a conversation, and segment the conversation by speaking party. In one aspect, the invention monitors telephone conversations in real time to detect conditions of interest (for instance, calls to non-allowed parties or calls of a prohibited nature from prison inmates). In another aspect, automated prosody measurement algorithms are used in conjunction with speaker segmentation to extract emotional content of the speech of participants within a particular conversation, and speaker interactions and emotions are displayed in graphical form. A conversation database is generated which contains conversation recordings, and derived data such as transcription text, derived emotions, alert conditions, and correctness probabilities associated with derived data. Investigative tools allow flexible queries of the conversation database.
1. A method for monitoring the emotional states of conversation participants, the method comprising:
monitoring a conversation between a plurality of participants;
determining a first emotional state of a first participant of the plurality of participants during a first segment of the monitored conversation;
determining a second emotional state of the first participant during a second segment of the monitored conversation;
displaying a timeline of the monitored conversation, wherein the timeline indicates the determined first emotional state of the first participant during the first segment of the monitored conversation and further indicates the determined second emotional state of the first participant during the second segment of the monitored conversation, and wherein the displayed timeline is operable by a user to initially display indications of the first emotional state during the first segment and to subsequently display indications of the second emotional state during the second segment.
2. The method of claim 1 , wherein the determined first emotional state comprises a first type of emotion, and wherein the determined second emotional state comprises a second type of emotion.
3. The method of claim 2 , wherein the first type of emotion is indicated on the timeline by a first color hue, and wherein the second type of emotional is indicated on the timeline by a second color hue.
4. The method of claim 1 , wherein the determined first emotional state comprises a first intensity of emotion, and wherein the determined second emotional state comprises a second intensity of emotion.
5. The method of claim 4 , wherein the determined first emotional intensity is indicated on the timeline by a first color saturation, and wherein the determined second emotional intensity is indicated on the timeline by a second color saturation.
6. The method of claim 1 , wherein the first emotional state is determined based on the detection of first prosodic indicators in the speech of the first participant in the monitored conversation, wherein the second emotional state is determined based on the detection of second prosodic indicators in the speech of the first participant.
7. The method of claim 6 , wherein at least one of the first prosodic indicators and the second prosodic indicators comprise a loudness, rate of speaking, and pauses in the monitored conversation.
8. The method of claim 1 , wherein the displayed timeline comprises an additional timeline for the display of determined emotional states for a second participant of the plurality of participants during the first segment and the second segment of the monitored conversation.
9. The method of claim 1 , wherein the determined first emotional state is indicated using a first emotionally indicative icon, and wherein the second determined emotional state is indicated using a second emotionally indicative icon.
10. The method of claim 9 , wherein a color hue and color intensity of at least one of the first indicative icon and the second indicative icon are varied.
11. A system for monitoring the emotional states of conversation participants, the system comprising:
a processor; and
a memory coupled to the processor, the memory storing computer-readable instructions that, upon execution by the processor, cause the system to:
monitor a conversation between a plurality of participants;
determine a first emotional state of a first participant of the plurality of participants during a first segment of the monitored conversation;
determine a second emotional state of the first participant during a second segment of the monitored conversation;
display a timeline of the monitored conversation, wherein the timeline indicates the determined first emotional state of the first participant during the first segment of the monitored conversation and further indicates the determined second emotional state of the first participant during the second segment of the monitored conversation, and wherein the displayed timeline is operable by a user to initially display indications of the first emotional state during the first segment and to subsequently display indications of the second emotional state during the second segment.
12. The system of claim 11 , wherein the determined first emotional state comprises a first type of emotion, and wherein the determined second emotional state comprises a second type of emotion.
13. The system of claim 12 , wherein the determined first type of emotion is indicated on the timeline by a first color hue, and wherein the second type of emotion is indicated on the timeline by a second color hue.
14. The system of claim 11 , wherein the determined first emotional state comprises a first intensity of emotion, and wherein the determined second emotional state comprises a second intensity of emotion.
15. The system of claim 14 , wherein the determined first emotional intensity is indicated on the timeline by a first color saturation, and wherein the determined second emotional intensity is indicated on the timeline by a second color saturation.
16. The system of claim 11 , wherein the first emotional state is determined based on the detection of first prosodic indicators in the speech of a first participant in the monitored conversation, wherein the second emotional state is determined based on the detection of second prosodic indicators in the speech of the first participant.
17. The system of claim 16 , wherein at least one of the first prosodic indicators and the second prosodic indicators comprise a loudness, rate of speaking, and pauses in the monitored conversation.
18. The system of claim 11 , wherein the displayed timeline comprises an additional timeline for the display of determined emotion states for a second participant of the plurality of participants during the first segment and the second segment of the monitored conversation.
19. The system of claim 11 , wherein the determined first emotional state is indicated using a first emotionally indicative icon, and wherein the second determined emotional state is indicated using a second emotionally indicative icon.
20. The system of claim 19 , wherein a color hue and color intensity of at least one of the first emotionally indicative icon and the second indicative icon are varied.