System and method for multi-modal audio mining of telephone conversations
A system and method for the automated monitoring of inmate telephone calls as well as multi-modal search, retrieval and playback capabilities for said calls. A general term for such capabilities is multi-modal audio mining. The invention is designed to provide an efficient means for organizations such as correctional facilities to identify and monitor the contents of telephone conversations and to provide evidence of possible inappropriate conduct and/or criminal activity of inmates by analyzing monitored telephone conversations for events, including, but not limited to, the addition of third parties, the discussion of particular topics, and the mention of certain entities.
1. A system for multi-modal audio mining of telephone data, the system comprising:
one or more circuits and/or processors configured to:
retrieve a data record from a call database, the data record including an audio inmate communication and metadata associated with the audio inmate communication;
perform speech recognition of the audio inmate communication, the speech recognition detecting non-verbal characteristics of the audio inmate communication;
generate a transcript of the audio inmate communication based on the speech recognition that includes the detected non-verbal characteristics;
identify a plurality of related data records in the call database based on the metadata of the retrieved data record, the plurality of related data records being identified based in part on a similarity of the non-verbal characteristics of the audio inmate communication to non-verbal characteristics of communications included in the plurality of related data records;
generate logical links between the retrieved data record and the plurality of related data records;
generate a visual representation of data record relationships based on the logical links; and
store the logical links and the transcript in association with each other in a second database.
2. The system of claim 1 , wherein the audio inmate communication is audio data of a monitored voice or video call.
3. The system of claim 1 , wherein the visual representation of the data record relationships is a link chart.
4. The system of claim 3 , wherein the link chart includes a plurality of nodes that each correspond to one of the retrieved data record and a plurality of links between the nodes corresponding to the logical links.
5. The system of claim 4 , wherein the one or more circuits and/or processors are configured to generate the logical links by identifying a predetermined number of metadata elements of the retrieved data record that match a predetermined number of metadata elements of a corresponding related data record from among the plurality of related data records.
6. The system of claim 5 , wherein the predetermined number is 1.
7. The system of claim 5 , wherein the predetermined number is greater than 1.
8. The system of claim 1 , wherein the metadata includes a plurality of metadata elements, the metadata elements including call parties, originating phone numbers, and terminating phone numbers.
9. A method for multi-modal audio mining of telephone data, the method comprising:
retrieving a data record from a call database, the data record including an audio inmate communication and metadata associated with the audio inmate communication;
performing speech recognition of the audio inmate communication, the speech recognition detecting non-verbal characteristics of the audio inmate communication;
generating a transcript of the audio inmate communication based on the speech recognition that includes the detected non-verbal characteristics;
identifying a plurality of related data records in the call database based on the metadata of the retrieved data record, the plurality of related data records being identified based in part on a similarity of the non-verbal characteristics of the audio inmate communication to non-verbal characteristics of communications included in the plurality of related data records;
generating logical links between the retrieved data record and the plurality of related data records;
generating a visual representation of data record relationships based on the logical links; and
storing the logical links and the transcript in association with each other in a second database.
10. The method of claim 9 , wherein the audio inmate communication is audio data of a monitored voice or video call.
11. The method of claim 9 , wherein the visual representation of the data record relationships is a link chart.
12. The method of claim 11 , wherein the link chart includes a plurality of nodes that each correspond to one of the retrieved data record and a plurality of links between the nodes corresponding to the logical links.
13. The method of claim 12 , wherein the generating the logical links includes identifying a predetermined number of metadata elements of the retrieved data record that match a predetermined number of metadata elements of a corresponding related data record from among the plurality of related data records.
14. The method of claim 13 , wherein the predetermined number is 1.
15. The method of claim 13 , wherein the predetermined number is greater than 1.
16. The method of claim 9 , wherein the metadata includes a plurality of metadata elements, the metadata elements including call parties, originating phone numbers, and terminating phone numbers.
17. A system for visually representing relationships among inmate communications, the system comprising:
a storage system that stores previous inmate communications;
a retrieval subsystem configured to:
receive a current audio inmate communication, the current audio inmate communication including a content portion and a metadata portion, the metadata portion including a plurality of metadata elements;
perform speech recognition of the current audio inmate communication, the speech recognition detecting non-verbal characteristics of the current audio inmate communication;
generate a transcript of the current audio inmate communication based on the speech recognition that includes the detected non-verbal characteristics;
access the storage system; and
retrieve a related inmate communication from among the stored previous inmate communications, the related inmate communication having a predetermined number of metadata elements that match corresponding ones of the plurality of metadata elements of the current inmate communication and having non-verbal characteristics of a communication that are substantially similar to non-verbal characteristics of the audio inmate communication; and
a linking subsystem configured to:
identify a logical link between the current inmate communication and the related inmate communication based on the matching metadata elements;
generate a visual representation of a relationship between the current inmate communication and the related inmate communication that illustrates the logical link; and
store the logical links and the transcript in association with each other in a second database.
18. The system of claim 17 , wherein the retrieval subsystem and the linking subsystem are configured to repeat their processes for a plurality of additional related inmate communications.
19. The system of claim 17 , wherein the visual representation is a link chart comprising a plurality of nodes corresponding to the current inmate communication and the related inmate communication, and a plurality of links illustrating connections between the nodes.
20. The system of claim 19 , wherein the plurality of links further include an indication of a type of the connection.