Automated systems and methods that generate affect-annotated timelines
The current document is directed to a methods and systems that use observational data collected by various devices and sensors to generate electronic-data representations of human conversations. The implementations of these methods and systems, disclosed in the current document, provide a highly extensible and generic platform for converting observational data into affect-annotated-timeline outputs that provide both a textual transcription of a conversation and a parallel set of affect annotations to the conversation. The affect-annotated-timeline outputs may be useful to researchers and developers, but also serve as inputs to any of a wide variety of downstream analytical processes and analysis systems that are, in turn, incorporated into many different types of special-purpose analysis and control systems.
1. An affect-annotated-timeline data structure stored in one of a data-storage device, a data-storage appliance, and an electronic memory within a computer system, the affect-annotated-timeline data structure generated from a monitored conversation by an automated affect-annotation system, the affect-annotated-timeline data structure comprising:
multiple affect-annotation records, each affect-annotation record
representing a conversation unit that
corresponds to a subinterval of the monitored conversation,
that is attributed to a particular source, and
that is generated, by the automated affect-annotation system, from one or more units of language for affect coding, referred to as “ULACs,” each ULAC
corresponding to a minimal aggregation of words extracted from a conversation that conveys an intra-contextual meaning, and
identified by the automated affect-annotation system based on lexical dependency graphs, and
containing
a textual representation of the conversation unit,
an indication of a source of the conversation unit,
an indication of a temporal duration of the conversation unit and an indication of a temporal position of the conversation unit within the monitored conversation, and
an affect-code probability distribution.
2. The affect-annotated-timeline data structure of claim 1 wherein a source of a conversation unit is one of:
a human conversation participant; and
an automated-system conversation participant.
3. The affect-annotated-timeline data structure of claim 2 wherein the monitored conversation includes two or more sources.
4. The affect-annotated-timeline data structure of claim 1 wherein an indication of a temporal duration of the conversation unit and an indication of a temporal position of the conversation unit within the monitored conversation includes one of:
an indication of a start time and an indication of a time-interval length;
an indication of an end time and an indication of a time-interval length; and
an indication of a start time and an indication of an end time.
5. The affect-annotated-timeline data structure of claim 1 wherein an affect-code probability distribution is a discrete probability distribution that indicates the probability that the conversation unit represented by an affect-annotation record corresponds to each of multiple affect codes that each corresponds to an emotional state and/or intention that corresponds to one or more observables, the observables including behavioral patterns and physiological conditions.
6. The affect-annotated-timeline data structure of claim 5 wherein the affect codes are each represented by a numerical indication unique to the affect code.
7. The affect-annotated-timeline data structure of claim 6 wherein the affect codes include:
negative affect codes that include
contempt/belligerence,
domineering/defensive/anger,
sadness/whining, and
tension;
a neutral affect code; and
positive affect codes that include
validation,
interest,
joy/affection, and
humor.
8. The affect-annotated-timeline data structure of claim 1 wherein each affect-annotation record further includes a sequence number.
9. The affect-annotated-timeline data structure of claim 8 wherein each affect-annotation record further includes one or more additional numeric, text-containing, and alphanumeric fields.
10. An affect-annotation system that receives conversation data generated from a monitored conversation and processes the input conversation data to produce an electronic representation of the monitored conversation, the affect-annotation system comprising:
one or more processors;
one or more memories and mass-storage devices; and
computer instructions, stored in one or more of the one or more memories and mass-storage devices that, when executed by one or more of the one or more processors, control the affect-annotation system to
receive conversation data, including one or more of
video data,
audio data, and
physiological data,
generate multiple affect-annotation records for each of multiple conversation units identified using the received conversation data,
each conversation unit corresponding to a subinterval of the monitored conversation, attributed to a particular source, generated, by the automated affect-annotation system, from one or more units of language for affect coding, referred to as “ULACs,” each ULAC corresponding to a minimal aggregation of words extracted from a conversation that conveys an intra-contextual meaning, and identified by the automated affect-annotation system based on lexical dependency graphs, and
each affect-annotation record containing a textual representation of a conversation unit, an indication of a source of the conversation unit, an indication of a temporal duration of the conversation unit, an indication of a temporal position of the conversation unit within the monitored conversation, and an affect-code probability distribution,
aggregate the generated affect-annotation records into an affect-annotated-timeline data structure, and
output the affect-annotated-timeline data structure to one or more of a display device, a data-storage appliance or device, and one or more downstream analysis systems.
11. The affect-annotation system of claim 10
wherein a source of a conversation unit is one of
a human conversation participant; and
an automated-system conversation participant; and
wherein the monitored conversation includes two or more sources.
12. The affect-annotation system of claim 10 wherein an indication of a temporal duration of the conversation unit and an indication of a temporal position of the conversation unit within the monitored conversation includes one of:
an indication of a start time and an indication of a time-interval length;
an indication of an end time and an indication of a time-interval length; and
an indication of a start time and an indication of an end time.
13. The affect-annotation system of claim 10 wherein an affect-code probability distribution is a discrete probability distribution that indicates the probability that the conversation unit represented by an affect-annotation record corresponds to each of multiple affect codes that each corresponds to an emotional state and/or intention that corresponds to one or more observables, the observables including behavioral patterns and physiological conditions.
14. The affect-annotation system of claim 10 wherein the affect codes are each represented by a numerical indication unique to the affect code.
15. The affect-annotation system of claim 14 wherein the affect codes include:
negative affect codes that include
contempt/belligerence,
domineering/defensive/anger,
sadness/whining, and
tension;
a neutral affect code; and
positive affect codes that include
validation,
interest,
joy/affection, and
humor.
16. The affect-annotation system of claim 10 wherein each affect-annotation record further includes a sequence number.