Method and apparatus for classifying and ranking interpretations for multimodal input fusion
View Patent ↗A method used in an electronic equipment ( 100 ) generates a set of joint multimodal interpretations ( 125 ) from a set of multimodal interpretations ( 115 ) generated by one or more modalities ( 105 ) during a turn, generates a set of integrated multimodal interpretations ( 135 ) including an integrated multimodal interpretation formed from each joint multimodal interpretation by unifying the type feature structure of each multimodal interpretation in the joint multimodal interpretation, and generates a multilevel confidence score for each integrated multimodal interpretation based on at least one of a context score, a content score, and a confidence score of the integrated multimodal interpretation. The method classifies the multimodal interpretations and generates a set of joint multimodal interpretations that comprises essentially all possible joint multimodal interpretations. The multilevel confidence scoring is based on up to eleven factors, which provides for an accurate ranking of the integrated multimodal interpretations.
1. A method for generating joint multimodal interpretations, comprising:
receiving a set of multimodal interpretations that are generated within a turn by a set of modalities, wherein each multimodal interpretation is one of either a non-ambiguous interpretation or an ambiguous interpretation of a subset of ambiguous interpretations of the set of multimodal interpretations, and wherein each multimodal interpretation is identified with the modality that has generated the multimodal interpretation, and wherein each multimodal interpretation is assigned an interpretation type and confidence score; and
generating a set of joint multimodal interpretations comprising all possible joint multimodal interpretations, each of which is formed from one of all possible combinations of multimodal interpretations of the set of multimodal interpretations such that each joint multimodal interpretation has no more than one ambiguous interpretation from each subset of ambiguous interpretations, and such that the multimodal interpretations of each joint multimodal interpretation satisfy an interpretation type relationship of a defined set of interpretation type relationships based on the interpretation type and modality of each multimodal interpretation.
2. The method according to claim 1 , wherein the set of joint multimodal interpretations includes at least one joint multimodal interpretation that includes two multimodal interpretations generated by one source that are not in the same subset of ambiguous interpretations.
3. The method according to claim 1 , wherein all possible combinations of multimodal interpretations of the set of multimodal interpretations include all possible combinations having multimodal interpretations from fewer than all of the one or more modalities.
4. The method according to claim 1 , wherein each multimodal interpretation is related to at least one other multimodal interpretation in the set of interpretation type relationships by a relationship that is one of direct super, direct sub, direct container, direct contained, and same.