Method, system and module for multi-modal data fusion
View Patent ↗A method for multi-modal data fusion ( 100 ), a multi-modal system data fusion system ( 10 ) and module ( 24 ) that in use operates by receiving segments ( 125 ) of multi-modal data associated respectively with a modalitiy. Initiating ( 130 ) a dynamically variable wait period after one of the segments is received is then performed. The dynamically variable wait period has a duration determined from data fusion timing statistics of the system ( 10 ). A waiting ( 140 ) for reception of any further segments during the dynamically variable wait period is then effected and thereafter a fusing ( 145 ) of the segments received provides fused data that is sent ( 160 ) to a dialog manager ( 25 ).
1. A method for multi-modal data fusion in a multi-modal system comprising a plurality of input modalities, the method comprising the steps of:
(i) Receiving one or more segments of multi-modal data, each of said segments being associated respectively with said modalities;
(ii) Initiating a dynamically variable wait period after one of said segments is received, said dynamically variable wait period having a duration determined from data fusion timing statistics of the system, wherein the duration of the dynamically variable wait period is determined from analysis of the following:
min (max(AvgDur m ) or (AvgTimeDiff+AvgDur)) where AvgDur m is a value indicative of the average duration associated with processing a user request into a one of said segments for a modality m, AvgTimeDiff is a value indicative of an average period for periods starting after completion of receiving one of said segments and ending after completion of receiving a further one of said segments, and AvgDur is a value indicative of the average duration associated with processing a user request into a one of said segments for every modality in the system;
(iii) Waiting for reception of any further said segments during said dynamically variable wait period; and
(iv) Fusing said segments received during said steps (i) to (iii) to provide fused data.
2. A method for multi-modal data fusion, as claimed in claim 1 , further characterized by repeating steps (ii) to (iv) if one or more said further said segments are received during said dynamically variable wait period.
3. A method for multi-modal data fusion, as claimed in claim 1 , including the further step of sending said fused data to a dialog manager.
4. A method for multi-modal data fusion, as claimed in claim 1 , wherein said segments are frames.
5. A method for multi-modal data fusion, as claimed in claim 4 , wherein said frames include temporal characteristics including at least part of said historical statistical data; the frames may also include semantic representations of an associated user request.
6. A multi-modal data fusion system comprising:
a plurality of modality processing modules;
a plurality of parsers coupled to a respective one of said modality processing modules; a multi-modal fusion module having inputs coupled to outputs of said parsers, wherein in use said fusion module receives one or more segments of multi-modal data from at least one of said parsers, and initiates a dynamically variable wait period after one of said segments is received, said dynamically variable wait period having a duration determined from data fusion timing statistics of the system, the fusion module then waits for reception of any further said segments during said dynamically variable wait period and fuses said segments received to provide fused data, wherein the duration of the dynamically variable wait period is determined from analysis of the following:
min (max(AvgDur m ) or (AvgTimeDiff+AvgDur)) where AvgDur m is a value indicative of the average duration associated with processing a user request into a one of said segments for a modality m, AvgTimeDiff is a value indicative of an average period for periods starting after completion of receiving one of said segments and ending after completion of receiving a further one of said segments, and AvgDur is a value indicative of the average duration associated with processing a user request into a one of said segments for every modality in the system.
7. A multi-modal data fusion system as claimed in claim 6 , wherein there is a dialogue manager coupled to an output of said fusion module.
8. A multi-modal data fusion system as claimed in claim 6 , there are user input devices coupled respectively to said modality processing modules.
9. A multi-modal fusion module having inputs for coupling to outputs of parsers, wherein in use said fusion module receives one or more segments of multi-modal data from at least one of said parsers, and initiates a dynamically variable wait period after one of said segments is received, said dynamically variable wait period having a duration determined from data fusion timing statistics of the system, the fusion module then waits for reception of any further said segments during said dynamically variable wait period and fuses said segments received to provide fused data, wherein the duration of the dynamically variable wait period is determined from analysis of the following:
min (max(AvgDur m ) or (AvgTimeDiff+AvgDur)) where AvgDur m is a value indicative of the average duration associated with processing a user request into a one of said segments for a modality m, AvgTimeDiff is a value indicative of an average period for periods starting after completion of receiving one of said segments and ending after completion of receiving a further one of said segments, and AvgDur is a value indicative of the average duration associated with processing a user request into a one of said segments for every modality in the system.