System and method for automatic detection of periods of heightened audience interest in broadcast electronic media

1. A method executed by a computer system for determining periods of time in an audio-visual signal encoding an event comprising:

receiving a data representing the audio-visual signal said signal comprised of data representing an audience attending said event;

extracting from the received audio-visual data at least one parameter data corresponding to an at least one time-slice within the audio-visual signal, said at least one parameter data representing a magnitude of heightened emotional state of the audience attending the event that is the subject of the of the audio-visual signal within the corresponding an at least one time slice;

storing in a data structure the at least one parameter data and an at least one time slice value corresponding to the at least one parameter;

determining an at least one time values of heightened interest by calculating a correlation between the extracted at least one parameter data representing a magnitude of heightened emotional state of the audience attending the event that is the subject of the of the audio-visual signal within the corresponding an at least one time slice and an at least one pre-determined parameter data values associated with a heightened audience interest level; and

storing the determined at least one time values in a data structure associated with the received audio-visual signal.

2. The method of claim 1 where the extracting step is comprised of:

extracting an audio portion of the audio-visual data stream; and

generating from the extracted audio an at least one parameters representing an at least one spectral vectors, each spectral vector corresponding to the at least one time slice.

3. The method of claim 2 where the determining step is comprised of:

for each time slice, automatically determining if the spectral vector of that time slice is part of a spectral pattern that is correlated with heightened audience interest; and

in dependence on the automatic determination whether the spectral vector is part of said spectral pattern, storing in a data structure at least one data values representing at least one time value corresponding to the determined spectral pattern.

4. The method of claim 1 where the extracting step is comprised of:

extracting an audio portion of the audio-visual data stream;

detecting in the extracted audio portion an at least one word text data item, said word text data item having a corresponding at least one time slice value; and

determining if the at least one word text data item is an element in a pre-determined word set correlated with heightened audience emotion or audience interest.

5. The method of claim 1 where the extracting step is comprised of:

receiving a data file comprised of text data representing at least one word spoken during the audio-visual data; and

determining if the at least one word text data item is an element in a pre-determined word set correlated with heightened audience emotion or audience interest.

6. The method of claim 5 where the data file comprised of text data is further comprised of time data values that indicate an at least one time values during the audio visual data corresponding to where the at least one words were spoken.

7. The method of claim 1 further comprising:

extracting an audio data portion of the audio-visual data stream;

automatically detecting an at least one sound level in a corresponding at least one time slice of the audio data portion; and

automatically detecting whether the at least one detected sound levels represent a pattern of sound levels correlate with a sound level pattern representing a higher emotive state of the audience.

8. The method of claim 1 further comprising:

extracting a video data portion of the audio-visual data stream;

automatically detecting at least one video data characteristics correlated with a heightened interest by the audience;

determining an at least one time slice value corresponding to the at least one detected video characteristics; and

storing in the data structure the at least one determined time slice values.

9. The method of claim 8 where the video data characteristics are comprised of one or both of variances in colour or luminosity.

10. The method of claim 8 further comprising:

extracting from the video data portion data representing at least one facial expression; and

determining whether the at least one extracted facial expression data is correlated with a pre-determined facial expression data correlated with heightened emotional interest by an audience member.

11. The method of claim 1 where the storing step is comprised of:

automatically generating and storing a data structure comprised of data that lists an at least one start and a corresponding at least one a stop time of a corresponding at least one period of time during the audio-visual signal determined to be correlated with a heightened audience interest level.

12. The method of claim 1 further comprising:

receiving at least one data representing a physiological response by an at least one viewer of the audio-visual signal; and

storing an at least one data value representing the physiological response of the at least one viewer in a data structure that defines a correspondence between the at least one physiological data value and the at least one time slice.

13. The method of claim 1 where the extracting step is comprised of:

determining an at least two time interval data comprised of a corresponding at least one start time slice value and a corresponding at least one stop time slice value where substantially the same video data during each of the at least two time intervals during the audio visual signal is repeated.

14. The method of claim 1 where the extracting step is comprised of extracting at least one parameter data representing an at least one audio sound level, and the storing step is comprised of storing at least one text data value corresponding to the at least one time slice and the calculating a correlation step is further comprised of using the text data in combination with the audio sound level data to determine the at least one time values of heightened audience interest.

15. The method of claim 13 where the storing step is comprised of storing at least one text data value corresponding to the at least one time slice and the calculating a correlation step is further comprised of using the text data in combination with the at least two video time interval data to determine the at least one time values of heightened audience interest.

16. The method of claim 1 where the at least one parameter data represents one of a colour or a luminosity variance.

17. The method of claim 1 where the at least one parameter data represents a facial expression characteristic.

18. The method of claim 1 where the determining step is comprised of determining an at least one time values of heightened audience interest by calculating a correlation among data variables representing at least one of: (i) an at least one audio spectral vector, (ii) an at least one audio sound level, (iii) an at least one colour, (iv) an at least one luminosity variance, (v) an at least one facial expression characteristic, (v) an at least one word data or (vi) at least one physiologic response.

19. The method of claim 18 where the determining step is comprised of calculating a matching score for one or more of the calculated correlations with the at least one predetermined audio spectral pattern, predetermined sound level pattern, predetermined colour or luminosity variance, predetermined facial characteristic and predetermined word set.

20. The method of claim 18 where the determining step is comprised of inputting the data variables into a machine learning engine and using an output of the machine learning engine for the calculating a correlation step.

21. The method of claim 20 where the machine learning engine is a neural network.

22. The method of claim 20 where the machine learning engine calculates a correlation matching score.

23. The method of claim 18 further comprising:

using a video portion of the audio-visual data to detect an at least one time slices where the video portion has repeated an at least one earlier video portion of the audio-visual data by detecting an at least one self-similar region in the visual component of the audio-visual data.

24. The method of claim 11 further comprising:

receiving the data structure comprised of at least one start and stop times; and

generating an edited version of the audio-visual data by using the received data structure to select corresponding portions of the audio-visual data.

25. The method of claim 24 where the generating step is further comprised of:

applying at least one mapping rule to an at least one time value in the data structure to specify an at least one edit point.

26. The method of claim 21 where the mapping rule is one of: setting the edit point to be at least one of: the time of a detected change in luminosity or colour that was detected just prior to the time value, or a predetermined pre-roll time before prior to the time value.

27. The method of claim 1 where the determining step is comprised of determining a logical condition where at substantially the same time slice values, either the audio spectral vectors or the audio sound levels are correlated with heightened audience interest and the word data are correlated with heightened audience interest.

28. A method executed by a computer system for determining in an audio-visual signal encoding an event that is comprised of data representing an audience attending the event, periods of time that have heightened audience interest level, comprising:

receiving a data representing the audio-visual signal;

extracting from the received audio-visual data at least one parameter data corresponding to an at least one time-slice within the audio-visual signal, said at least one parameter data representing a magnitude of a sound level within a predetermined audio spectral range of the audio-visual signal within the corresponding an at least one time slice;

receiving data representing an at least one word text data item representing at least one word spoken during the audio-visual data said at least one word text data item having a corresponding at least one time slice value;

storing in a data structure the at least one parameter data and an at least one time slice value corresponding to the at least one parameter;

storing in the data structure the at least one word text data item and corresponding at least one time slice value;

determining an at least one time values of heightened audience interest by calculating a correlation between the extracted at least one parameter data representing a magnitude of a sound level within a predetermined audio spectral range of the audio-visual signal within the corresponding an at least one time slice, the stored at least one word text data items and an at least one pre-determined parameter data values and an at least one pre-determined text data items associated with a heightened audience interest level; and

storing the determined at least one time values in a data structure associated with the audio-visual signal.

29. The method of claim 28 where the determining step is comprised of calculating a matching score for one or more of the calculated correlations with the at least one predetermined audio spectral pattern, predetermined sound level pattern, predetermined colour or luminosity variance, predetermined facial characteristic and predetermined word set.

30. The method of claim 28 where the determining step is comprised of inputting the data variables into a machine learning engine and using an output of the machine learning engine for the calculating a correlation step.

31. The method of claim 30 where the machine learning engine is a neural network.

32. The method of claim 30 where the machine learning engine calculates a correlation matching score.

33. The method of claim 28 further comprising:

using a video portion of the audio-visual data to detect an at least one time slices where the video portion has repeated an at least one earlier video portion of the audio visual data by detecting an at least one self-similar region in the visual component of the audio-visual data.

34. The method of claim 28 further comprising:

automatically generating and storing a data structure comprised of data that lists an at least one start and a corresponding at least one a stop time of a corresponding at least one period of time during the audio-visual signal determined to be correlated with a heightened audience interest level; and

generating an edited version of the audio-visual data by using the stored data structure comprised of data that lists an at least one start and a corresponding at least one a stop time to select an at least one corresponding portions of the audio-visual data.

35. The method of claim 34 where the generating step is further comprised of:

applying at least one mapping rule to an at least one time value in the data structure to specify an at least one edit point.

36. The method of claim 35 where the mapping rule is one of: setting the edit point to be the time of the scene transition that was detected just prior to the time value, setting the edit point to be a predetermined pre-roll time before prior to the time value.