System and a method for analyzing a video
The invention relates to a computer implemented method and system for analyzing a video. The method comprises the steps of receiving, via a receiving module, a video data comprising a series of images showing a subject; extracting, via an extracting module, a transcript derived from an audio data associated with the video data; aligning, via an aligning module, the series of images of the video data with the transcript derived from the audio data associated with the video data based on timestamps derived from the video data; analyzing, via an analyzing module, gestures of the subject from the series of images, comprising the steps of: identifying a plurality of reference points from each of the series of images showing the subject; segmenting the series of images in accordance with one or more selected texts comprising the transcript; identifying a defined gesture type for each of the segmented images based on the identified plurality of reference points; and processing, via a processing module, the defined gesture types in correlation with respective one or more selected texts comprising the transcript.
1 . A computer implemented method for analysing a video, comprising the steps of:
receiving, via a receiving module, a video data comprising a series of images showing a subject;
extracting, via an extracting module, a transcript derived from an audio data associated with the video data;
aligning, via an aligning module, the series of images of the video data with the transcript derived from the audio data associated with the video data based on timestamps derived from the video data;
analysing, via an analysing module, gestures of the subject from the series of images, comprising the steps of:
identifying a plurality of reference points from each of the series of images showing the subject;
segmenting the series of images in accordance with one or more selected texts comprising the transcript;
identifying a defined gesture type for each of the segmented images based on the identified plurality of reference points;
processing, via a processing module, the defined gesture types in correlation with respective one or more selected texts comprising the transcript; and
presenting, via a visualization module, a correlation between the defined gesture type with the respective one or more selected texts comprising the transcript; wherein the visualization module is adapted to display one or more of the following views, comprising:
a relation view showing visual linkages of the defined gesture types with the respective one or more selected text;
an exploration view showing a spatial distribution and/or a temporal distribution of the defined gesture types in association with the respective selected texts comprising the transcript;
a dynamic view showing trajectories of one or more reference points in a selected image from the series of images; and
a video view showing the video being analysed.
2 . The computer implemented method according to claim 1 , further comprising timestamping one or more of the identified plurality of reference points and the one or more selected texts comprising the transcript in accordance with the timestamps derived from the video data.
3 . The computer implemented method according to claim 1 , wherein the processing step comprises processing the defined gesture type and the respective one or more selected texts comprising the transcript in respect of time.
4 . The computer implemented method according to claim 1 , wherein the step of identifying a plurality of reference points from each of the series of images showing the subject further comprises mapping the plurality of reference points in respect of a predefined spatial region.
5 . The computer implemented method according to claim 4 , wherein the processing step comprises processing the defined gesture type with the respective one or more selected texts comprising the transcript in respect of the predefined spatial region.
6 . The computer implemented method according to claim 4 , further comprising determining a distribution of the defined gesture types in the predefined spatial region in accordance with the one or more selected texts comprising the transcript.
7 . The computer implemented method according to claim 6 , further comprising displaying the distribution of the defined gesture types in the predefined spatial region in the form of one or more of a heatmap and by showing trajectories of one or more reference points of the plurality of reference points.
8 . The computer implemented method according to claim 1 , wherein the step of identifying a defined gesture type for each of the segmented images based on the identified plurality of reference points comprises calculating a distance between gestures from two consecutive segmented images in respect of a selected reference point from the plurality of reference points.
9 . The computer implemented method according to claim 8 , wherein the step of calculating a distance between two consecutive segmented images in respect of a selected reference point comprises computation based on the equation:
D
(
F
,
G
)
=
1
∑
k
=
0
8
F
c
k
*
∑
k
=
0
8
F
c
k
*
F
xy
k
-
G
xy
k
wherein F ck is a factor of confidence probability for k-th reference point of the plurality of reference points; and F xyk and G xyk are x and y coordinates of k-th reference point of two consecutive images F and G, respectively.
10 . The computer implemented method according to claim 1 , further comprising presenting, via a visualization module, a correlation between the defined gesture types and the one or more selected texts comprising the transcript.
11 . The computer implemented method according to claim 1 , wherein the defined gesture types comprise at least a closed gesture, an open gesture and other gestures.
12 . The computer implemented method according to claim 1 , wherein the defined gesture types are presented as figurative indicators arranged adjacent to and/or via one or more visual linkages with the corresponding one or more selected texts.
13 . The computer implemented method according to claim 1 , wherein the one or more selected texts comprise semantic phrases of the transcript comprising one or more of noun phrases, verb phrases, prepositional phrases and subject-verb object phrases.
14 . The computer implemented method according to claim 1 , further comprising presenting at least one of a horizontal movement and a vertical movement of one or more reference points in respect of time, wherein the one or more reference points comprises two reference points representing left hand and right hand of the subject.
15 . The computer implemented method according to claim 1 , further comprising searching, via a searching module, by inputting either a defined gesture type or a selected text; and outputting a corresponding one of a selected text or a defined gesture type, respectively.
16 . A system for analysing a video, comprising:
a receiving module for receiving a video data comprising a series of images showing a subject;
an extracting module for extracting a transcript derived from an audio data associated with the video data;
an aligning module for aligning the series of images of the video data with the transcript derived from the audio data associated with the video data based on timestamps derived from the video data;
an analysing module for analysing gestures of the subject from the series of images, comprising the steps of:
identifying a plurality of reference points from each of the series of images showing the subject;
segmenting the series of images in accordance with one or more selected texts comprising the transcript;
identifying a defined gesture type for each of the segmented images based on the identified plurality of reference points;
a processing module for processing the defined gesture types in correlation with respective one or more selected texts comprising the transcript; and
a visualization module for presenting a correlation between the defined gesture type with the respective one or more selected texts comprising the transcript; wherein the visualization module is adapted to display one or more of the following views, comprising:
a relation view showing visual linkages of the defined gesture types with the respective one or more selected text;
an exploration view showing a spatial distribution and/or a temporal distribution of the defined gesture types in association with the respective selected texts comprising the transcript;
a dynamic view showing trajectories of one or more reference points in a selected image from the series of images; and
a video view showing the video being analysed.
17 . The system according to claim 16 , further comprising a searching module for searching, by inputting either a defined gesture type or a selected text; and outputting a corresponding one of a selected text or a defined gesture type, respectively.
18 . A non-transitory computer readable medium storing machine-readable instructions executable by a processor to configure a system for analysing a video by implementing the steps of:
receiving, via a receiving module, a video data comprising a series of images showing a subject;
extracting, via an extracting module, a transcript derived from an audio data associated with the video data;
aligning, via an aligning module, the series of images of the video data with the transcript derived from the audio data associated with the video data based on timestamps derived from the video data;
analysing, via an analysing module, gestures of the subject from the series of images, comprising the steps of:
identifying a plurality of reference points from each of the series of images showing the subject;
segmenting the series of images in accordance with one or more selected texts comprising the transcript;
identifying a defined gesture type for each of the segmented images based on the identified plurality of reference points;
processing, via a processing module, the defined gesture types in correlation with respective one or more selected texts comprising the transcript; and
presenting, via a visualization module, a correlation between the defined gesture type with the respective one or more selected texts comprising the transcript; wherein the visualization module is adapted to display one or more of the following views, comprising:
a relation view showing visual linkages of the defined gesture types with the respective one or more selected text;
an exploration view showing a spatial distribution and/or a temporal distribution of the defined gesture types in association with the respective selected texts comprising the transcript;
a dynamic view showing trajectories of one or more reference points in a selected image from the series of images; and
a video view showing the video being analysed.