Methods and systems for segmenting multimedia content
A method and a system are provided for segmenting a multimedia content. The method estimates a count of a plurality of multimedia segments in the multimedia content, and a duration of each of the plurality of multimedia segments in the multimedia content. The method determines a cost function associated with a multimedia segment from the plurality of multimedia segments, based on the count of the plurality of multimedia segments, and the duration of each of the plurality of multimedia segments. The method further determines an updated count of the plurality of multimedia segments, and an updated duration of each of the plurality of multimedia segments until the cost function satisfies a pre-defined criteria. Based on the updated count of the plurality of multimedia segments, and the updated duration of each of the plurality of multimedia segments, the method segments the multimedia content into the plurality of multimedia segments.
1. A multimedia content server configured to segment multimedia content, comprising:
one or more processors; and
memory comprising a set of instructions, wherein
the set of instructions, with the one or more processors, are configured to
extract a set of visual keywords and a set of textual keywords from the multimedia content,
estimate a count of a plurality of multimedia segments in the multimedia content, and
segment the multimedia content into the plurality of multimedia segments based on a cost function associated with each of the plurality of multimedia segments, wherein
the cost function is determined based on the estimated count, a first score, and a second score, and is computed iteratively until a predefined criteria is met, wherein
the extraction of the set of visual keywords is represented by
W S,E vis =∪ t S vis :t E vis W m vis
where W m vis represents a set of visual keywords in the m-th slide in the multimedia content,
W S,E vis represents the set of visual keywords in the multimedia segment within time segment t S,E ,
t S represents a start time of the multimedia segment, and
t E represents at end time of the multimedia segment.
2. The multimedia content server of claim 1 , wherein the set of instructions, with the one or more processors, are further configured to
extract an image from the multimedia content and compare the image with a stored glyph on a pixel-by-pixel basis,
extract the set of visual of visual keywords from the multimedia content based on the comparing of the image with a stored glyph on the pixel-by-pixel basis, and
determine a visual saliency score associated with each keyword in the set of visual keywords.
3. The multimedia content server of claim 2 , wherein the set of instructions, with the one or more processors, are further configured to
determine a textual saliency score associated with each keyword in the set of textual keywords.
4. The multimedia content server of claim 3 , wherein each keyword comprises the set of visual keywords and the set of textual keywords.
5. The multimedia content server of claim 4 , wherein the set of instructions, with the one or more processors, are further configured to determine a set of key phrases from each keyword, wherein each keyword corresponds to one media segment in the plurality of multimedia segments.
6. The multimedia content server of claim 1 , wherein the extraction of the set of textual keywords is represented by
W S,E text =∪ t S :t E W n text
W n text represents a set of textual keywords in the nth sentence in the multimedia content, and,
W S,E text represents the set of textual keywords in the multimedia segment within time segment t S,E .
7. The multimedia content server of claim 6 , wherein the set of instructions, with the one or more processors, are further configured to remove one or more stop words and document dependent stop words from the set of textual keywords,
the one or more stop words correspond to conjunctions, prepositions, or both, and
the one or more document dependent stop words correspond to one or more keywords from the set of textual keywords that have a high number of occurrences.
8. The multimedia content server of claim 1 , wherein the set of instructions, with the one or more processors, are further configured to remove one or more stop words and one or more document dependent stop words from the set of visual keywords prior to computing a visual saliency score.
9. The multimedia content server of claim 1 , wherein the set of instructions, with the one or more processors, are further configured to
compare the set of textual keywords with the set of visual keywords to determine a subset of textual keywords,
the determination of the subset of textual keywords filters the set of textual keywords.
10. The multimedia content server of claim 9 , wherein the set of instructions, with the one or more processors, are further configured to sort the plurality of keywords in accordance with a saliency score associated with each of the plurality of keywords,
the saliency score comprises a textual saliency score and a visual saliency score, and
the plurality of keywords comprises the subset of textual keywords and the set of visual keywords.
11. The multimedia content server of claim 10 , wherein the set of instructions, with the one or more processors, are further configured to update the saliency score of each of the plurality of keywords based on at least a position of the keywords in a sorted list of the plurality of keywords.
12. The multimedia content server of claim 11 , wherein the update of the saliency score is represented by
s p =z i s p
where s p is the saliency score associated with p-th keyword such that W p ∈W s,e ,
W s,e denotes the subset of textual keywords and the set of visual keywords,
i is a location of the p-th keyword in the sorted list, and
z is a weighting factor.
13. The multimedia content server of claim 11 , wherein, when estimating the count of the multimedia segments, the set of instructions, with the one or more processors, are further configured to
estimate a set of partition points of each of the estimated count of the multimedia segments as defined by
τ opt ={t m1 ,t m2 , . . . ,t mk },
where
t mk ∈τ vis ∀k
where k is the estimated count of the multimedia segments in the multimedia content.
14. The multimedia content server of claim 13 , wherein the set of instructions, with the one or more processors, are further configured to determine the first score for consecutive multimedia segments,
the consecutive multimedia segments are denoted by S m k−1 ,m k , and S m k ,m k+1 and
the first score is denoted by C(S m k−1 ,m k ,S m k ,m k+1 ) representative of the degree of similarity between two consecutive multimedia segments and determined by
First score=first factor+second factor
where the first factor denotes a summation of a saliency score of the plurality of keywords common in the consecutive multimedia segments S m k−1 ,m k and S m k ,m k+1 normalized by the summation of the saliency scores of the plurality of keywords present in the multimedia segment S m k−1 ,m k , and
where the second factor denotes a summation of the saliency score of the plurality of keywords that are common between the consecutive multimedia segments S m k−1 ,m k and S m k ,m k+1 normalized by the summation of the saliency scores of the plurality of keywords present in the multimedia segment S m k ,m k+1 .
15. The multimedia content server of claim 14 , wherein the set of instructions, with the one or more processors, are further configured to determine the second score for the consecutive multimedia segments,
the consecutive multimedia segments are denoted by S m k−1 ,m k and S m k ,m k+1 and
the second score is denoted by D(S m k−1 ,m k ,S m k ,m k+1 ) is configured to capture the degree of dissimilarity of dissimilar words between two consecutive multimedia segments and determined by
Second score=third factor+fourth factor
where the third factor denotes a summation of a saliency score of the plurality of keywords that are present in the multimedia segment S m k−1 ,m k but are not present in the multimedia segment S m k ,m k+1 , the third factor being normalized by the summation of the saliency scores of the plurality of keywords present in the multimedia segment S m k−1 ,m k , and
where the fourth factor denotes a summation of the saliency score of the plurality of keywords that are present in the multimedia segment S m k−1 ,m k but are not present in the multimedia segment S m k ,m k+1 , the fourth factor being normalized by the summation of the saliency scores of the plurality keywords present in the multimedia segment S m k ,m k+1 .
16. The multimedia content server of claim 1 , wherein the cost function is determined by
τ
opt
=
argmin
τ
vis
∑
k
=
1
k
(
α
,
C
(
S
m
k
-
1
,
m
k
,
S
m
k
,
m
k
+
1
)
-
(
1
-
α
)
·
D
(
S
m
k
-
1
,
m
k
,
S
m
k
,
m
k
+
1
)
)
where α denotes weighting factor associated with each parameter in the cost function,
S m k−1 ,m k denotes a multimedia segment between the time instant t m k−1 ,t m k ,
S m k ,m k+1 denotes a multimedia segment between the time instant t m k ,t m k+1 ,
C(S m k−1 ,m k ,S m k ,m k+1 ) denotes the first score, and
D(S m k−1 ,m k ,S m k ,m k+1 ) denotes the second score.
17. The multimedia content server of claim 1 , wherein the set of instructions, with the one or more processors, are further configured to
determine a set of partition points in the multimedia content based on the cost function,
segment the multimedia content into the plurality of multimedia segments based on the set of partition points,
determine a set of key phrases from the plurality of keywords associated with each of the multimedia segment based on a saliency score associated with each of the plurality of keywords, and
create a table of contents deterministic of association between the set of key phrases and the plurality of multimedia segments.
18. A multimedia content server configured to segment multimedia content, comprising:
one or more processors; and
memory comprising a set of instructions, wherein
the set of instructions, with the one or more processors, are configured to
extract a set of visual keywords and a set of textual keywords from the multimedia content,
estimate a count of a plurality of multimedia segments in the multimedia content, and
segment the multimedia content into the plurality of multimedia segments based on a cost function associated with each of the plurality of multimedia segments, wherein
the cost function is determined based on the estimated count, a first score, and a second score, and is computed iteratively until a predefined criteria is met, wherein the cost function is determined by
τ
opt
=
argmin
τ
vis
∑
k
=
1
k
(
α
,
C
(
S
m
k
-
1
,
m
k
,
S
m
k
,
m
k
+
1
)
-
(
1
-
α
)
·
D
(
S
m
k
-
1
,
m
k
,
S
m
k
,
m
k
+
1
)
)
where α denotes weighting factor associated with each parameter in the cost function,
S m k−1 ,m k denotes a multimedia segment between the time instant t m k−1 ,t m k ,
S m k ,m k+1 denotes a multimedia segment between the time instant t m k ,t m k+1 ,
C(S m k−1 ,m k ,S m k ,m k+1 ) denotes the first score, and
D(S m k−1 ,m k ,S m k ,m k+1 ) denotes the second score.