Method and system for multimodal clue based personalized app function recommendation
A method for multimodal clue based personalized app function recommendation is provided. The method includes receiving an app search query from a user, obtaining a plurality of real-time clues collected from the user and extracting a plurality of features to represent the collected clues, wherein the plurality of features include structured features and text features. The method also includes generating a joint representation of the multimodal clues based on the plurality of extracted features and creating a logistic regression model based on the joint representation of the multimodal clues. Further, the method includes ranking a list of app functions using the logistic regression model and displaying the ranked app function list for the user.
1. A method for multimodal clue based personalized app function recommendation, comprising:
receiving an app search query from a user;
obtaining a plurality of real-time clues collected from the user;
extracting a plurality of features to represent the plurality of real-time clues collected from the user, wherein the plurality of features include structured features and text features;
based on the plurality of extracted features, generating a joint representation of a multimodal clue, comprising:
using a multimodal Deep Boltzmann Machine (DBM) to construct two separate two-layer DBMs to model distribution over the structured features and the text features respectively; and
combining the two separate two-layer DBMs by adding an additional layer of binary hidden units on the top of the two-layer DBMs;
based on the joint representation of the multimodal clue, creating a logistic regression model;
ranking a list of app functions using the logistic regression model; and
displaying a ranked app function list for the user.
2. The method according to claim 1 , wherein obtaining the plurality of real-time clues collected from the user further includes:
collecting structured spatiotemporal signals captured by multiple types of sensors, wherein the structured spatiotemporal signals include at least one of time, latitude, longitude, speed, and GPS accuracy; and
collecting unstructured text data from one of app content pages and the user.
3. The method according to claim 1 , wherein ranking the list of app functions using the logistic regression model further includes:
calculating app scores to filter out irrelevant app functions; and
based on the calculated app scores, ranking the list of relevant app functions that are scored.
4. The method according to claim 3 , wherein:
provided that document representations of an app function repository F are denoted as d F , a score with respect to each app function fεF along with a query q is calculated by:
score( q,f )=Π wεq λp MLE ( w|d f )+(1−λ) p MLE ( w|d F )
wherein
p
MLE
(
w
❘
f
)
=
count
(
w
,
d
f
)
d
f
,
and
p
MLE
(
w
❘
F
)
=
count
(
w
,
d
F
)
d
F
;
count (•) denotes a total number of times word w occurred in a document; and λ is a smoothing parameter.
5. The method according to claim 1 , further including:
based on the size of a display screen of a mobile device, setting, by the user, a total number of recommended app functions included in the ranked app function list.
6. The method according to claim 1 , wherein:
provided that v s εR D denotes a structured input, and h s (1) ,h s (2) ε{0,1} are binary stochastic hidden units, a probability that the structured feature two-layer DBM assigns to vertex v s is defined by:
p
(
v
s
;
θ
s
)
=
∑
h
s
(
1
)
,
h
s
(
2
)
p
(
v
s
,
h
s
(
1
)
,
h
s
(
2
)
;
θ
s
)
=
1
z
(
θ
s
)
∑
h
s
(
1
)
,
h
s
(
2
)
exp
(
-
E
(
v
s
,
h
s
(
1
)
,
h
s
(
2
)
;
θ
s
)
)
wherein
E
(
v
s
,
h
s
(
1
)
,
h
s
(
2
)
;
θ
s
)
=
-
∑
i
=
1
D
(
v
si
-
b
i
)
2
2
σ
i
2
+
∑
i
=
1
D
∑
j
=
1
F
1
v
si
σ
i
W
ij
(
1
)
h
j
(
1
)
+
∑
j
=
1
F
1
∑
l
=
1
F
2
h
j
(
1
)
W
jl
(
2
)
h
l
(
2
)
is an energy function, and θ s ={b,W (1) ,W (2) ,σ} are model parameters.
7. A system for multimodal clue based personalized app function recommendation, comprising:
a mobile clues module configured to obtain a plurality of real-time clues collected from a user;
a user preference repository module configured to:
extract a plurality of features to represent the plurality of real-time clues collected from the user, wherein the plurality of features include structured features and text features;
use a multimodal Deep Boltzmann Machine (DBM) to construct two separate two-layer DBMs to model distribution over the structured features and the text features, respectively; and
combine the two separate two-layer DBMs by adding an additional layer of binary hidden units on the top of the two-layer DBMs; and
based on the plurality of extracted features, generate a joint representation of a multimodal clue;
an app function recommendation module configured to create a logistic regression model based on the joint representation of the multimodal clue and rank a list of app functions using the logistic regression model; and
an app functions module configured to display a ranked app function list for the user.
8. The system according to claim 7 , wherein the mobile clues module is further configured to:
collect structured spatiotemporal signals captured by multiple types of sensors, wherein the structured spatiotemporal signals include time, latitude, longitude, speed, and GPS accuracy; and
collect unstructured text data from one of app content pages and the user.
9. The system according to claim 7 , wherein the app function recommendation module is further includes:
a prediction model creating module configured to create a logistic regression model trained with the joint representation of the multimodal clues;
an app scorer configured to calculate app scores to filter out irrelevant app functions; and
a result ranking module configured to, based on the calculated app scores, rank the list of relevant app functions that are scored.
10. The system according to claim 9 , wherein:
provided that document representations of an app function repository F are denoted as d F , a score with respect to each app function fεF along with a query q is calculated by:
score( q,f )=Π wεq λp MLE ( w|d f )+(1−λ) p MLE ( w|d F )
wherein
p
MLE
(
w
❘
f
)
=
count
(
w
,
d
f
)
d
f
,
and
p
MLE
(
w
❘
F
)
=
count
(
w
,
d
F
)
d
F
;
count (•denotes a total number of times word w occurred in a document; and λ is a smoothing parameter.
11. The system according to claim 7 , wherein:
based on the size of a display screen of a mobile device, the user sets a total number of recommended app functions included in the ranked app function list.
12. The system according to claim 7 , wherein:
provided that v s εR D denotes a structured input, and h s (1) ,h s (2) ε{0,1} are binary stochastic hidden units, a probability that the structured feature two-layer DBM assigns to vertex v s is defined by:
p
(
v
s
;
θ
s
)
=
∑
h
s
(
1
)
,
h
s
(
2
)
p
(
v
s
,
h
s
(
1
)
,
h
s
(
2
)
;
θ
s
)
=
1
z
(
θ
s
)
∑
h
s
(
1
)
,
h
s
(
2
)
exp
(
-
E
(
v
s
,
h
s
(
1
)
,
h
s
(
2
)
;
θ
s
)
)
wherein
E
(
v
s
,
h
s
(
1
)
,
h
s
(
2
)
;
θ
s
)
=
-
∑
i
=
1
D
(
v
si
-
b
i
)
2
2
σ
i
2
+
∑
i
=
1
D
∑
j
=
1
F
1
v
si
σ
i
W
ij
(
1
)
h
j
(
1
)
+
∑
j
=
1
F
1
∑
l
=
1
F
2
h
j
(
1
)
W
jl
(
2
)
h
l
(
2
)
is an energy function, and θ s ={b,W (1) ,W (2) ,σ} are model parameters.
13. A non-transitory computer-readable medium having computer program for, when being executed by a processor, performing a method for multimodal clue based personalized app function recommendation, the method comprising:
receiving an app search query from a user;
obtaining a plurality of real-time clues collected from the user;
extracting a plurality of features to represent the plurality of real-time clues collected from the user, wherein the plurality of features include structured features and text features;
based on the plurality of extracted features, generating a joint representation of a multimodal clue, comprising:
using a multimodal Deep Boltzmann Machine (DBM) to construct two separate two-layer DBMs to model distribution over the structured features and the text features respectively; and
combining the two separate two-layer DBMs by adding an additional layer of binary hidden units on the top of the two-layer DBMs;
based on the joint representation of the multimodal clue, creating a logistic regression model;
ranking a list of app functions using the logistic regression model; and
displaying a ranked app function list for the user.
14. The non-transitory computer-readable medium according to claim 13 , wherein obtaining the plurality of real-time clues collected from the user further includes:
collecting structured spatiotemporal signals captured by multiple types of sensors, wherein the structured spatiotemporal signals include time, latitude, longitude, speed, and GPS accuracy; and
collecting unstructured text data from one of app content pages and the user.
15. The non-transitory computer-readable medium according to claim 13 , wherein ranking the list of app functions using the logistic regression model further includes:
calculating app scores to filter out irrelevant app functions; and
based on the calculated app scores, ranking the list of relevant app functions that are scored.
16. The non-transitory computer-readable medium according to claim 13 , the method further including:
based on the size of a display screen of a mobile device, setting, by the user, a total number of recommended app functions included in the ranked app function list.