Corrected mean-covariance RBMs and general high-order semi-RBMs for large-scale collaborative filtering and prediction
View Patent ↗Systems and methods are disclosed for operating a Restricted Boltzmann Machine (RBM) by determining a corrected energy function of high-order semi-RBMs (hs-RBMs) without self-interaction; performing distributed pre-training of the hs-RBM; adjusting weights of the hs-RBM using contrastive divergence; generating predictions by Gibbs Sampling or by determining conditional probabilities with hidden units integrated out; and generating predictions.
1. In a system comprising a processor and a memory that includes a plurality of components that are executable by the processor, the plurality of components having a data receiver component that receives training data, and a training component that trains a generative stochastic artificial neural network including hidden units u and visible units v that can learn a probability distribution over its set of inputs, said system operating according to a method comprising the following computer-executable acts:
determining a corrected energy function of high-order semi-Restricted Boltzmann Machines (hs-RBMs), said determination made without self-interaction;
performing distributed pre-training of the hs-RBM by a pseudo-log likelihood method using alternating direction of multipliers (ADMM);
adjusting weights of the hs-RBM using contrastive divergence;
generating predictions by Gibbs Sampling or by determining conditional probabilities with hidden units integrated out; and
outputting the predictions so generated;
wherein the training includes a mean-covariance RBM (mcRBM) that defines a joint distribution over configurations of the visible units u and the hidden units h by an energy function in which there is no connection between hidden units and the energy function is defined by the following relationship:
E
(
v
,
h
g
,
h
m
)
=
1
2
∑
f
(
∑
i
v
i
C
if
)
2
(
∑
k
h
k
w
kf
)
-
∑
i
α
i
v
i
-
∑
k
b
k
h
k
g
-
∑
ij
v
i
h
j
m
w
ij
-
∑
k
c
k
h
k
m
where h m and h g , respectively, denotes mean hidden units and covariance hidden units.
2. The method of claim 1 , further comprising modifying the energy function of the mcRBM for modeling binary data as:
E
(
v
,
h
g
,
h
m
)
=
-
1
2
∑
f
(
(
∑
i
v
i
C
if
)
2
-
∑
i
v
i
2
Cif
2
)
(
∑
k
h
k
w
kf
)
-
∑
ij
v
i
h
j
m
w
ij
+
bias_terms
.
3. The method of claim 2 , further comprising assigning an absolute value of the energy terms such that a final energy function is bounded for hs-RBMs with an odd number of feature interactions.
4. The method of claim 3 , further comprising maximizing pseudo-log-likelihood of training data and alternating direction of multipliers methodology to split computations across different visible units such that parameters as a warm start are learned and using contrastive divergence to tune the parameters to change the log-likelihood of the training data.