Machine-learning models based on non-local neural networks
In one embodiment, a method includes training a baseline machine-learning model based on a neural network comprising a plurality of stages, wherein each stage comprises a plurality of neural blocks, accessing a plurality of training samples comprising a plurality of content objects, respectively, determining one or more non-local operations, wherein each non-local operation is based on one or more pairwise functions and one or more unary functions, generating one or more non-local blocks based on the plurality of training samples and the one or more non-local operations, determining a stage from the plurality of stages of the neural network, and training a non-local machine-learning model by inserting each of the one or more non-local blocks in between at least two of the plurality of neural blocks in the determined stage of the neural network.
1. A method comprising, by one or more computing systems:
training a baseline machine-learning model based on a neural network comprising a plurality of stages, wherein each stage comprises a plurality of neural blocks;
accessing a plurality of training samples comprising a plurality of content objects, respectively;
determining one or more non-local operations, wherein each non-local operation is based on one or more pairwise functions and one or more unary functions, wherein each of the one or more non-local operations is associated with a respective plurality of weight matrices;
generating one or more non-local blocks based on the plurality of training samples and the one or more non-local operations, wherein the generation comprises initializing the respective plurality of weight matrices associated with each of the one or more non-local operations, and wherein each of the one or more non-local blocks comprises one or more residual connections configured for allowing initial behavior of the baseline machine-learning model to occur after the non-local block is inserted into the baseline machine-learning model if the respective plurality of weight matrices are initialized as zero;
determining a stage from the plurality of stages of the neural network; and
training a non-local machine-learning model by inserting each of the one or more non-local blocks in between at least two of the plurality of neural blocks in the determined stage of the neural network.
2. The method of claim 1 , wherein the neural network comprises one or more of a convolutional neural network or a recurrent neural network.
3. The method of claim 1 , wherein each of the plurality of content objects comprises one or more of a text, an audio clip, an image, or a video.
4. The method of claim 1 , wherein the neural network is based on one or more of a two-dimensional architecture or a three-dimensional architecture.
5. The method of claim 1 , further comprising:
generating a plurality of feature representations for the plurality content objects based on the baseline machine-learning model, respectively.
6. The method of claim 5 , wherein generating each of the one or more non-local blocks comprises:
applying each of the one or more non-local operations to the feature representation of one of the plurality of content objects.
7. The method of claim 5 , further comprising:
determining, for each of the plurality of content objects, an output position and a plurality of positions associated with the output position.
8. The method of claim 7 , wherein the output position is in one or more of space, time, or spacetime.
9. The method of claim 7 , wherein each of the one or more non-local operations is based on
y
i
=
1
C
(
x
)
∑
∀
j
f
(
x
i
,
x
j
)
g
(
x
j
)
,
and wherein
x i indicates the feature representation at the output position;
x j indicates the feature representation at one of the plurality of positions;
y i indicates an output response at the output position;
ƒ(x i , x j ) indicates the pairwise function;
g(x j ) indicates the unary function; and
C(x) indicates a normalization factor.
10. The method of claim 9 , where the pairwise function is based on one or more of:
a Gaussian function ƒ(x i , x j )=e x i T x j ;
an embedded Gaussian function ƒ(x i , x j )=e θ(x i ) T ϕ(x j ) , wherein θ is an embedding for x i , and ϕ is an embedding for x j ;
a dot product function ƒ(x i , x j )=θ(x i ) T ϕ(x j ); or
a concatenation function ƒ(x i , x j )=ReLU(W ƒ T [θ(x i ), ϕ(x j )]), wherein ReLU indicates a function of a rectified linear unit, and wherein w ƒ is a weight vector projecting a concatenated vector of θ(x i ) and ϕ(x j ) to a scalar.
11. The method of claim 5 , further comprising:
generating, for each of the plurality of content objects, a subsampled content object by applying subsampling to the feature representation of the content object, wherein the subsampled content object is associated with a subsampled feature representation.
12. The method of claim 11 , wherein the subsampling comprises pooling, the pooling comprises one or more of max pooling or average pooling.
13. The method of claim 11 , wherein generating each of the one or more non-local blocks comprises:
applying each of the one or more non-local operations to the feature representation of one of the plurality of content objects and the subsampled feature representation of the subsampled content object corresponding to the content object.
14. The method of claim 11 , further comprising:
determining, for each of the plurality of content objects, an output position; and
determining, for each of the plurality of subsampled content objects corresponding to the content object, a plurality of positions associated with the output position.
15. The method of claim 14 , wherein each of the one or more non-local operations is based on a function
y
i
=
1
C
(
x
^
)
∑
∀
j
f
(
x
i
,
x
^
j
)
g
(
x
^
j
)
,
wherein:
x i indicates the feature representation at the output position;
{circumflex over (x)} j indicates the subsampled feature representation at one of the plurality of positions;
y i indicates an output response at the output position;
ƒ(x i , {circumflex over (x)} j ) indicates the pairwise function;
g({circumflex over (x)} j ) indicates the unary function; and
C({circumflex over (x)}) indicates a normalization factor.
16. The method of claim 15 , where the pairwise function is based on one or more of:
a Gaussian function ƒ(x i , {circumflex over (x)} j )=e x i T {circumflex over (x)} j
an embedded Gaussian function ƒ(x i , {circumflex over (x)} j )=e θ(x i ) T ϕ({circumflex over (x)} j ) , wherein θ is an embedding for x i and ϕ is an embedding for {circumflex over (x)} j ;
a dot-product function ƒ(x i , {circumflex over (x)} j )=e θ(x i ) T ϕ({circumflex over (x)} j ) ; or
a concatenation function ƒ(x i , {circumflex over (x)} j )=ReLU(W ƒ T [θ(x i ), ϕ({circumflex over (x)} j )]), wherein ReLU indicates a function of a rectified linear unit, and wherein wf is a weight vector projecting a concatenated vector of θ(x i ) and ϕ({circumflex over (x)} j to a scalar.
17. The method of claim 1 , further comprising:
receiving a querying content object; and
determining a category for the querying content object based on the non-local machine-learning model.
18. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
train a baseline machine-learning model based on a neural network comprising a plurality of stages, wherein each stage comprises a plurality of neural blocks;
access a plurality of training samples comprising a plurality of content objects, respectively;
determine one or more non-local operations, wherein each non-local operation is based on one or more pairwise functions and one or more unary functions, wherein each of the one or more non-local operations is associated with a respective plurality of weight matrices;
generate one or more non-local blocks based on the plurality of training samples and the one or more non-local operations, wherein the generation comprises initializing the respective plurality of weight matrices associated with each of the one or more non-local operations, and wherein each of the one or more non-local blocks comprises one or more residual connections configured for allowing initial behavior of the baseline machine-learning model to occur after the non-local block is inserted into the baseline machine-learning model if the respective plurality of weight matrices are initialized as zero;
determine a stage from the plurality of stages of the neural network; and
train a non-local machine-learning model by inserting each of the one or more non-local blocks in between at least two of the plurality of neural blocks in the determined stage of the neural network.
19. A system comprising: one or more processors; and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:
train a baseline machine-learning model based on a neural network comprising a plurality of stages, wherein each stage comprises a plurality of neural blocks;
access a plurality of training samples comprising a plurality of content objects, respectively;
determine one or more non-local operations, wherein each non-local operation is based on one or more pairwise functions and one or more unary functions, wherein each of the one or more non-local operations is associated with a respective plurality of weight matrices;
generate one or more non-local blocks based on the plurality of training samples and the one or more non-local operations, wherein the generation comprises initializing the respective plurality of weight matrices associated with each of the one or more non-local operations, and wherein each of the one or more non-local blocks comprises one or more residual connections configured for allowing initial behavior of the baseline machine-learning model to occur after the non-local block is inserted into the baseline machine-learning model if the respective plurality of weight matrices are initialized as zero;
determine a stage from the plurality of stages of the neural network; and
train a non-local machine-learning model by inserting each of the one or more non-local blocks in between at least two of the plurality of neural blocks in the determined stage of the neural network.