Clustering videos using a self-supervised DNN
Systems and methods are provided for clustering videos. The system accesses a plurality of content items, the plurality of content items comprising a first set of RGB video frames and a second set of optical flow frames corresponding to the first set of RGB video frames. The system processes the first set of RGB video frames by a first machine learning model to generate a first optimal assignment for the first set of RGB video frames, the first optimal assignment representing initial clustering of the first set of RGB video frames. The system generates an updated first optimal assignment for the first set of RGB video frames based on the first optimal assignment for the first set of RGB video frames and a second optimal assignment of the second set of optical flow frames, the second optimal assignment representing initial clustering of the second set of optical flow frames.
1. A method comprising:
accessing a plurality of content items, the plurality of content items comprising a first set of red, green, and blue (RGB) video frames and a second set of optical flow frames corresponding to the first set of RGB video frames;
processing the first set of RGB video frames by a first machine learning model to generate a first optimal assignment for the first set of RGB video frames, the first optimal assignment representing initial clustering of the first set of RGB video frames;
generating an updated first optimal assignment for the first set of RGB video frames based on both the first optimal assignment for the first set of RGB video frames and a second optimal assignment of the second set of optical flow frames, the second optimal assignment representing initial clustering of the second set of optical flow frames;
generating a set of augmentations of the first set of RGB video frames;
processing the set of augmentations by the first machine learning model to generate a third optimal assignment for the set of augmentations;
generating an updated third optimal assignment for the set of augmentations based on the third optimal assignment for the set of augmentations and the second optimal assignment of the second set of optical flow frames; and
updating one or more parameters of the first machine learning model based on a loss computed as a function of the updated first optimal assignment and the updated third optimal assignment.
2. The method of claim 1 , further comprising generating the second set of optical flow frames by:
obtaining first and second video frames from the first set of RGB video frames;
computing a difference frame comprising motion information based on a deviation between the first and second video frames; and
storing the difference frame as one of the second set of optical flow frames.
3. The method of claim 1 , further comprising:
processing the second set of optical flow frames by a second machine learning model to generate the second optimal assignment of the second set of optical flow frames.
4. The method of claim 3 , further comprising:
generating an updated second optimal assignment for the second set of optical flow frames based on the first optimal assignment for the first set of RGB video frames.
5. The method of claim 4 , further comprising:
computing a deviation between the updated first optimal assignment and the updated second optimal assignment; and
updating one or more parameters of at least one of the first or second machine learning models based on the computed deviation.
6. The method of claim 3 , wherein the first and second machine learning models each comprise a deep neural network (DNN) comprising one or more encoders.
7. The method of claim 3 , wherein the first machine learning model is trained to generate a first set of features corresponding to the first set of RGB video frames, and wherein the second machine learning model is trained to generate a second set of features corresponding to the second set of optical flow frames.
8. The method of claim 7 , wherein the first and second machine learning models are trained in an unsupervised manner end-to-end.
9. The method of claim 1 , wherein the initial clustering of the first set of RGB video frames represents different human activity depicted in the first set of RGB video frames.
10. The method of claim 1 , further comprising:
generating a first set of vectors in response to processing the first set of RGB video frames by the first machine learning model, the first set of vectors representing features of the first set of RGB video frames; and
matching the first set of vectors to prototype cluster centers to generate the first optimal assignment for the first set of RGB video frames.
11. The method of claim 10 , further comprising applying a Sinkhorn-Knopp technique to match the first set of vectors to the prototype cluster centers.
12. The method of claim 10 , further comprising applying a trained regularization term to equally space the prototype cluster centers.
13. The method of claim 1 , wherein generating the updated first optimal assignment for the first set of RGB video frames comprises applying a k-means algorithm to the first optimal assignment for the first set of RGB video frames and the second optimal assignment of the second set of optical flow frames.
14. A system comprising:
at least one processor programmed to perform operations comprising:
accessing a plurality of content items, the plurality of content items comprising a first set of red, green, and blue (RGB) video frames and a second set of optical flow frames corresponding to the first set of RGB video frames;
processing the first set of RGB video frames by a first machine learning model to generate a first optimal assignment for the first set of RGB video frames, the first optimal assignment representing initial clustering of the first set of RGB video frames;
generating an updated first optimal assignment for the first set of RGB video frames based on both the first optimal assignment for the first set of RGB video frames and a second optimal assignment of the second set of optical flow frames, the second optimal assignment representing initial clustering of the second set of optical flow frames;
generating a set of augmentations of the first set of RGB video frames;
processing the set of augmentations by the first machine learning model to generate a third optimal assignment for the set of augmentations;
generating an updated third optimal assignment for the set of augmentations based on the third optimal assignment for the set of augmentations and the second optimal assignment of the second set of optical flow frames; and
updating one or more parameters of the first machine learning model based on a loss computed as a function of the updated first optimal assignment and the updated third optimal assignment.
15. The system of claim 14 , the operations further comprising generating the second set of optical flow frames by:
obtaining first and second video frames from the first set of RGB video frames;
computing a difference frame based on a deviation between the first and second video frames comprising motion information; and
storing the difference frame as one of the second set of optical flow frames.
16. The system of claim 14 , the operations further comprising:
processing the second set of optical flow frames by a second machine learning model to generate the second optimal assignment of the second set of optical flow frames.
17. The system of claim 16 , the operations further comprising:
generating an updated second optimal assignment for the second set of optical flow frames based on the first optimal assignment for the first set of RGB video frames.
18. A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:
accessing a plurality of content items, the plurality of content items comprising a first set of red, green, and blue (RGB) video frames and a second set of optical flow frames corresponding to the first set of RGB video frames;
processing the first set of RGB video frames by a first machine learning model to generate a first optimal assignment for the first set of RGB video frames, the first optimal assignment representing initial clustering of the first set of RGB video frames;
generating an updated first optimal assignment for the first set of RGB video frames based on both the first optimal assignment for the first set of RGB video frames and a second optimal assignment of the second set of optical flow frames, the second optimal assignment representing initial clustering of the second set of optical flow frames;
generating a set of augmentations of the first set of RGB video frames;
processing the set of augmentations by the first machine learning model to generate a third optimal assignment for the set of augmentations;
generating an updated third optimal assignment for the set of augmentations based on the third optimal assignment for the set of augmentations and the second optimal assignment of the second set of optical flow frames; and
updating one or more parameters of the first machine learning model based on a loss computed as a function of the updated first optimal assignment and the updated third optimal assignment.
19. The non-transitory machine-readable storage medium of claim 18 , the operations comprising generating the second set of optical flow frames by:
obtaining first and second video frames from the first set of RGB video frames;
computing a difference frame comprising motion information based on a deviation between the first and second video frames; and
storing the difference frame as one of the second set of optical flow frames.
20. The non-transitory machine-readable storage medium of claim 18 , the operations comprising:
processing the second set of optical flow frames by a second machine learning model to generate the second optimal assignment of the second set of optical flow frames.