System and method for automatic video categorization

1. An apparatus comprising:

a memory configured to store:

a set of object categories, each object category of the set of object categories corresponding to at least one object; and

a set of motion categories, each motion category of the set of motion categories corresponding to at least one motion;

a hardware processor communicatively coupled to the memory, the hardware processor configured to:

receive a video;

split the video into an ordered series of frames, each frame of the ordered series of frames corresponding to a time of the frame within the video and ordered within the ordered series of frames according to the time; and

in response to splitting the video into the ordered series of frames:

for each frame of the ordered series of frames:

determine that the frame comprises an image of an object of a given object category of the set of object categories;

assign the given object category to the frame; and

store the assigned object category in an ordered series of object category assignments, the assigned object category ordered within the ordered series of object category assignments according to the time of the frame within the video;

determine, based on at least a subset of the ordered series of object category assignments, that the video used to generate the ordered series of object category assignments depicts a motion of a given motion category of the set of motion categories; and

assign the given motion category to the video.

2. The apparatus of claim 1 , wherein:

determining that the frame comprises the image of the object of the given object category comprises:

generating a set of first scores, each score of the set of first scores corresponding to an object category of the set of object categories and representing a probability that the frame comprises an image of an object of the object category; and

identifying from the set of first scores a largest score, wherein the largest score corresponds to the given object category; and

determining that the video used to generate the ordered series of object category assignments depicts the motion of the given motion category comprises:

generating, based on at least the subset of the ordered series of object category assignments, a set of second scores, each score of the set of second scores corresponding to a motion category of the set of motion categories and representing a probability that the video used to generate the ordered series of object category assignments depicts a motion of the motion category; and

identifying from the set of second scores a largest score, wherein the largest score corresponds to the given motion category.

3. The apparatus of claim 2 , wherein generating, for each frame, the set of first scores comprises applying a convolutional neural network to input generated from pixels of the frame.

4. The apparatus of claim 2 , wherein generating the set of second scores comprises applying a recurrent neural network to input comprising at least the subset of the ordered series of object category assignments.

5. The apparatus of claim 4 , wherein:

the video is associated with a known motion category; and

in response to assigning the given motion category to the video, the hardware processor is further configured to:

compare the assigned motion category to the known motion category;

determine that the assigned motion category and the known motion category do not match; and

in response to determining that the assigned motion category and the known motion category do not match, use the subset of the ordered series of object category assignments and the known motion category to update the recurrent neural network.

6. The apparatus of claim 1 , wherein:

the first video comprises a first number of frames; and

the ordered series of frames comprises a second number of frames, the first number of frames greater than the second number of frames.

7. The apparatus of claim 1 , wherein the hardware processor is further configured to generate the subset of the ordered series of object category assignments from the ordered series of object category assignments such that, for each object category assignment of the subset of the ordered series of object category assignments, the object category assignment is different from an immediately subsequent object category assignment in the subset of the ordered series of object category assignments.

8. A method comprising:

receiving a video;

splitting the video into an ordered series of frames, each frame of the ordered series of frames corresponding to a time of the frame within the video and ordered within the ordered series of frames according to the time; and

in response to splitting the video into the ordered series of frames:

for each frame of the ordered series of frames:

determining that the frame comprises an image of an object of a given object category of a set of object categories, wherein each object category of the set of object categories corresponds to at least one object;

assigning the given object category to the frame; and

storing the assigned object category in an ordered series of object category assignments, the assigned object category ordered within the ordered series of object category assignments according to the time of the frame within the video;

determining, based on at least a subset of the ordered series of object category assignments, that the video used to generate the ordered series of object category assignments depicts a motion of a given motion category of a set of motion categories, wherein each motion category of the set of motion categories corresponds to at least one motion; and

assigning the given motion category to the video.

9. The method of claim 8 , wherein:

determining that the frame comprises the image of the object of the given object category comprises:

generating a set of first scores, each score of the set of first scores corresponding to an object category of the set of object categories and representing a probability that the frame comprises an image of an object of the object category; and

identifying from the set of first scores a largest score, wherein the largest score corresponds to the given object category; and

determining that the video used to generate the ordered series of object category assignments depicts the motion of the given motion category comprises:

generating, based on at least the subset of the ordered series of object category assignments, a set of second scores, each score of the set of second scores corresponding to a motion category of the set of motion categories and representing a probability that the video used to generate the ordered series of object category assignments depicts a motion of the motion category; and

identifying from the set of second scores a largest score, wherein the largest score corresponds to the given motion category.

10. The method of claim 9 , wherein generating, for each frame, the set of first scores comprises applying a convolutional neural network to input generated from pixels of the frame.

11. The method of claim 9 , wherein generating the set of second scores comprises applying a recurrent neural network to input comprising at least the subset of the ordered series of object category assignments.

12. The method of claim 11 , wherein:

the video is associated with a known motion category; and

in response to assigning the given motion category to the video, the method further comprises:

comparing the assigned motion category to the known motion category;

determining that the assigned motion category and the known motion category do not match; and

in response to determining that the assigned motion category and the known motion category do not match, using the subset of the ordered series of object category assignments and the known motion category to update the recurrent neural network.

13. The method of claim 8 , wherein:

the first video comprises a first number of frames; and

the ordered series of frames comprises a second number of frames, the first number of frames greater than the second number of frames.

14. The method of claim 8 , further comprising generating the subset of the ordered series of object category assignments from the ordered series of object category assignments such that, for each object category assignment of the subset of the ordered series of object category assignments, the object category assignment is different from an immediately subsequent object category assignment in the subset of the ordered series of object category assignments.

15. A computer program comprising executable instructions stored in a non-transitory computer-readable medium that when executed by a hardware processor causes the hardware processor to:

receive a video;

split the video into an ordered series of frames, each frame of the ordered series of frames corresponding to a time of the frame within the video and ordered within the ordered series of frames according to the time; and

in response to splitting the video into the ordered series of frames:

for each frame of the ordered series of frames:

determine that the frame comprises an image of an object of a given object category of the set of object categories, wherein each object category of the set of object categories corresponds to at least one object;

assign the given object category to the frame; and

store the assigned object category in an ordered series of object category assignments, the assigned object category ordered within the ordered series of object category assignments according to the time of the frame within the video;

determine, based on at least a subset of the ordered series of object category assignments, that the video used to generate the ordered series of object category assignments depicts a motion of a given motion category of the set of motion categories, wherein each motion category of the set of motion categories corresponds to at least one motion; and

assign the given motion category to the video.

16. The computer program of claim 15 , wherein:

determining that the frame comprises the image of the object of the given object category comprises:

generating a set of first scores, each score of the set of first scores corresponding to an object category of the set of object categories and representing a probability that the frame comprises an image of an object of the object category; and

identifying from the set of first scores a largest score, wherein the largest score corresponds to the given object category; and

determining that the video used to generate the ordered series of object category assignments depicts the motion of the given motion category comprises:

generating, based on at least the subset of the ordered series of object category assignments, a set of second scores, each score of the set of second scores corresponding to a motion category of the set of motion categories and representing a probability that the video used to generate the ordered series of object category assignments depicts a motion of the motion category; and

identifying from the set of second scores a largest score, wherein the largest score corresponds to the given motion category.

17. The computer program of claim 16 , wherein generating the set of second scores comprises applying a recurrent neural network to input comprising at least the subset of the ordered series of object category assignments.

18. The computer program of claim 17 , wherein:

the video is associated with a known motion category; and

when executed by the hardware processor the computer program further causes the hardware processor, in response to assigning the given motion category to the video, to:

compare the assigned motion category to the known motion category;

determine that the assigned motion category and the known motion category do not match; and

in response to determining that the assigned motion category and the known motion category do not match, use the subset of the ordered series of object category assignments and the known motion category to update the recurrent neural network.

19. The computer program of claim 15 , wherein:

the first video comprises a first number of frames; and

the ordered series of frames comprises a second number of frames, the first number of frames greater than the second number of frames.

20. The computer program of claim 15 , wherein when executed by the hardware processor the computer program further causes the hardware processor to generate the subset of the ordered series of object category assignments from the ordered series of object category assignments such that, for each object category assignment of the subset of the ordered series of object category assignments, the object category assignment is different from an immediately subsequent object category assignment in the subset of the ordered series of object category assignments.