Systems and methods for determining video feature descriptors based on convolutional neural networks
Systems, methods, and non-transitory computer-readable media can acquire video content for which video feature descriptors are to be determined. The video content can be processed based at least in part on a convolutional neural network including a set of two-dimensional convolutional layers and a set of three-dimensional convolutional layers. One or more outputs can be generated from the convolutional neural network. A plurality of video feature descriptors for the video content can be determined based at least in part on the one or more outputs from the convolutional neural network.
1. A computer-implemented method comprising:
processing, by a computing system, video content based at least in part on a convolutional neural network that includes at least one two-dimensional convolutional layer and at least one three-dimensional convolutional layer, wherein at least a portion of signals outputted by the at least one two-dimensional convolutional layer are inputted into the at least one three-dimensional convolutional layer, and wherein the convolutional neural network generates one or more outputs; and
determining, by the computing system, a plurality of video feature descriptors for the video content based at least in part on the one or more outputs.
2. The computer-implemented method of claim 1 , wherein the video feature descriptors provide an indication that one or more concepts are represented in subject matter captured by the video content.
3. The computer-implemented method of claim 2 , wherein the video feature descriptors provide a set of metrics indicating respective likelihoods of the one or more concepts being represented in the video content.
4. The computer-implemented method of claim 2 , wherein the one or more concepts include at least a scene, an object, or an action.
5. The computer-implemented method of claim 2 , wherein the video content is categorized based on the indication that one or more concepts that are represented in the subject matter captured by the video content.
6. The computer-implemented method of claim 1 , wherein the convolutional neural network also includes at least one fully-connected layer.
7. The computer-implemented method of claim 6 , wherein at least a portion of signals outputted by the at least one three-dimensional convolutional layer are inputted into the at least one fully-connected layer, and wherein the at least one fully-connected layer produces the one or more outputs.
8. The computer-implemented method of claim 1 , wherein the convolutional neural network also includes at least one softmax layer.
9. The computer-implemented method of claim 8 , wherein at least a portion of signals outputted by the at least one fully-connected layer are normalized by the at least one softmax layer to produce the one or more outputs.
10. The computer-implemented method of claim 2 , further comprising:
training, by the computing system, the convolutional neural network based at least in part on a set of training content items, wherein each training content item is associated with at least one label that describes at least one concept in the training content item.
11. A system comprising:
at least one processor; and
a memory storing instructions that, when executed by the at least one processor, cause the system to perform:
processing video content based at least in part on a convolutional neural network that includes at least one two-dimensional convolutional layer and at least one three-dimensional convolutional layer, wherein at least a portion of signals outputted by the at least one two-dimensional convolutional layer are inputted into the at least one three-dimensional convolutional layer, and wherein the convolutional neural network generates one or more outputs; and
determining a plurality of video feature descriptors for the video content based at least in part on the one or more outputs.
12. The system of claim 11 , wherein the video feature descriptors provide an indication that one or more concepts are represented in subject matter captured by the video content.
13. The system of claim 12 , wherein the video feature descriptors provide a set of metrics indicating respective likelihoods of the one or more concepts being represented in the video content.
14. The system of claim 12 , wherein the one or more concepts include at least a scene, an object, or an action.
15. The system of claim 12 , wherein the video content is categorized based on the indication that one or more concepts that are represented in the subject matter captured by the video content.
16. A non-transitory computer-readable storage medium including instructions that, when executed by at least one processor of a computing system, cause the computing system to perform:
processing video content based at least in part on a convolutional neural network that includes at least one two-dimensional convolutional layer and at least one three-dimensional convolutional layer, wherein at least a portion of signals outputted by the at least one two-dimensional convolutional layer are inputted into the at least one three-dimensional convolutional layer, and wherein the convolutional neural network generates one or more outputs; and
determining a plurality of video feature descriptors for the video content based at least in part on the one or more outputs.
17. The non-transitory computer-readable storage medium of claim 16 , wherein the video feature descriptors provide an indication that one or more concepts are represented in subject matter captured by the video content.
18. The non-transitory computer-readable storage medium of claim 17 , wherein the video feature descriptors provide a set of metrics indicating respective likelihoods of the one or more concepts being represented in the video content.
19. The non-transitory computer-readable storage medium of claim 17 , wherein the one or more concepts include at least a scene, an object, or an action.
20. The non-transitory computer-readable storage medium of claim 17 , wherein the video content is categorized based on the indication that one or more concepts that are represented in the subject matter captured by the video content.