Video summarization using semantic information
An apparatus for video summarization using sematic information is described herein. The apparatus includes a controller, a scoring mechanism, and a summarizer. The controller is to segment an incoming video stream into a plurality of activity segments, wherein each frame is associated with an activity. The scoring mechanism is to calculate a score for each frame of each activity, wherein the score is based on a plurality of objects in each frame. The summarizer is to summarize the activity segments based on the score for each frame.
1. An electronic device comprising:
an image capture sensor;
memory to store video segments;
wireless communication circuitry to transmit data; and
processor circuitry to:
process a first image of a first video segment from the image capture sensor with a neural network to determine a first score for the first image, the neural network to detect actions associated with images, the actions associated with labels;
determine a second score for the first video segment based on first scores for corresponding images in the first video segment; and
determine, based on the second score, whether to retain the first video segment in the memory.
2. The electronic device of claim 1 , wherein the neural network is to output a confidence that the first image is associated with a first one of the labels.
3. The electronic device of claim 1 , wherein the first video segment has a duration of at least five seconds.
4. The electronic device of claim 1 , wherein the labels correspond to a group of labels having a size of at least hundreds of labels.
5. The electronic device of claim 1 , wherein the electronic device is a wearable device.
6. The electronic device of claim 1 , wherein the wireless communication circuitry includes at least one of WiFi hardware, Bluetooth hardware or cellular hardware.
7. The electronic device of claim 1 , wherein the neural network is a convolutional neural network.
8. The electronic device of claim 1 , wherein the neural network is trained to detect the actions associated with the images.
9. The electronic device of claim 1 , further including:
a display;
a microphone; and
at least one of a keyboard, a touchpad or a touchscreen.
10. At least one memory device comprising computer readable instructions that, when executed, cause at least one processor of an electronic device to at least:
process a first image of a first video segment from an image capture sensor of the electronic device with a neural network to determine a first score for the first image, the neural network to detect an action associated with at least one image, the action associated with a label;
determine a second score for the first video segment based on first scores for corresponding images in the first video segment; and
determine, based on the second score, whether to retain the first video segment in the electronic device.
11. The at least one memory device of claim 10 , wherein the neural network is to output a confidence that the first image is associated with the label.
12. The at least one memory device of claim 10 , wherein the first video segment has a duration of at least five seconds.
13. The at least one memory device of claim 10 , wherein the label is one of a group of labels having a size of at least hundreds of labels.
14. The at least one memory device of claim 10 , wherein the neural network is a convolutional neural network trained to detect the action.
15. An electronic device comprising:
means for capturing images;
means for storing video segments;
means for transmitting data; and
means for determining whether to retain a first video segment, the means for determining to:
process a first image of the first video segment with a neural network to determine a first score for the first image, the neural network to detect actions associated with images, the actions associated with labels;
determine a second score for the first video segment based on first scores for corresponding images in the first video segment; and
determine, based on the second score, whether to retain the first video segment.
16. The electronic device of claim 15 , wherein the neural network is to output a confidence that the first image is associated with a first one of the labels.
17. The electronic device of claim 15 , wherein the first video segment has a duration of at least five seconds.
18. The electronic device of claim 15 , wherein the labels correspond to a group of labels having a size of at least hundreds of labels.
19. The electronic device of claim 15 , wherein the means for transmitting is to transmit via a WiFi network.
20. The electronic device of claim 15 , wherein the neural network is a convolutional neural network to detect the actions associated with the images.
21. A method comprising:
processing, by executing an instruction with at least one processor of an electronic device, a first image of a first video segment with a neural network to determine a first score for the first image, the neural network to detect actions associated with images, the actions associated with labels, the first video segment from an image capture sensor of the electronic device;
determining, by executing an instruction with the at least one processor, a second score for the first video segment based on first scores for corresponding images in the first video segment; and
determining, by executing an instruction with the at least one processor, whether to retain the first video segment in the electronic device based on the second score.
22. The method of claim 21 , wherein the neural network is to output a confidence that the first image is associated with a first one of the labels.
23. The method of claim 21 , wherein the first video segment has a duration of at least five seconds.
24. The method of claim 21 , wherein the labels correspond to a group of labels having a size of at least hundreds of labels.
25. The method of claim 21 , wherein the neural network is a convolutional neural network trained to detect the actions associated with the images.