Object recognition from videos using recurrent neural networks

1. A computer-implemented method, comprising:

obtaining multiple frames from a video, wherein each frame of the multiple frames depicts an object to be recognized; and

processing, using an object recognition model, the multiple frames to generate data that represents a classification of the object to be recognized,

wherein the object recognition model is a recurrent neural network that comprises a long short-term memory (LSTM) layer and multiple feature extraction layers,

wherein the LSTM layer includes a convolutional input gate, a convolutional forget gate, a convolutional memory block, and a convolutional output gate that use convolutions to process data, and wherein the processing comprises, for each frame of the multiple frames:

processing, using the multiple feature extraction layers, the frame to generate feature data that represents features of the frame; and

processing, using the LSTM layer, the feature data to generate an LSTM output and to update an internal state of the LSTM layer.

2. The method of claim 1 ,

wherein the multiple frames are arranged in an order according to their time of occurrence in the video, and

wherein processing the multiple frames further comprises processing each frame of the multiple frames using the LSTM layer in the order according to their time of occurrence in the video to generate the LSTM output and to update the internal state of the LSTM layer.

3. The method of claim 2 ,

wherein the recurrent neural network further comprises one or more classification layers, and

wherein processing the multiple frames further comprises processing, using the one or more classification layers, the LSTM output to generate the data that represents the classification of the object to be recognized.

4. The method of claim 3 ,

wherein the recurrent neural network further comprises a backward LSTM layer, and

wherein processing the plurality of frames further comprises:

processing each frame of the multiple frames using the backward LSTM layer in a reversed order according to their time of occurrence in the video to generate a backward LSTM output and to update an internal state of the backward LSTM layer, and

processing, using the one or more classification layers, the LSTM output and the backward LSTM output to generate the data that represents the classification of the object to be recognized.

5. The method of claim 1 , wherein the classification includes a respective score for each object category in a predetermined set of object categories, the respective score for each of the object categories representing a likelihood that the object to be recognized belongs to the object category.

6. The method of claim 1 , wherein obtaining the multiple frames from the video comprises selecting the multiple frames from the video based on a predetermined time interval.

7. The method of claim 1 , wherein obtaining the multiple frames from the video comprises selecting the multiple frames from the video based on a viewpoint of the object to be recognized.

8. The method of claim 1 , wherein obtaining the multiple frames from the video comprises selecting the multiple frames from the video based on a processing capability of a processor.

9. The method of claim 1 , wherein obtaining the multiple frames from the video comprises obtaining the video using a camera mounted on a robotic arm manipulator.

10. The method of claim 1 , wherein a count of the multiple frames is five or fewer.

11. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:

obtaining multiple frames from a video, wherein each frame of the multiple frames depicts an object to be recognized; and

processing, using an object recognition model, the multiple frames to generate data that represents a classification of the object to be recognized,

wherein the object recognition model is a recurrent neural network that comprises a long short-term memory (LSTM) layer and multiple feature extraction layers,

wherein the LSTM layer includes a convolutional input gate, a convolutional forget gate, a convolutional memory block, and a convolutional output gate that use convolutions to process data, and wherein the processing comprises, for each frame of the multiple frames:

processing, using the multiple feature extraction layers, the frame to generate feature data that represents features of the frame; and

processing, using the LSTM layer, the feature data to generate an LSTM output and to update an internal state of the LSTM layer.

12. The system of claim 11 ,

wherein the multiple frames are arranged in an order according to their time of occurrence in the video, and

wherein processing the multiple frames further comprises processing each frame of the multiple frames using the LSTM layer in the order according to their time of occurrence in the video to generate the LSTM output and to update the internal state of the LSTM layer.

13. The system of claim 12 ,

wherein the recurrent neural network further comprises one or more classification layers, and

wherein processing the multiple frames further comprises processing, using the one or more classification layers, the LSTM output to generate the data that represents the classification of the object to be recognized.

14. The system of claim 13 ,

wherein the recurrent neural network further comprises a backward LSTM layer, and

wherein processing the plurality of frames further comprises:

processing each frame of the multiple frames using the backward LSTM layer in a reversed order according to their time of occurrence in the video to generate a backward LSTM output and to update an internal state of the backward LSTM layer, and

processing, using the one or more classification layers, the LSTM output and the backward LSTM output to generate the data that represents the classification of the object to be recognized.

15. The system of claim 11 , wherein the classification includes a respective score for each object category in a predetermined set of object categories, the respective score for each of the object categories representing a likelihood that the object to be recognized belongs to the object category.

16. The system of claim 11 , wherein obtaining the multiple frames from the video comprises selecting the multiple frames from the video based on a predetermined time interval.

17. The system of claim 11 , wherein obtaining the multiple frames from the video comprises selecting the multiple frames from the video based on a viewpoint of the object to be recognized.

18. The system of claim 11 , wherein obtaining the multiple frames from the video comprises selecting the multiple frames from the video based on a processing capability of a processor.

19. The system of claim 11 , wherein obtaining the multiple frames from the video comprises obtaining the video using a camera mounted on a robotic arm manipulator.

20. The system of claim 11 , wherein a count of the multiple frames is five or fewer.

21. A computer program product encoded on one or more non-transitory computer storage media, the computer program product comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

obtaining multiple frames from a video, wherein each frame of the multiple frames depicts an object to be recognized; and

processing, using an object recognition model, the multiple frames to generate data that represents a classification of the object to be recognized,

wherein the object recognition model is a recurrent neural network that comprises a long short-term memory (LSTM) layer and multiple feature extraction layers,

wherein the LSTM layer includes a convolutional input gate, a convolutional forget gate, a convolutional memory block, and a convolutional output gate that use convolutions to process data, and wherein the processing comprises, for each frame of the multiple frames:

processing, using the multiple feature extraction layers, the frame to generate feature data that represents features of the frame; and

processing, using the LSTM layer, the feature data to generate an LSTM output and to update an internal state of the LSTM layer.

22. The computer program product of claim 21 ,

wherein the multiple frames are arranged in an order according to their time of occurrence in the video, and

wherein processing the multiple frames further comprises processing each frame of the multiple frames using the LSTM layer in the order according to their time of occurrence in the video to generate the LSTM output and to update the internal state of the LSTM layer.

23. The computer program product of claim 22 ,

wherein the recurrent neural network further comprises one or more classification layers, and

wherein processing the multiple frames further comprises processing, using the one or more classification layers, the LSTM output to generate the data that represents the classification of the object to be recognized.

24. The computer program product of claim 23 ,

wherein the recurrent neural network further comprises a backward LSTM layer, and

wherein processing the plurality of frames further comprises:

processing each frame of the multiple frames using the backward LSTM layer in a reversed order according to their time of occurrence in the video to generate a backward LSTM output and to update an internal state of the backward LSTM layer, and

processing, using the one or more classification layers, the LSTM output and the backward LSTM output to generate the data that represents the classification of the object to be recognized.

25. The computer program product of claim 21 , wherein the classification includes a respective score for each object category in a predetermined set of object categories, the respective score for each of the object categories representing a likelihood that the object to be recognized belongs to the object category.

26. The computer program product of claim 21 , wherein obtaining the multiple frames from the video comprises selecting the multiple frames from the video based on a predetermined time interval.

27. The computer program product of claim 21 , wherein obtaining the multiple frames from the video comprises selecting the multiple frames from the video based on a viewpoint of the object to be recognized.

28. The computer program product of claim 21 , wherein obtaining the multiple frames from the video comprises selecting the multiple frames from the video based on a processing capability of a processor.

29. The computer program product of claim 21 , wherein obtaining the multiple frames from the video comprises obtaining the video using a camera mounted on a robotic arm manipulator.

30. The computer program product of claim 21 , wherein a count of the multiple frames is five or fewer.