Document identification method and system
Disclosed are computer-implemented methods, non-transitory computer-readable media, and systems for document identification. One computer-implemented method includes extracting a sequence of image frames from a video clip capturing an impact of an object against a surface. A stream of audio signals is extracted from the video clip. Using a trained document identification model, a first score and a second score is generated based on the sequence of image frames and the stream of audio signals, respectively. The trained document identification model is trained with a plurality of historical video clips, where each video clip captures an impact of a document against a surface. An identification score is generated based on the first score and the second score. Whether the object in the video clip is a document is determined based on a comparison between the identification score and an identification threshold.
1. A computer-implemented method for document identification, comprising:
extracting, using an image frame extraction device, a sequence of image frames from a video clip, wherein the video clip captures an impact of an object against a surface;
extracting, using an audio signal extraction device, a stream of audio signals from the video clip;
generating, using a processing device and a trained document identification model, a first score based on the sequence of image frames and a second score based on the stream of audio signals, wherein the trained document identification model is trained with a plurality of historical video clips, and wherein each of the plurality of historical video clips captures an impact of a document against a surface;
generating, using the processing device, an identification score based on the first score and the second score; and
identifying, using the processing device and based on a comparison between the identification score and an identification threshold, if the object in the video clip is a document.
2. The computer-implemented method of claim 1 , wherein generating the identification score comprises:
determining, using the processing device, a weighted sum of the first score and the second score; or
processing the first score and the second score through one or more groups of neural network layers to generate the identification score.
3. The computer-implemented method of claim 1 , further comprising:
extracting, using the image frame extraction device, a sequence of historical image frames from each of the plurality of historical video clips;
extracting, using the audio signal extraction device, a stream of historical audio signals from each of the plurality of historical video clips; and
training, using the processing device and as the trained document identification model, a document identification model with the sequence of historical image frames and the stream of historical audio signals.
4. The computer-implemented method of claim 3 , wherein the document identification model comprises a convolutional neural subnetwork, and wherein training the document identification model comprises training the convolutional neural subnetwork with the sequence of historical image frames from each of the plurality of historical video clips.
5. The computer-implemented method of claim 3 , wherein the document identification model comprises a recurrent neural subnetwork, and wherein training the document identification model comprises training the recurrent neural subnetwork with the stream of historical audio signals from each of the plurality of historical video clips.
6. The computer-implemented method of claim 1 , wherein the trained document identification model comprises a trained convolutional neural subnetwork, and wherein generating the first score comprises using the trained convolutional neural subnetwork to generate the first score based on the sequence of image frames.
7. The computer-implemented method of claim 1 , wherein the trained document identification model comprises a trained recurrent neural subnetwork, and wherein generating the second score comprises using the trained recurrent neural subnetwork to generate the second score based on the stream of audio signals.
8. A non-transitory computer-readable medium storing one or more instructions executable by a computer system to perform operations for document identification, comprising:
extracting, using an image frame extraction device, a sequence of image frames from a video clip, wherein the video clip captures an impact of an object against a surface;
extracting, using an audio signal extraction device, a stream of audio signals from the video clip;
generating, using a processing device and a trained document identification model, a first score based on the sequence of image frames and a second score based on the stream of audio signals, wherein the trained document identification model is trained with a plurality of historical video clips, and wherein each of the plurality of historical video clips captures an impact of a document against a surface;
generating, using the processing device, an identification score based on the first score and the second score; and
identifying, using the processing device and based on a comparison between the identification score and an identification threshold, if the object in the video clip is a document.
9. The non-transitory computer-readable medium of claim 8 , wherein generating the identification score comprises:
determining, using the processing device, a weighted sum of the first score and the second score; or
processing the first score and the second score through one or more groups of neural network layers to generate the identification score.
10. The non-transitory computer-readable medium of claim 8 , further comprising:
extracting, using the image frame extraction device, a sequence of historical image frames from each of the plurality of historical video clips;
extracting, using the audio signal extraction device, a stream of historical audio signals from each of the plurality of historical video clips; and
training, using the processing device and as the trained document identification model, a document identification model with the sequence of historical image frames and the stream of historical audio signals.
11. The non-transitory computer-readable medium of claim 10 , wherein the document identification model comprises a convolutional neural subnetwork, and wherein training the document identification model comprises training the convolutional neural subnetwork with the sequence of historical image frames from each of the plurality of historical video clips.
12. The non-transitory computer-readable medium of claim 10 , wherein the document identification model comprises a recurrent neural subnetwork, and wherein training the document identification model comprises training the recurrent neural subnetwork with the stream of historical audio signals from each of the plurality of historical video clips.
13. The non-transitory computer-readable medium of claim 8 , wherein the trained document identification model comprises a trained convolutional neural subnetwork, and wherein generating the first score comprises using the trained convolutional neural subnetwork to generate the first score based on the sequence of image frames.
14. The non-transitory computer-readable medium of claim 8 , wherein the trained document identification model comprises a trained recurrent neural subnetwork, and wherein generating the second score comprises using the trained recurrent neural subnetwork to generate the second score based on the stream of audio signals.
15. A computer-implemented system for document identification, comprising:
one or more computers; and
one or more computer memory devices interoperably coupled with the one or more computers and having tangible, non-transitory, machine-readable media storing one or more instructions that, when executed by the one or more computers, cause the one or more computers to perform one or more operations comprising:
extracting, using an image frame extraction device, a sequence of image frames from a video clip, wherein the video clip captures an impact of an object against a surface;
extracting, using an audio signal extraction device, a stream of audio signals from the video clip;
generating, using a processing device and a trained document identification model, a first score based on the sequence of image frames and a second score based on the stream of audio signals, wherein the trained document identification model is trained with a plurality of historical video clips, and wherein each of the plurality of historical video clips captures an impact of a document against a surface;
generating, using the processing device, an identification score based on the first score and the second score; and
identifying, using the processing device and based on a comparison between the identification score and an identification threshold, if the object in the video clip is a document.
16. The computer-implemented system of claim 15 , wherein generating the identification score comprises:
determining, using the processing device, a weighted sum of the first score and the second score; or
processing the first score and the second score through one or more groups of neural network layers to generate the identification score.
17. The computer-implemented system of claim 15 , further comprising:
extracting, using the image frame extraction device, a sequence of historical image frames from each of the plurality of historical video clips;
extracting, using the audio signal extraction device, a stream of historical audio signals from each of the plurality of historical video clips; and
training, using the processing device and as the trained document identification model, a document identification model with the sequence of historical image frames and the stream of historical audio signals.
18. The computer-implemented system of claim 17 , wherein:
the document identification model comprises a convolutional neural subnetwork, and wherein training the document identification model comprises training the convolutional neural subnetwork with the sequence of historical image frames from each of the plurality of historical video clips; or
the document identification model comprises a recurrent neural subnetwork, and wherein training the document identification model comprises training the recurrent neural subnetwork with the stream of historical audio signals from each of the plurality of historical video clips.
19. The computer-implemented system of claim 15 , wherein the trained document identification model comprises a trained convolutional neural subnetwork, and wherein generating the first score comprises using the trained convolutional neural subnetwork to generate the first score based on the sequence of image frames.
20. The computer-implemented system of claim 15 , wherein the trained document identification model comprises a trained recurrent neural subnetwork, and wherein generating the second score comprises using the trained recurrent neural subnetwork to generate the second score based on the stream of audio signals.