Systems and methods for image based content capture and extraction utilizing deep learning neural network and bounding box detection training techniques
Systems, methods, and computer program products for image recognition in which instructions are executable by a processor to dynamically generate simulated documents and corresponding images, which are then used to train a fully convolutional neural network. A plurality of document components are provided, and the processor selects subsets of the document components. The document components in each subset are used to dynamically generate a corresponding simulated document and a simulated document image. The convolutional neural network processes the simulated document image to produce a recognition output. Information corresponding to the document components from which the image was generated is used as an expected output. The recognition output and expected output are compared, and weights of the convolutional neural network are adjusted based on the differences between them.
1. An image recognition system, comprising: a processor; and a non-transitory computer-readable medium; and stored instructions translatable by the processor to perform: receiving or obtaining a plurality of document components representing one or more feature types, wherein each of the plurality of document components corresponds to a piece of information; selecting one or more subsets of the plurality of document components; for each subset of the plurality of document components thus selected, performing: dynamically generating, from the subset of document components, one or more simulated document images of a simulated document; providing the one or more simulated document images and pieces of information corresponding to the document components of the subset as input to a convolutional neural network; and training the convolutional neural network to recognize the feature types using the pieces of information corresponding to the document components of the subset as expected outputs of processing the one or more simulated document images dynamically generated from the simulated document which is dynamically generated from the subset of document components.
2. The image recognition system of claim 1 , wherein the one or more simulated document images incorporate one or more image variations, wherein the processor selects the one or more image variations from the group consisting of tilting, skewing, lighting variation, cropping, or decentering.
3. The image recognition system of claim 1 , wherein the instructions are further translatable by the processor to perform:
receiving an input image from an input/output device communicatively connected to the image recognition system over a network;
performing, using the trained convolutional neural network, a set of convolutional operations on the input image so as to produce one or more heat maps, wherein each of the heat maps indicates a probability of finding a corresponding one of the feature types at each pixel in the heat map; and
extracting, from the input image, one of the feature types from an area of the input image indicated by the corresponding one of the heat maps.
4. The image recognition system of claim 3 , wherein the convolutional neural network produces heat maps corresponding to feature types including: text; face; signature; document background; and image background.
5. The image recognition system of claim 3 , wherein the processor comprises one or more computers that form a public cloud network.
6. The image recognition system of claim 3 , wherein the input/output device comprises a mobile client device.
7. The image recognition system of claim 3 , wherein the processor performs preprocessing on each input image prior to providing the preprocessed image to the convolutional neural network, wherein the preprocessing includes at least conversion of colors to grayscale.
8. The image recognition system of claim 1 , wherein the instructions are further translatable by the processor to perform:
receiving an input image from an input/output device communicatively connected to the image recognition system over a network;
performing, using the trained convolutional neural network, a set of convolutional operations on the input image so as to produce one or more bounding boxes, wherein each of the one or more bounding boxes indicates an area of the input image in which a corresponding feature type is located; and
extracting, from the input image, each feature type from the area of the input image indicated by the corresponding bounding box.
9. A method, comprising: receiving or obtaining, by an image recognition system executing on a processor, a plurality of document components representing one or more feature types, wherein each of the plurality of document components corresponds to a piece of information; selecting, by the image recognition system, one or more subsets of the plurality of document components; for each of the subsets of the plurality of document components thus selected, performing, by the image recognition system: dynamically generating, from the subset of document components, one or more simulated images of a simulated document; providing the one or more simulated document images and pieces of information corresponding to the document components of the subset as input to a convolutional neural network; and training the convolutional neural network to recognize the feature types using the pieces of information corresponding to the document components of the subset as expected outputs of processing the one or more simulated document images dynamically generated from the simulated document which is dynamically generated from the subset of document components.
10. The method of claim 9 , wherein dynamically generating the simulated document image from the simulated document for each of the subsets of the plurality of document components further comprises implementing one or more image variations in the simulated document image, wherein the one or more image variations are selecting from the group consisting of tilting, skewing, lighting variation, cropping, or decentering.
11. The method of claim 9 , wherein training the convolutional neural network comprises:
performing image recognition processing on the one or more simulated document images and thereby producing a corresponding set of recognition outputs;
comparing the set of recognition outputs to the expected outputs for the one or more simulated document images; and
adjusting a set of weights of the convolutional neural network in dependence upon differences between the set of recognition outputs and the expected outputs for the one or more simulated document images.
12. The method of claim 9 , further comprising:
providing one or more captured images to the convolutional neural network; and
performing, by the convolutional neural network, image recognition processing on the captured images, wherein the image recognition processing includes:
performing a set of convolutional operations on the captured images and thereby determining or identifying, for each of the captured images, one or more corresponding heat maps, wherein each of the heat maps indicates a probability of finding the corresponding feature type at each pixel in the heat map; and
extracting information from the input images, wherein for each of the input images, information corresponding to a specific feature type is extracted from an area of the input image indicated by the heat map corresponding to the specific feature type.
13. The method of claim 12 , wherein the convolutional neural network produces heat maps corresponding to feature types including text, face, signature, document background, and image background.
14. The method of claim 12 , further comprising:
receiving the one or more captured images from an input device.
15. The method of claim 14 , wherein the input device comprises a mobile client device.
16. The method of claim 12 , further comprising:
performing preprocessing on each captured image prior to providing the captured image to the convolutional neural network, wherein the preprocessing includes at least conversion of colors to grayscale.
17. The method of claim 9 , further comprising:
providing one or more captured images to the convolutional neural network; and
performing, by the convolutional neural network, image recognition processing on the captured images, wherein the image recognition processing includes:
performing a set of convolutional operations on the captured images and thereby determining or identifying, for each of the captured images, one or more corresponding bounding boxes, wherein each of the bounding boxes indicates an area of the captured image in which a corresponding feature type is located; and
extracting information from the input images, wherein for each of the input images, information of a specific feature type is extracted from the area of the input image indicated by the bounding box corresponding to the specific feature type.
18. A computer program product comprising a non-transitory computer-readable medium storing instructions translatable by one or more processors to perform:
receiving or obtaining a plurality of document components representing one or more feature types, wherein each of the plurality of document components corresponds to a piece of information;
selecting one or more subsets of the plurality of document components;
for each of the subsets of the plurality of document components thus selected, performing:
dynamically generating, from the subset of document components, one or more images of a simulated document;
providing the one or more simulated document images and pieces of information corresponding to the document components of the subset as input to a convolutional neural network; and
training the convolutional neural network to recognize the feature types using the pieces of information corresponding to the document components of the subset as expected outputs of processing the one or more simulated document images dynamically generated from the simulated document which is dynamically generated from the subset of document components.
19. The computer program product of claim 18 , wherein the instructions are further translatable by the one or more processors to perform:
performing image recognition processing on the one or more simulated document images and thereby producing a corresponding set of recognition outputs;
comparing the set of recognition outputs to the expected outputs for the one or more simulated document images; and
adjusting a set of weights of the convolutional neural network in dependence upon differences between the set of recognition outputs and the expected outputs for the one or more simulated document images.
20. The computer program product of claim 19 , wherein the instructions are further translatable by the one or more processors to perform:
providing one or more captured images to the convolutional neural network; and
performing, by the convolutional neural network, image recognition processing on the captured images, wherein the image recognition processing includes:
performing a set of convolutional operations on the captured images and thereby determining or identifying, for each of the captured images, one or more corresponding heat maps, wherein each of the heat maps indicates a probability of finding the corresponding feature type at each pixel in the heat map; and
extracting information from the input images, wherein for each of the input images, information corresponding to a specific feature type is extracted from an area of the input image indicated by the heat map corresponding to the specific feature type.