Computer vision image feature identification via multi-label few-shot model
A technique making use of a few-shot model to determine graphical features present in an image based on a small set of examples with known graphical features. Where a support set including a number of images that each have a known combination of graphical features, the image recognition can identify unknown combinations of those graphical features in any number of query images. In an embodiment of the present disclosure examples of a filled-out form are used to interpret any number of additional filled out versions of the form.
1. A multi-model image content recognition system comprising:
a first model configured to receive an input set including a query image and a support set of images, wherein first model is trained to output a descriptive vector for each image of the input set including a descriptive query vector and descriptive support vectors;
a second model configured to receive each descriptive support vector concatenated with the descriptive query vector, wherein the second model is trained to output a comparison vector for each input, the comparison vector indicating a degree of similarity between the query image and a respective member of the support set of images; and
a third model trained to output a solution vector indicating graphic features present in the query image based on the comparison vectors.
2. The system of claim 1 , wherein the third model further bases the solution vector on a matrix of support set graphic features.
3. The system of claim 1 , wherein the first model, the second model, and the third model are neural network models.
4. The system of claim 3 , wherein the first model, the second model, and the third model are few-shot models.
5. The system of claim 1 , wherein each image of the support set of images and the query image are each a representation of a filled-out form document, the filled-out form documents adhering to a same form template, and the graphic features are check boxes positioned at predetermined locations on the form template.
6. The system of claim 1 , wherein the support set of images includes at least one instance of each of a set of graphic features and the query image includes an unknown combination of the graphic features that is not present within the support set of images.
7. A method comprising:
receiving, by a first model, an input set including a query image and a support set of images;
outputting, by the first model, a descriptive vector for each image of the input set including a descriptive query vector and descriptive support vectors;
receiving, by a second model, each descriptive support vector concatenated with the descriptive query vector;
outputting, by the second model, a comparison vector for each descriptive support vector concatenated with the descriptive query vector, the comparison vector indicating a degree of similarity between the query image and a respective member of the support set of images; and
projecting, by a third model, a solution vector indicating graphic features present in the query image based on the comparison vectors.
8. The method of claim 7 , further comprising:
computing an unprojected solution based on the comparison vectors and an inverted matrix indicating graphic features of the support set of images, wherein the unprojected solution is used by the third model to project the solution vector.
9. The method of claim 7 , further comprising:
training the first model on a plurality of images including metadata indicating present graphic features.
10. The method of claim 7 , wherein the third model further bases the solution vector on a matrix of support set graphic features.
11. The method of claim 7 , wherein the first model, the second model, and the third model are neural network models.
12. The method of claim 11 , wherein the first model, the second model, and the third model are few-shot models.
13. The method of claim 7 , wherein each image of the support set of images and the query image are each a representation of a filled-out form document, the filled-out form documents adhering to a same form template, and the graphic features are check boxes positioned at predetermined locations on the form template.
14. The method of claim 7 , wherein the support set of images includes at least one instance of each of a set of graphic features and the query image includes an unknown combination of the graphic features that is not present within the support set of images.
15. A method comprising:
receiving, by a series of neural network models, an input set including a query image and a support set of images, wherein the support set of images includes at least one instance of each of a set of graphic features and the query image includes an unknown combination of the graphic features that is not present within the support set of images;
generating, by the series of neural network models, a set of partial products that progressively develop a list of graphic features included in the unknown combination of graphic features of the query image, wherein the series of neural network models vectorizes the input set and then compares and projects the partial products; and
outputting, by the series of neural network models, the list of graphic features included in the unknown combination of graphic features of the query image.
16. The method of claim 15 , wherein the series of neural network models are few-shot models.
17. The method of claim 15 , further comprising:
training the series of neural network models on a plurality of images including metadata indicating present graphic features.
18. The method of claim 15 , wherein the input set further includes a matrix indicating the instances of each of graphic feature present in each respective image of the support set of images.
19. The method of claim 18 , wherein the partial products further include an inverted version of the matrix.
20. The method of claim 15 , further comprising:
executing computations including partial products external to and interspersed between the series of neural network models.