Granted Patent B2

US 12,469,318 · App. 17/932,639 · Granted Nov 11, 2025

Inventors: Zhong Fang Yuan (Xi'an, CN); Tong Liu (Xi'an, CN); Yi Chen Zhong (Shanghai, CN); Xiang Yu Yang (Xi'an, CN); Guan Chao Li (Shanghai, CN)

Assignee: International Business Machines Corporation

G06V30/153G06V10/7747G06V10/82G06V30/1823

Abstract

Claims (81)

1 . A computer program product for performing optical character recognition processing of an image, the computer program product comprising a computer readable storage medium having computer readable program code embodied therein that is executable to perform operations, the operations comprising:

providing a vector encoder trained to encode images, comprising digital images representing text, into vectors in a vector space, wherein vectors of images representing similar text have a high degree of cohesion in the vector space, and wherein vectors of images representing dissimilar text have a low degree of cohesion in the vector space;

processing an input image to determine sub-images, of the input image, wherein the sub-images bound the text represented in the input image;

inputting the sub-images to the vector encoder to output sub-image vectors, wherein the sub-image vectors represent the sub-images in the vector space;

using the vector encoder to generate a search vector for search text, wherein the search vector represents the search text in the vector space;

determining the sub-image vectors that match the search vector; and

applying optical character recognition to at least one region of the input image including the sub-images having the sub-image vectors matching the search vector based on closeness of the sub-image vectors and the search vector in the vector space.

2 . The computer program product of claim 1 , wherein the operations further comprise:

converting the search text to a search image, wherein using the vector encoder to generate the search vector comprises inputting the search image to the vector encoder to output the search vector.

3 . The computer program product of claim 1 , wherein the vector encoder is trained to encode images by:

inputting a positive set of images representing similar text to a base text into an embedding model to generate positive vectors for the positive set of images representing similar text to the base text, wherein the positive vectors are sufficiently close in the vector space to indicate similarity of the text represented by the positive set of images to the base text;

inputting a negative set of images representing dissimilar text to the base text into the embedding model to generate negative vectors for the negative set of images representing dissimilar text to the base text, wherein the negative vectors are sufficiently separated in the vector space to indicate dissimilarity of the text represented by the negative set of images;

training the vector encoder to output the positive vectors for the positive set of images; and

training the vector encoder to output the negative vectors for the negative set of images.

4 . The computer program product of claim 3 ,

wherein the positive set of images is generated by:

for the base text, generating a plurality of similar text formed by slight variations in characters, font and style of the base text; and

converting the base text and the similar text into the positive set of images, wherein the negative set of images is generated by:

for the base text, determining dissimilar text to the base text; and

converting the dissimilar text into the negative set of images.

5 . The computer program product of claim 3 , wherein the positive vectors produced by the embedding model for the positive set of images comprise intermediate vectors, wherein the operations further comprise:

performing fusion embedding of the intermediate vectors to generate the positive vectors for the positive set of images.

6 . The computer program product of claim 3 , wherein the vector encoder comprises a double tower network comprising a positive encoder having a positive embedding layer and a negative encoder having a negative embedding layer,

wherein the training the vector encoder to output the positive vectors comprises training the positive embedding layer to output the positive vectors from the positive set of images, and

wherein the training the vector encoder to output the negative vectors comprises training the negative embedding layer to output the negative vectors from the negative set of images.

7 . The computer program product of claim 1 , wherein the vector encoder comprises a double tower network comprising a first encoder having a first embedding layer and a second encoder having a second embedding layer, wherein the vector encoder outputs a vector for a sub-image of the sub-images by:

outputting, by the first embedding layer, a first vector for the sub-image;

outputting, by the second embedding layer, a second vector for the sub-image; and

combining the first vector and the second vector to obtain the vector for the sub-image.

8 . The computer program product of claim 1 , wherein the vector encoder includes an embedding layer to output an intermediate vector from an input image and includes a weight matrix to apply hidden states to the intermediate vector to generate a vector for the input image.

9 . The computer program product of claim 8 , wherein the weight matrix implements a transfer encoder to adjust the intermediate vector.

10 . A system for performing optical character recognition processing of an image, comprising:

at least one processor; and

a computer readable storage medium having computer readable program code embodied therein that is executable by the at least one processor to perform operations, the operations comprising:

providing a vector encoder trained to encode images, comprising digital images representing text, into vectors in a vector space, wherein vectors of images representing similar text have a high degree of cohesion in the vector space, and wherein vectors of images representing dissimilar text have a low degree of cohesion in the vector space;

processing an input image to determine sub-images, of the input image, wherein the sub-images bound the text represented in the input image;

inputting the sub-images to the vector encoder to output sub-image vectors, wherein the sub-image vectors represent the sub-images in the vector space;

using the vector encoder to generate a search vector for search text, wherein the search vector represents the search text in the vector space;

determining the sub-image vectors that match the search vector; and

applying optical character recognition to at least one region of the input image including the sub-images having the sub-image vectors matching the search vector based on closeness of the sub-image vectors and the search vector in the vector space.

11 . The system of claim 10 , wherein the operations further comprise:

converting the search text to a search image, wherein using the vector encoder to generate the search vector comprises inputting the search image to the vector encoder to output the search vector.

12 . The system of claim 10 , wherein the vector encoder is trained to encode images by:

inputting a positive set of images representing similar text to a base text into an embedding model to generate positive vectors for the positive set of images representing similar text to the base text, wherein the positive vectors are sufficiently close in the vector space to indicate similarity of the text represented by the positive set of images to the base text;

inputting a negative set of images representing dissimilar text to the base text into the embedding model to generate negative vectors for the negative set of images representing dissimilar text to the base text, wherein the negative vectors are sufficiently separated in the vector space to indicate dissimilarity of the text represented by the negative set of images;

training the vector encoder to output the positive vectors for the positive set of images; and

training the vector encoder to output the negative vectors for the negative set of images.

13 . The system of claim 12 ,

wherein the positive set of images is generated by:

for the base text, generating a plurality of similar text formed by slight variations in characters, font and style of the base text; and

converting the base text and the similar text into the positive set of images, wherein the negative set of images is generated by:

for the base text, determining dissimilar text to the base text; and

converting the dissimilar text into the negative set of images.

14 . The system of claim 12 , wherein the vector encoder comprises a double tower network comprising a positive encoder having a positive embedding layer and a negative encoder having a negative embedding layer,

wherein the training the vector encoder to output the positive vectors comprises training the positive embedding layer to output the positive vectors from the positive set of images, and

wherein the training the vector encoder to output the negative vectors comprises training the negative embedding layer to output the negative vectors from the negative set of images.

15 . The system of claim 12 , wherein the vector encoder comprises a double tower network comprising a first encoder having a first embedding layer and a second encoder having a second embedding layer, wherein the vector encoder outputs a vector for a sub-image of the sub-images by:

outputting, by the first embedding layer, a first vector for the sub-image;

outputting, by the second embedding layer, a second vector for the sub-image; and

combining the first vector and the second vector to obtain the vector for the sub-image.

16 . A computer implemented method for performing optical character recognition processing of an image, comprising:

providing a vector encoder trained to encode images, comprising digital images representing text, into vectors in a vector space, wherein vectors of images representing similar text have a high degree of cohesion in the vector space, and wherein vectors of images representing dissimilar text have a low degree of cohesion in the vector space;

processing an input image to determine sub-images, of the input image, wherein the sub-images bound the text represented in the input image;

inputting the sub-images to the vector encoder to output sub-image vectors, wherein the sub-image vectors represent the sub-images in the vector space;

using the vector encoder to generate a search vector for search text, wherein the search vector represents the search text in the vector space;

determining the sub-image vectors that match the search vector; and

applying optical character recognition to at least one region of the input image including the sub-images having the sub-image vectors matching the search vector based on closeness of the sub-image vectors and the search vector in the vector space.

17 . The computer implemented method of claim 16 , further comprising:

converting the search text to a search image, wherein using the vector encoder to generate the search vector comprises inputting the search image to the vector encoder to output the search vector.

18 . The computer implemented method of claim 16 , wherein the vector encoder is trained to encode images by:

inputting a positive set of images representing similar text to a base text into an embedding model to generate positive vectors for the positive set of images representing similar text to the base text, wherein the positive vectors are sufficiently close in the vector space to indicate similarity of the text represented by the positive set of images to the base text;

inputting a negative set of images representing dissimilar text to the base text into the embedding model to generate negative vectors for the negative set of images representing dissimilar text to the base text, wherein the negative vectors are sufficiently separated in the vector space to indicate dissimilarity of the text represented by the negative set of images;

training the vector encoder to output the positive vectors for the positive set of images; and

training the vector encoder to output the negative vectors for the negative set of images.

19 . The computer implemented method of claim 18 , wherein the vector encoder comprises a double tower network comprising a positive encoder having a positive embedding layer and a negative encoder having a negative embedding layer,

wherein the training the vector encoder to output the positive vectors comprises training the positive embedding layer to output the positive vectors from the positive set of images, and

wherein the training the vector encoder to output the negative vectors comprises training the negative embedding layer to output the negative vectors from the negative set of images.

20 . The computer implemented method of claim 16 , wherein the vector encoder comprises a double tower network comprising a first encoder having a first embedding layer and a second encoder having a second embedding layer, wherein the vector encoder outputs a vector for a sub-image of the sub-images by:

outputting, by the first embedding layer, a first vector for the sub-image;

outputting, by the second embedding layer, a second vector for the sub-image; and

combining the first vector and the second vector to obtain the vector for the sub-image.

Assignments (1)

Continuity (1)

References Cited (15)

Cited By (1)

US 12,737,590