Pre-training for scene text detection
Embodiments of the present disclosure relate to a method, device and computer readable storage medium of scene text detection. In the method, a first visual representation of a first image is generated with an image encoding process. A first textual representation of a first text unit in the first image is generated with a text encoding process based on a first plurality of symbols obtained by masking a first symbol of a plurality of symbols in the first text unit. A first prediction of the masked first symbol is determined with a decoding process based on the first visual and textual representations. At least the image encoding process is updating according to at least a first training objective to increase at least similarity of the first prediction and the masked first symbol.
1. A method of scene text detection, comprising:
generating, with an image encoding process, a first visual representation of a first image;
generating, with a text encoding process, based on a first plurality of symbols, a first textual representation of a first text unit in the first image, the first plurality of symbols obtained by masking a first symbol of a plurality of symbols in the first text unit;
determining, with a decoding process, a first prediction of the masked first symbol based on the first visual and textual representations; and
updating at least the image encoding process according to at least a first training objective to increase at least similarity of the first prediction and the masked first symbol.
2. The method of claim 1 , wherein generating the first textual representation comprises:
extracting a plurality of symbol representations from the first plurality of symbols; and
generating the first textual representation by aggregating the plurality of symbol representations.
3. The method of claim 1 , wherein the first text unit comprises a part of a text in the first image.
4. The method of claim 1 , further comprising:
generating, with the text encoding process, based on a second plurality of symbols, a second textual representation of a second text unit in the first image, the second plurality of symbols obtained by masking a second symbol of a plurality of symbols in the second text unit; and
determining, with the decoding process, a second prediction of the masked second symbol based on the first visual representation and the second textual representations,
where the first training objective is further to increase similarity of the second prediction and the masked second symbol.
5. The method of claim 1 , wherein updating the image encoding process comprises:
updating the image encoding process according to further a second training objective to increase at least similarity of the first visual and textual representations.
6. The method of claim 5 , further comprising:
generating, with the image encoding process, a second visual representation of a second image; and
generating, with the text encoding process, based on a third plurality of symbols, a third textual representation of a third text unit in the second image, the third plurality of symbols obtained by masking a third symbol of a plurality of symbols in the third text unit,
wherein the second training objective is further to increase similarity of the second visual representation and the third textual representation.
7. The method of claim 6 , wherein the second training objective is further to decrease similarity of the second visual representation and the first textual representation and similarity of the first visual representation and the third textual representation.
8. The method of claim 1 , wherein updating at least the image encoding process comprises:
updating the image encoding process along with the text encoding process.
9. An electronic device, comprising:
at least one processor; and
at least one memory communicatively coupled to the at least one processor and comprising computer-readable instructions that upon execution by the at least one processor cause the at least one processor to perform acts comprising:
generating, with an image encoding process, a first visual representation of a first image;
generating, with a text encoding process, based on a first plurality of symbols, a first textual representation of a first text unit in the first image, the first plurality of symbols obtained by masking a first symbol of a plurality of symbols in the first text unit;
determining, with a decoding process, a first prediction of the masked first symbol based on the first visual and textual representations; and
updating at least the image encoding process according to at least a first training objective to increase at least similarity of the first prediction and the masked first symbol.
10. The electronic device of claim 9 , wherein generating the first textual representation comprises:
extracting a plurality of symbol representations from the first plurality of symbols; and
generating the first textual representation by aggregating the plurality of symbol representations.
11. The electronic device of claim 9 , wherein the first text unit comprises a part of a text in the first image.
12. The electronic device of claim 9 , wherein the acts further comprise:
generating, with the text encoding process, based on a second plurality of symbols, a second textual representation of a second text unit in the first image, the second plurality of symbols obtained by masking a second symbol of a plurality of symbols in the second text unit; and
determining, with the decoding process, a second prediction of the masked second symbol based on the first visual representation and the second textual representations,
where the first training objective is further to increase similarity of the second prediction and the masked second symbol.
13. The electronic device of claim 9 , wherein updating the image encoding process comprises:
updating the image encoding process according to further a second training objective to increase at least similarity of the first visual and textual representations.
14. The electronic device of claim 13 , wherein the acts further comprise:
generating, with the image encoding process, a second visual representation of a second image; and
generating, with the text encoding process, based on a third plurality of symbols, a third textual representation of a third text unit in the second image, the third plurality of symbols obtained by masking a third symbol of a plurality of symbols in the third text unit,
wherein the second training objective is further to increase similarity of the second visual representation and the third textual representation.
15. The electronic device of claim 14 , wherein the second training objective is further to decrease similarity of the second visual representation and the first textual representation and similarity of the first visual representation and the third textual representation.
16. The electronic device of claim 9 , wherein updating at least the image encoding process comprises:
updating the image encoding process along with the text encoding process.
17. A non-transitory computer-readable storage medium, storing computer-readable instructions that upon execution by a computing device cause the computing device to perform acts comprising:
generating, with an image encoding process, a first visual representation of a first image;
generating, with a text encoding process, based on a first plurality of symbols, a first textual representation of a first text unit in the first image, the first plurality of symbols obtained by masking a first symbol of a plurality of symbols in the first text unit;
determining, with a decoding process, a first prediction of the masked first symbol based on the first visual and textual representations; and
updating at least the image encoding process according to at least a first training objective to increase at least similarity of the first prediction and the masked first symbol.
18. The computer-readable storage medium of claim 17 , wherein generating the first textual representation comprises:
extracting a plurality of symbol representations from the first plurality of symbols; and
generating the first textual representation by aggregating the plurality of symbol representations.
19. The computer-readable storage medium of claim 17 , wherein updating the image encoding process comprises:
updating the image encoding process according to further a second training objective to increase at least similarity of the first visual and textual representations.
20. The computer-readable storage medium of claim 19 , wherein the acts further comprise:
generating, with the image encoding process, a second visual representation of a second image; and
generating, with the text encoding process, based on a third plurality of symbols, a third textual representation of a third text unit in the second image, the third plurality of symbols obtained by masking a third symbol of a plurality of symbols in the third text unit,
wherein the second training objective is further to increase similarity of the second visual representation and the third textual representation and to decrease similarity of the second visual representation and the first textual representation and similarity of the first visual representation and the third textual representation.