Performing semantic segmentation of form images using deep learning
The present disclosure relates to generating fillable digital forms corresponding to paper forms using a form conversion neural network to determine low-level and high-level semantic characteristics of the paper forms. For example, one or more embodiments applies a digitized paper form to an encoder that outputs feature maps to a reconstruction decoder, a low-level semantic decoder, and one or more high-level semantic decoders. The reconstruction decoder generates a reconstructed layout of the digitized paper form. The low-level and high-level semantic decoders determine low-level and high-level semantic characteristics of each pixel of the digitized paper form, which provide a probability of the element type to which the pixel belongs. The semantic decoders then classify each pixel and generate corresponding semantic segmentation maps based on those probabilities. The system then generates a fillable digital form using the reconstructed layout and the semantic segmentation maps.
1. In a digital media environment in which paper forms are converted into corresponding fillable digital forms, a method comprising:
using a form conversion neural network trained to determine semantic structures of digitized paper forms by:
generating a feature map by processing a digitized paper form comprising a plurality of pixels using a neural network encoder;
determining, for the plurality of pixels, one or more low-level semantic characteristics indicative of one or more low-level element types of content within the digitized paper form by processing the feature map using a neural network low-level semantic decoder; and
determining, for the plurality of pixels, one or more high-level semantic characteristics indicative of one or more high-level element types of content within the digitized paper form by processing the feature map using one or more neural network high-level semantic decoders that are in parallel with the neural network low-level semantic decoder; and
generating a fillable digital form based on the determined one or more low-level semantic characteristics and the determined one or more high-level semantic characteristics of the digitized paper form.
2. The method of claim 1 , further comprising:
generating a reconstructed layout corresponding to the digitized paper form using a trained form conversion neural network reconstruction decoder,
wherein generating the fillable digital form is further based on the reconstructed layout.
3. The method of claim 1 , wherein:
determining the one or more low-level semantic characteristics comprises determining one or more of a text run, a widget, an image, or an element border,
determining the one or more high-level semantic characteristics comprises determining one or more of a text block, a field, a list, or a table.
4. A non-transitory computer readable storage medium including a set of instructions that, when executed by at least one processor, cause a computing device to:
use a form conversion neural network trained to determine semantic structures of digitized paper forms by:
generating a feature map by processing a digitized paper form comprising a plurality of pixels using a neural network encoder;
determining, for the plurality of pixels, one or more low-level semantic characteristics indicative of one or more low-level element types of content within the digitized paper form by processing the feature map using a neural network low-level semantic decoder; and
determining, for the plurality of pixels, one or more high-level semantic characteristics indicative of one or more high-level element types of content within the digitized paper form by processing the feature map using one or more neural network high-level semantic decoders that are in parallel with the neural network low-level semantic decoder; and
generate a fillable digital form based on the determined one or more low-level semantic characteristics and the determined one or more high-level semantic characteristics.
5. The non-transitory computer readable storage medium of claim 4 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:
generate a low-level semantic segmentation map based on the determined one or more low-level semantic characteristics for the plurality of pixels; and
generate one or more high-level semantic segmentation maps based on the determined one or more high-level semantic characteristics for the plurality of pixels.
6. The non-transitory computer readable storage medium of claim 5 , further comprising instructions that, when executed by the at least one processor, cause the computing device to process the feature map using a neural network reconstruction decoder to generate a reconstructed layout corresponding to the digitized paper form,
wherein, the instructions, when executed by the at least one processor, cause the computing device to generate the fillable digital form further based on the reconstructed layout.
7. The non-transitory computer readable storage medium of claim 6 , wherein the instructions, when executed by the at least one processor, cause the computing device to generate the fillable digital form by combining the low-level semantic segmentation map, the one or more high-level semantic segmentation maps, and the reconstructed layout.
8. The non-transitory computer readable storage medium of claim 4 , wherein determining one or more low-level semantic characteristics for the plurality of pixels by processing the feature map using a neural network low-level semantic decoder comprises:
passing an encoder-layer-level feature map from a layer of the neural network encoder directly to a corresponding layer of the neural network low-level semantic decoder using a skip connection; and
combining the encoder-layer-level feature map with a previous feature map of a previous layer of the neural network low-level semantic decoder.
9. The non-transitory computer readable storage medium of claim 4 , wherein determining one or more high-level semantic characteristics for the plurality of pixels by processing the feature map using the one or more neural network high-level semantic decoders comprises:
passing an encoder-layer-level feature map from a layer of the neural network encoder directly to a corresponding layer of a neural network high-level semantic decoder of the one or more neural network high-level semantic decoders using a skip connection; and
combining the encoder-layer-level feature map with a previous feature map of a previous layer of the neural network high-level semantic decoder.
10. The non-transitory computer readable storage medium of claim 4 , further comprising instructions that, when executed by the at least one processor, cause the computing device to use the form conversion neural network trained to determine semantic structures of digitized paper forms by generating a common trunk feature map by processing the feature map using a form conversion neural network common semantic segmentation trunk.
11. The non-transitory computer readable storage medium of claim 10 , wherein generating a common trunk feature map by processing the feature map using the form conversion neural network common semantic segmentation trunk comprises:
passing an encoder-layer-level feature map from a layer of the neural network encoder directly to a corresponding layer of the form conversion neural network common semantic segmentation trunk using a skip connection; and
combining the encoder-layer-level feature map with a previous feature map of a previous layer of the form conversion neural network common semantic segmentation trunk to generate a combined feature map.
12. The non-transitory computer readable storage medium of claim 11 , wherein combining the encoder-layer-level feature map with the previous feature map comprises concatenating the encoder-layer-level feature map with the previous feature map.
13. The non-transitory computer readable storage medium of claim 11 , wherein:
determining one or more low-level semantic characteristics for the plurality of pixels by processing the feature map using a neural network low-level semantic decoder comprises processing the common trunk feature map using the neural network low-level semantic decoder; and
determining one or more high-level semantic characteristics for the plurality of pixels by processing the feature map using one or more neural network high-level semantic decoders comprises processing the common trunk feature map using the one or more neural network high-level semantic decoders.
14. The non-transitory computer readable storage medium of claim 4 , wherein:
determining, for the plurality of pixels, the one or more low-level semantic characteristics indicative of the one or more low-level element types of content within the digitized paper form comprises determining low-level semantic characteristics indicative of at least one of a text run, a widget, an image, or a border; and
determining, for the plurality of pixels, the one or more high-level semantic characteristics indicative of the one or more high-level element types of content within the digitized paper form comprises determining high-level semantic characteristics indicative of at least one of a text block, a field, a list, or a table.
15. A system for converting paper forms into corresponding fillable digital forms, comprising:
a memory comprising:
analytics training data associated with a plurality of pixels of training digitized paper forms; and
a neural network encoder that outputs feature maps to a form conversion neural network reconstruction decoder, a neural network low-level semantic decoder, and one or more neural network high-level semantic decoders that are in parallel with the neural network low-level semantic decoder;
at least one server; and
at least one non-transitory computer readable storage medium storing instructions thereon that, when executed by the at least one server, cause the system to:
generate a feature map by processing a digitized paper form comprising a plurality of pixels using the neural network encoder;
determine, for the plurality of pixels, one or more low-level semantic characteristics indicative of one or more low-level element types of content within the digitized paper form by processing the feature map using the neural network low-level semantic decoder; and
determine, for the plurality of pixels, one or more high-level semantic characteristics indicative of one or more high-level element types of content within the digitized paper form by processing the feature map using the one or more neural network high-level semantic decoders.
16. The system of claim 15 , wherein the instructions, when executed by the at least one server, cause the system to:
generate the feature map by processing the digitized paper form comprising the plurality of pixels using the neural network encoder by generating a training feature map by processing the digitized paper form using the neural network encoder;
determine one or more low-level semantic characteristics for the plurality of pixels by processing the feature map using the neural network low-level semantic decoder by processing the training feature map using the neural network low-level semantic decoder and comparing a low-level output to a low-level ground truth to determine a low-level loss;
determine one or more high-level semantic characteristics for the plurality of pixels by processing the feature map using the one or more neural network high-level semantic decoders by processing the training feature map using the one or more neural network high-level semantic decoders and comparing one or more high-level outputs to a corresponding high-level ground truth to determine one or more high-level losses;
generate a combined loss comprising the low-level loss and the one or more high-level losses; and
back propagate the combined loss to modify parameters of the neural network low-level semantic decoder, the one or more neural network high-level semantic decoders, and the neural network encoder.
17. The system of claim 16 , wherein the low-level output comprises a low-level semantic segmentation map and the one or more high-level outputs comprise one or more high-level semantic segmentation maps.
18. The system of claim 16 , further comprising instructions that, when executed by the at least one server, cause the system to process the training feature map using the form conversion neural network reconstruction decoder and comparing a reconstruction output to a reconstruction ground truth to determine a reconstruction loss, wherein the combined loss further comprises the reconstruction loss and the instructions, when executed by the at least one server, back propagate the combined loss to further modify parameters of the form conversion neural network reconstruction decoder.
19. The system of claim 15 , wherein the instructions, when executed by the at least one server, cause the system to:
determine one or more low-level semantic characteristics for the plurality of pixels by processing the feature map using a trained neural network low-level semantic decoder; and
determine one or more high-level semantic characteristics for the plurality of pixels by processing the feature map using one or more trained neural network high-level semantic decoders.
20. The system of claim 15 , further comprising instructions that, when executed by the at least one server, cause the system to generate a tillable digital form based on the determined one or more low-level semantic characteristics and the determined one or more high-level semantic characteristics.