Systems and methods for automated license plate recognition
Disclosed herein are methods, systems, and apparatus for automated license plate recognition and methods for training a machine learning model to undertake automated license plate recognition. For example, a method can comprise dividing an image or video frame comprising a license plate into a plurality of image patches, determining a positional vector for each of the image patches, adding the positional vector to each of the image patches and inputting the image patches and their associated positional vectors to a text-adapted vision transformer. The text-adapted vision transformer can be configured to output a prediction concerning the license plate number of the license plate.
1. One or more non-transitory computer-readable media comprising instructions stored thereon, that when executed by one or more processors, cause the one or more processors to perform operations comprising:
dividing an image or video frame comprising a license plate in the image or video frame into a plurality of image patches, wherein the image or video frame is divided horizontally and vertically to obtain the image patches, and wherein at least one of the image patches comprises a portion of a character of a license plate number of the license plate;
determining a positional vector for each of the image patches, wherein the positional vector represents a spatial position of each of the image patches in the image or video frame;
adding the positional vector to each of the image patches and inputting the image patches and their associated positional vectors to a transformer encoder of a text-adapted vision transformer run on one or more devices; and
obtaining a prediction, outputted by the text-adapted vision transformer, concerning the license plate number of the license plate.
2. The one or more non-transitory computer-readable media of claim 1 , wherein the image or video frame is divided horizontally and vertically into a N×N grid of image patches, wherein N is an integer between 4 and 256.
3. The one or more non-transitory computer-readable media of claim 2 , wherein the text-adapted vision transformer comprises a linear projection layer, wherein the linear projection layer is configured to flatten each image patch of the N×N grid of image patches to 1×((H/N)×(W/N)), wherein a resulting input to the transformer encoder is then M×1×((H/N)×(W/N)), wherein H is a height of the image or video frame in pixels, wherein W is a width of the image or video frame in pixels, and wherein M is equaled to N multiplied by N.
4. The one or more non-transitory computer-readable media of claim 3 , further comprising adding a two-dimensional vector representing a spatial position of each of the image patches to the image patches.
5. The one or more non-transitory computer-readable media of claim 1 , wherein the text-adapted vision transformer is run on an edge device, wherein the edge device is coupled to a carrier vehicle, and wherein the image or video frame is captured using one or more cameras of the edge device while the carrier vehicle is in motion.
6. The one or more non-transitory computer-readable media of claim 1 , wherein the text-adapted vision transformer is run on a server, wherein the image or video frame is captured using one or more cameras of an edge device communicatively coupled to the server, wherein the image or video frame is transmitted by the edge device to the server, wherein the edge device is coupled to a carrier vehicle, and wherein the image or video frame is captured by the one or more cameras of the edge device while the carrier vehicle is in motion.
7. The one or more non-transitory computer-readable media of claim 1 , wherein each character of the license plate number is separately predicted by the transformer encoder of the text-adapted vision transformer.
8. The one or more non-transitory computer-readable media of claim 1 , wherein the text-adapted vision transformer is trained on a plate dataset comprising a plurality of real plate-text pairs, wherein the real plate-text pairs comprise images or video frames of real-life license plates and an annotated license plate number associated with each of the real-life license plates.
9. The one or more non-transitory computer-readable media of claim 8 , wherein the real-life license plates in the plate dataset comprise license plates with differing U.S. state plate aesthetics or configurations, license plates with differing non-U.S. country or region plate aesthetics or configurations, license plates with differing plate character configurations or styles, license plates with differing levels of blur associated with the images or video frames, and license plate with differing levels of exposure associated with the images or video frames.
10. The one or more non-transitory computer-readable media of claim 8 , wherein the text-adapted vision transformer is pre-trained on a plurality of image-text pairs prior to being trained on the plate dataset, wherein the image-text pairs comprise images or video frames of real-life objects comprising text and an annotation of the text.
11. The one or more non-transitory computer-readable media of claim 8 , wherein the text-adapted vision transformer is further trained on artificially-generated plate-text pairs, wherein the artificially-generated plate-text pairs comprise images of non-real license plates artificially generated by a latent diffusion model and a non-real license plate number associated with each of the non-real license plates.
12. The one or more non-transitory computer-readable media of claim 11 , wherein the latent diffusion model is trained using the plate dataset used to train the text-adapted vision transformer.
13. The one or more non-transitory computer-readable media of claim 11 , wherein the non-real license plate number is generated by a random plate number generator, and wherein at least one of the images of the non-real license plates is generated based on the non-real license plate number and one or more plate features provided as inputs to the latent diffusion model.
14. The one or more non-transitory computer-readable media of claim 13 , wherein the one or more plate features comprise at least one of a U.S. state plate aesthetic or configuration, a non-U.S. country or region plate aesthetic or configuration, a plate configuration or style, a level of noise associated with a license plate image, a level of blur associated with the license plate image, and a level of exposure associated with the license plate image.
15. One or more non-transitory computer-readable media comprising instructions stored thereon, that when executed by one or more processors, cause the one or more processors to perform operations comprising:
pre-training a text-adapted vision transformer on a plurality of image-text pairs, wherein the image-text pairs comprise images or video frames of real-life objects comprising text and an annotation of the text;
training the text-adapted vision transformer on a plate dataset comprising a plurality of real plate-text pairs, wherein the real plate-text pairs comprise images or video frames of real-life license plates and an annotated license plate number associated with each of the real-life license plates; and
further training the text-adapted vision transformer on artificially-generated plate-text pairs, wherein the artificially-generated plate-text pairs comprise images of non-real license plates artificially generated by a latent diffusion model and a non-real license plate number associated with each of the non-real license plates.
16. The one or more non-transitory computer-readable media of claim 15 , wherein the non-real license plate number is generated by a random plate number generator, and wherein at least one of the images of the non-real license plates is generated based on the non-real license plate number and one or more plate features provided as inputs to the latent diffusion model.
17. The one or more non-transitory computer-readable media of claim 16 , wherein the one or more plate features comprise at least one of a U.S. state plate aesthetic or configuration, a non-U.S. country or region plate aesthetic or configuration, a plate character configuration or style, a level of blur associated with the artificially-generated image, and a level of exposure associated with the artificially-generated image.
18. The one or more non-transitory computer-readable media of claim 16 , wherein the one or more plate features are selected or changed based on an accuracy of predictions made by the text-adapted vision transformer.
19. The one or more non-transitory computer-readable media of claim 18 , wherein the instructions further cause the one or more processors to perform operations comprising providing a prompt to the latent diffusion model to generate additional images of non-real license plates based in part on common plate features resulting in low accuracy predictions made by the text-adapted vision transformer.
20. The one or more non-transitory computer-readable media of claim 15 , wherein the latent diffusion model is trained using the plate dataset used to train the text-adapted vision transformer.