License plate recognition method, device thereof, and user equipment

1. A license plate recognition method comprising following steps:

acquiring a vehicle image and inputting the vehicle image to a vehicle searching convolutional neural network (CNN) to search whether there is a vehicle corresponding to the vehicle image:

acquiring a license plate image on the vehicle image, and inputting the license plate image to a license plate searching CNN to search whether there is a license plate corresponding to the license plate image, if there is the vehicle image of the vehicle;

inputting the license plate image to a tensor neural network to detect the license plate image is abnormal or not, if there is a license plate corresponding to the license plate image;

acquiring a recognition image of the license plate image, and inputting the recognition image to a determining neural network to judge an area number, letters, and numbers in the recognition image, and outputting an area number image corresponding to the area number, a letter image corresponding to the letters, and a number image corresponding to the numbers, if the license plate image is normal;

inputting the area number image, the letter image, the number image to an area number recognition CNN, a letter recognition CNN, a number recognition CNN respectively to recognize an area number, letters, and numbers and outputting the area number, letters, and numbers respectively; and

outputting a license plate number recognition result in a form of “area number letter number” according to the area number, letters, and numbers output;

wherein the step of acquiring the vehicle image and inputting the vehicle image to the vehicle searching CNN to search whether there is the vehicle corresponding to the vehicle image comprises steps:

acquiring the vehicle image in a video stream data, the vehicle image being represented in a tensor mode;

obtaining a first feature map according to multiple geometric space transformation correction, convolution processing, and pooling processing in the vehicle image by a space transformation network and the CNN;

outputting a column vector by vectoring the first feature map and connecting a plurality of fully connected layers to obtain a weight matrix of the fully connected layers;

performing a tensor column decomposition on the weight matrix of the fully connected layer to extract a first feature vector; and

inputting the first feature vector to an output layer of the CNN to classify and predict, and optimizing adjustable parameters to search whether there is a vehicle corresponding to the vehicle image.

2. The license plate recognition method according to claim 1 , wherein the step of inputting the license plate image to the license plate searching CNN to search whether there is the license plate corresponding to the license plate image comprises steps:

obtaining a second feature map according to multiple geometric space transformation correction, convolution processing, and pooling processing on the license plate image by the space transformation network and the CNN;

outputting the column vector by vectoring the second feature map and connecting a plurality of fully connected layers to obtain the weight matrix of the fully connected layers;

performing the tensor column decomposition on the weight matrix of the folly connected layer to extract a second feature vector; and

inputting the second feature vector to the output layer of the CNN to classify and predict, and optimizing the adjustable parameters to search whether there is the license plate corresponding to the license plate image.

3. The license plate recognition method according to claim 1 , wherein the step of optimizing the adjustable parameters is to optimize all the adjustable parameters in the network according to a back-propagation (BP) algorithm, wherein an update of a space transformation matrix, a weight and an offset is according to a local gradient calculation of an error returned from a latter layer.

4. The license plate recognition method according to claim 3 , wherein the step of inputting the license plate image to the tensor neural network to detect the license plate image is abnormal or not comprises steps:

inputting the license plate image to the trained tensor neural network directly to obtain a corresponding direct probability density ratio; and

comparing the direct probability density ratio with a critical direct probability density ratio obtained after completing a training of the tensor neural network, if the direct probability density ratio is greater than the critical directly probability density ratio, the license plate image is normal; if not, the license plate image is abnormal.

5. A license plate recognition device, comprising:

a data acquisition module configured to acquire a vehicle image in a vehicle video stream image, acquire a license plate image in the vehicle image, and acquire a recognition image in the license plate image;

a vehicle search module configured to input the vehicle image to a vehicle searching CNN to search whether there is a vehicle corresponding to the vehicle image;

a license plate search module configured to input the license plate image to the license plate searching CNN to search whether there is a license plate corresponding to the license plate image;

an anomaly detection module configured to input the license plate image to a tensor neural network to detect the license plate image is abnormal or not; and

a license plate recognition module configured to input the recognition image to a determining neural network to judge an area number, letters, and numbers in the recognition image, and output an area number image corresponding to the area number, a letter image corresponding to the letters, and a number image corresponding to the numbers, and then input the area number image, the letter image, the number image to an area number recognition CNN, a letter recognition CNN, a number recognition CNN respectively to recognize an area number, letters, and numbers and outputting the area number, letters, and numbers respectively; and output a license plate number recognition result in a form of “area number letter number” according to the area number, letters, and numbers output;

wherein the vehicle search module comprises:

a vehicle image processing unit configured to obtain a first feature map according to multiple geometric space transformation correction, convolution processing, and pooling processing in the vehicle image by a space transformation network and the CNN;

a weight matrix acquisition unit configured to output a column vector by vectoring the first feature map and connecting a plurality of fully connected layers to obtain a weight matrix of the fully connected layers;

a tensor decomposition unit configured to perform a tensor column decomposition on the weight matrix of the fully connected layer to extract a first feature vector: and

a classification prediction unit configured to input the first feature vector to an output layer of the CNN to classify and predict, and optimizing adjustable parameters to search whether there is the vehicle corresponding to the vehicle image.

6. The license plate recognition device according to claim 5 , wherein the license plate search module comprises:

a license plate image processing unit configured to obtain a second feature map according to multiple geometric space transformation correction, convolution processing, and pooling processing on the license plate image by the space transformation network and the CNN;

the weight matrix acquisition unit configured to output the column vector by vectoring the second feature map and connecting a plurality of fully connected layers to obtain the weight matrix of the fully connected layers:

the tensor decomposition unit configured to perform the tensor column decomposition on the weight matrix of the fully connected layer to extract a second feature vector; and

the classification prediction unit configured to input the second feature vector to the output layer of the CNN to classify and predict, and optimizing the adjustable parameters to search whether there is the license plate corresponding to the license plate image.

7. The license plate recognition device according to claim 6 , wherein the anomaly detection module comprises:

a tensor neural network unit configured to obtain a corresponding direct probability density ratio according to the license plate image input: and

a comparison unit configured to compare the direct probability density ratio with a critical direct probability density ratio obtained after completing a training of the tensor neural network; if the direct probability density ratio is greater than the critical directly probability density ratio, the license plate image is normal; if not, the license plate image is abnormal.

8. A user equipment, comprising a license plate recognition device; wherein the license plate recognition device comprises:

a data acquisition module configured to acquire a vehicle image in a vehicle video stream image, acquire a license plate image in the vehicle image, and acquire a recognition image in the license plate image;

a vehicle search module configured to input the vehicle image to a vehicle searching CNN to search whether there is a vehicle corresponding to the vehicle image:

a license plate search module configured to input the license plate image to the license plate searching CNN to search whether there is a license plate corresponding to the license plate image;

an anomaly detection module configured to input the license plate image to a tensor neural network to detect the license plate image is abnormal or not; and

a license plate recognition module configured to input the recognition image to a determining neural network to judge an area number, letters, and numbers in the recognition image, and output an area number image corresponding to the area number, a letter image corresponding to the letters, and a number image corresponding to the number, and then input the area number image, the letter image, the number image to ail area number recognition CNN, a letter recognition CNN, a number recognition CNN respectively to recognize an area number, letters, and numbers and outputting the area number, letters, and numbers respectively: and output a license plate number recognition result in a form of “area number letter number” according to the area number, letters, and numbers output;

wherein the vehicle search module comprises:

a vehicle image processing unit configured to obtain a first feature map according to multiple geometric space transformation correction, convolution processing, and pooling processing in the vehicle image by a space transformation network and the CNN;

a weight matrix acquisition unit configured to output a column vector by vectoring the first feature map and connecting a plurality of fully connected layers to obtain a weight matrix of the folly connected layers;

a tensor decomposition unit configured to perform a tensor column decomposition on the weight matrix of the fully connected layer to extract a first feature vector; and

a classification prediction unit configured to input the first feature vector to an output layer of the CNN to classify and predict, and optimizing adjustable parameters to search whether there is the vehicle corresponding to the vehicle image.

9. The user equipment according to claim 8 , wherein the license plate search module comprises:

a license plate image processing unit configured to obtain a second feature map according to multiple geometric space transformation correction, convolution processing, and pooling processing on the license plate image by the space transformation network and the CNN;

the weight matrix acquisition unit configured to output the column vector by vectoring the second feature map and connecting a plurality of fully connected layers to obtain the weight matrix of the folly connected layers;

the tensor decomposition unit configured to perform the tensor column decomposition on the weight matrix of the folly connected layer to extract a second feature vector; and

the classification prediction unit configured to input the second feature vector to the output layer of the CNN to classify and predict, and optimizing the adjustable parameters to search whether there is the license plate corresponding to the license plate image.

10. The user equipment according to claim 9 , wherein the anomaly detection module comprises:

a tensor neural network unit configured to obtain a corresponding direct probability density ratio according to the license plate image input; and

a comparison unit configured to compare the direct probability density ratio with a critical direct probability density ratio obtained after completing a training of the tensor neural network; if the direct probability density ratio is greater than the critical directly probability density ratio, the license plate image is normal; if not, the license plate image is abnormal.