Voting-based multi-plate recognition

1 . A camera device comprising:

an image sensor;

a set of processors comprising a vision processor unit (VPU);

one or more memory devices storing instructions that, when executed by the set of processors, perform operations comprising:

updating a tracklet by, for each respective frame of a frame sequence obtained with the image sensor:

determining a respective bounding box that surrounds a respective sub-image of the respective frame;

determining, with the VPU, a respective string and respective confidence values associated with the respective string by providing, as an input, an object recognition model with the respective sub-image;

updating the tracklet to comprise the respective bounding box based on the respective string and at least one string generated by the object recognition model for a previous frame; and

updating a voting table by adding the respective confidence values to the voting table;

generating an aggregated string based on the voting table by, for a set of positions of the aggregated string, determining a set of characters associated with a set of maximum confidence values associated with the set of positions indicated by the voting table; and

associating the aggregated string with the tracklet in a data structure;

wherein the frame sequence comprises a first frame and a second frame, and wherein the first frame and the second frame are not consecutive frames, the operations further comprising:

determining an intersection over union (IoU) value of a first sub-image of the first frame and a second sub-image of the second frame; and

updating a scan region of the second frame by masking the second sub-image in response to a determination that the IoU value is less than an IoU threshold, wherein determining the respective bounding box comprises determining the respective bounding box based on the scan region.

2 . The camera device of claim 1 , wherein the image sensor directly provides image data to the VPU.

3 . The camera device of claim 1 , the operations further comprising:

determining a plurality of quality scores by, for each frame of the frame sequence, determining a respective quality score of the plurality of quality scores based on a respective sharpness of the respective frame;

identifying a display image by selecting an image or a sub-image of a candidate frame associated with a greatest quality score of the plurality of quality scores; and

transmitting, from the camera device, the display image to a remote data store.

4 . The camera device of claim 1 , the operations further comprising:

determining a mean slope based on a plurality of sub-images of a plurality of bounding boxes;

determining a character bounding box of a character shown in a license plate; and

rotating a sub-image surrounded by the character bounding box based on the mean slope.

5 . The camera device of claim 1 , wherein the tracklet associates 20 consecutive frames.

6 . A camera device comprising:

an image sensor;

a set of processors comprising a vision processor unit (VPU);

one or more memory devices storing instructions that, when executed by the set of processors, perform operations comprising:

updating a tracklet by, for each respective frame of a frame sequence obtained with the image sensor:

determining a respective bounding box that surrounds a respective sub-image of the respective frame;

determining, with the VPU, a respective string and respective confidence values associated with the respective string by providing, as an input, an object recognition model with the respective sub-image;

updating the tracklet to comprise the respective bounding box based on the respective string and at least one string generated by the object recognition model for a previous frame; and

updating a voting table by adding the respective confidence values to the voting table;

generating an aggregated string based on the voting table by, for a set of positions of the aggregated string, determining a set of characters associated with a set of maximum confidence values associated with the set of positions indicated by the voting table; and

associating the aggregated string with the tracklet in a data structure;

wherein the frame sequence comprises a first frame and a second frame;

the first frame and the second frame are consecutively arranged in the frame sequence;

updating the tracklet comprises:

determining an edit distance between a first sequence of the first frame and a second sequence of the second frame; and

updating the tracklet in response to a detection that the edit distance satisfies an edit distance threshold; and

a time difference between obtaining the first frame and determining the edit distance is less than 100 milliseconds (ms).

7 . A method comprising:

updating a tracklet by, for each respective frame of a frame sequence obtained with an image sensor of a camera device:

determining a respective bounding box that surrounds a respective sub-image of the respective frame;

determining a respective string and respective confidence values associated with the respective string by providing, as an input, an object recognition model with the respective sub-image;

updating the tracklet to comprise the respective bounding box based on the respective string and at least one string generated by the object recognition model for a previous frame; and

updating a voting table by adding the respective confidence values to the voting table;

generating an aggregated string based on the voting table by, for a set of positions of the aggregated string, determining a set of characters associated with a set of maximum confidence values associated with the set of positions indicated by the voting table; and

associating the aggregated string with the tracklet in a data structure;

wherein the frame sequence comprises a first frame and a second frame, and

wherein the first frame and the second frame are not consecutive frames, and further comprising:

determining an intersection over union (IoU) value of a first sub-image of the first frame and a second sub-image of the second frame; and

updating a scan region of the second frame by masking the second sub-image in response to a determination that the IoU value is less than an IoU threshold, wherein determining the respective bounding box comprises determining the respective bounding box based on the scan region.

8 . The method of claim 7 , wherein:

the frame sequence comprises the first frame, the second frame, and a third frame;

determining, for each respective frame of the frame sequence, the respective bounding box comprises:

determining a first bounding box of the first frame, the first bounding box surrounding a first sub-image; and

determining a second bounding box of the second frame, the second bounding box surrounding a second sub-image; and

updating the tracklet to comprise the respective bounding box comprises:

determining a first character sequence by providing the object recognition model with the first sub-image;

determining a second character sequence by providing the object recognition model with the second sub-image;

computing an edit distance based on the first character sequence and the second character sequence; and

updating the tracklet to comprise the respective bounding box based on a result indicating whether the edit distance satisfies an edit distance threshold.

9 . The method of claim 7 , wherein determining the respective string comprises resizing the respective sub-image by a factor greater than or equal to three.

10 . The method of claim 7 , further comprising:

modifying the aggregated string by switching a first character of the aggregated string with a second character to generate a modified string;

generating a new license plate image by providing, as an input to an image generation model, an image portion obtained from the frame sequence and the modified string; and

storing the modified string and the new license plate image in a data structure.

11 . The method of claim 10 , wherein the second character is one of the character “0,” the character “O,” or the character “D”.

12 . The method of claim 10 , further comprising:

accessing, with a server, the data structure to retrieve the new license plate image and the modified string;

training a machine learning model based on the new license plate image and the modified string; and

sending, from the server, parameters of the trained machine learning model to one or more memory devices of the camera device.

13 . A set of non-transitory, machine-readable media storing instructions that, when executed by a set of processors, perform operations comprising:

updating a tracklet by, for each respective frame of a frame sequence obtained with an image sensor of a camera device:

determining a respective bounding box that surrounds a respective sub-image of the respective frame;

determining a respective string and respective confidence values associated with the respective string using an object recognition model based on the respective sub-image;

updating the tracklet to comprise the respective bounding box based on the respective string and at least one string generated by the object recognition model for a previous frame; and

updating a voting table by adding the respective confidence values to the voting table;

generating an aggregated string based on the voting table by, for a set of positions of the aggregated string, determining a set of characters associated with a set of maximum confidence values associated with the set of positions indicated by the voting table; and

associating the aggregated string with the tracklet in a data structure;

wherein the frame sequence comprises a first frame and a second frame, and wherein the first frame and the second frame are not consecutive frames, the operations further comprising:

determining an intersection over union (IoU) value of a first sub-image of the first frame and a second sub-image of the second frame; and

updating a scan region of the second frame by masking the second sub-image in response to a determination that the IoU value is less than an IoU threshold, wherein determining the respective bounding box comprises determining the respective bounding box based on the scan region.

14 . The set of non-transitory, machine-readable media of claim 13 , wherein the tracklet is a first device tracklet, and wherein the aggregated string is a first aggregated string, the operations further comprising:

receiving, from a second device, a tracklet identifier for a second device tracklet, wherein the tracklet identifier is associated with an aggregated string in a memory of the second device; and

associating the tracklet identifier with the first device tracklet in response to a detection that an aggregated string associated with the second device tracklet matches the first aggregated string.

15 . The set of non-transitory, machine-readable media of claim 13 , wherein a set of weight parameter values and a set of activation parameter values of the object recognition model are stored as 8-bit values.

16 . The set of non-transitory, machine-readable media of claim 15 , the operations further comprising:

modifying the aggregated string by switching a first character of the aggregated string with a second character to generate a modified string;

generating a new license plate image by providing, as an input to an image generation model, an image obtained from the frame sequence and the modified string; and

storing the modified string and the new license plate image in a data store.

17 . The set of non-transitory, machine-readable media of claim 13 , the operations further comprising:

determining a mean slope based on a plurality of sub-images of a plurality of bounding boxes of a frame of the frame sequence; and

for each identified character of an identified license plate of the frame:

determining a respective bounding box correlated with the respective license plate; and

rotating the respective bounding box of the frame based on the mean slope.

18 . The set of non-transitory, machine-readable media of claim 13 , the operations further comprising:

the frame sequence comprising the first frame, the second frame, and a third frame;

determining, for each respective frame of the frame sequence, the respective bounding box by:

determining a first bounding box of the first frame, the first bounding box surrounding a first sub-image; and

determining a second bounding box of the second frame, the second bounding box surrounding a second sub-image; and

updating the tracklet to comprise the respective bounding box by:

determining a first character sequence by providing the object recognition model with the first sub-image;

determining a second character sequence by providing the object recognition model with the second sub-image;

computing an edit distance based on the first character sequence and the second character sequence; and

updating the tracklet to comprise the respective bounding box based on a result indicating whether the edit distance satisfies an edit distance threshold.

19 . The set of non-transitory, machine-readable media of claim 13 , wherein determining the respective string comprises resizing the respective sub-image.