System and method for online real-time multi-object tracking

1. A system comprising:

a data processor; and

a memory for storing an online real-time multi-object tracking system, executable by the data processor, the online real-time multi-object tracking system being configured to:

receive image frame data from at least one camera;

detect objects in the image frame data;

extract an appearance feature from each of the objects;

generate similarity data corresponding to a similarity between objects detected in a first image frame compared with objects detected in a second image frame;

maintain a template for each of the objects detected in the image frame data, wherein the template corresponds to the appearance feature extracted from each of the objects;

update the template based on the similarity data; and

match the objects detected in the first image frame with the objects detected in the second image frame based on the similarity data.

2. The system of claim 1 being further configured to create a finite state machine for each of the objects detected, wherein the finite state machine for each of the objects represents a current object detection state of a plurality of possible object detection states for a corresponding object.

3. The system of claim 1 being further configured to generate the similarity data using a motion feature similarity and an appearance feature similarity.

4. The system of claim 1 being further configured to generate the similarity data using a motion feature similarity based on a prediction of a Kalman filter and a bounding box position of a detected object.

5. The system claim 1 wherein a number of the appearance feature extracted from each of the objects is at least two, wherein a number of the template for each of the objects is at least two.

6. The system claim 1 wherein the at least one camera is associated with an autonomous vehicle.

7. The system claim 1 wherein the first image frame is earlier than the second image frame on a timeline.

8. A method comprising:

receiving image frame data from at least one camera;

detecting objects in the image frame data;

extracting an appearance feature from each of the objects;

generating similarity data corresponding to a similarity between objects detected in a first image frame compared with objects detected in a second image frame;

maintaining a template for each of the objects detected in the image frame data, wherein the template corresponds to the appearance feature extracted from each of the objects;

updating the template based on the similarity data; and

matching the objects detected in the first image frame with the objects detected in the second image frame based on the similarity data.

9. The method of claim 8 further comprising using the similarity data to generate data association results corresponding to a best match between the objects in the first image frame and the objects in the second image frame.

10. The method of claim 9 further comprising causing state transitions in finite state machines for each of the objects according to the data association results.

11. The method of claim 10 further comprising outputting output data corresponding to states of the finite state machines for each of the objects.

12. The method of claim 11 further comprising smoothing the output data.

13. The method of claim 11 wherein the output data is adjusted based on a weighted average calculation for data of the objects detected, or a prediction of a Kalman filter.

14. The method of claim 10 wherein the finite state machines for each of the objects comprise states from the group consisting of: initialized, tracked, lost, and removed.

15. A non-transitory machine-useable storage medium embodying instructions which, when executed by a machine, cause the machine to:

receive image frame data from at least one camera;

detect objects in the image frame data;

extract an appearance feature from each of the objects;

generate similarity data corresponding to a similarity between objects detected in a first image frame compared with objects detected in a second image frame;

maintain a template for each of the objects detected in the image frame data, wherein the template corresponds to the appearance feature extracted from each of the objects;

update the template based on the similarity data; and

match the objects detected in the first image frame with the objects detected in the second image frame based on the similarity data.

16. The non-transitory machine-useable storage medium of claim 15 being further configured to generate the similarity data based on determining a similarity score between a position of a bounding box predicted by a Kalman filter and a position of a bounding box of each of the objects detected in an image frame.

17. The non-transitory machine-useable storage medium of claim 15 wherein the appearance feature is extracted by a pre-trained convolutional neural network (CNN).

18. The non-transitory machine-useable storage medium of claim 15 being further configured to determine a similarity score for a detected object and a templated object based on the appearance feature, wherein the non-transitory machine-useable storage medium is configured to update the template when:

the similarity score is less than a first threshold; and

a confidence level of a bounding box of the detected object is higher than a second threshold.

19. The non-transitory machine-useable storage medium of claim 15 being further configured to use the similarity data to generate data association results corresponding to a best matching between the objects in the first image frame and the objects in the second image frame, wherein the data association results are generated based on results comprising: matched objects detected with object detection results, unmatched objects detected, and unmatched object detection results.

20. The non-transitory machine-useable storage medium of claim 15 being further configured to generate the similarity data using an appearance feature similarity calculated as a Euclidean distance between the appearance feature of an object defined as the template and the appearance feature of the detected objects.