Ball detection and tracking device, system and method

1. A ball detection and tracking system, the system comprising:

a plurality of visual detectors each having one or more light sensors for sensing a plurality of frames of images; and

a mobile ball detection and tracking device coupled with the visual detectors for receiving the frames from the visual detection devices, the ball detection and tracking device having a processor and a non-transitory computer-readable memory storing a detection and tracking agent including:

a detection module that ranks a plurality of blob detection algorithms based on a detection metric, selects a best ranked detection algorithm to be a base detection algorithm, and uses the base detection algorithm to identify one or more first candidate blobs within a first frame of the frames and one or more second candidate blobs within a second frame of the frames; and

a tracking module that generates a track for each of the first candidate blobs including a predicted location of each of the tracks within the second frame, ranks each of the second candidate blobs based on a blob metric and each of the tracks based on a track metric, and assigns one of the second candidate blobs to a best ranked one of the tracks if the assignment satisfies a cost threshold as measured by a cost function:

wherein the cost threshold is dynamically adjusted each time one of the blobs is associated with one of the tracks based on a cost of associating the one of the blobs with the one of the tracks as measured by the cost function.

2. The system of claim 1 , wherein the tracking module determines if any of the second candidate blobs satisfy the cost threshold for the best ranked one of the tracks, and of those that do satisfy the cost threshold, assigns the second blob having the best rank to the best ranked one of the tracks.

3. The system of claim 2 , wherein, at each iteration, after excluding any of the second candidate blobs that have already been assigned and any of the tracks that have been assigned to, the tracking module iteratively:

determines if any of the second candidate blobs satisfy the cost threshold for the best ranked one of the tracks; and

of those that do satisfy the cost threshold, assigns the second blob having the best rank to the best ranked one of the tracks until all of the tracks have been assigned to.

4. The system of claim 1 , wherein after using the base detection algorithm, the detection module:

uses one or more other detection algorithms of the plurality of detection algorithms to identify one or more other first candidate blobs within a first frame of the frames and one or more other second candidate blobs within a second frame of the frames;

classifies one of the second blobs as a ball blob, and for each of the other detection algorithms, classifies one of the other second blobs identified by the other detection algorithm as other ball blobs; and

combines the characteristics of the ball blob and each of the other ball blobs to generate a unionized blob.

5. The system of claim 1 , wherein the second frame has an initial exposure level, the detection module:

adjusts the second frame to have a second exposure level and uses the base detection algorithm to identify one or more other second candidate blobs within the second frame as adjusted;

classifies one of the second blobs as a ball blob and one of the other second blobs as another ball blob; and

creates a unionized ball blob that only includes pixels that are common to both the ball blob and the other ball blob.

6. The system of claim 1 , wherein the tracking module selects the cost function from a plurality of stored cost functions based on a cost score of each of the cost functions.

7. The system of claim 1 , wherein the tracking module determines if the assigning of the one of the second candidate blobs to the best ranked one of the tracks satisfies a second cost threshold as measured by a second cost function and refrains from making the assignment if the assignment does not satisfy the second cost threshold.

8. The system of claim 1 , wherein the tracking module refrains from generating a track for one or more of the first blobs if the blobs do not satisfy a blob metric threshold as measured by the blob metric.

9. The system of claim 1 , wherein the tracking module determines whether one of the tracks is a ball track based on one or more of a group consisting of an age of the one of the tracks, visibility of the one of the tracks, and a number of the frames in which the one of the tracks has had the best rank based on the track metric.

10. The system of claim 1 , wherein selecting the best ranked detection algorithm to be the base detection algorithm comprises:

excluding from selection any of the blob detection algorithms that result in invisibility of the ball for one or more frames; and

if all of the blob detection algorithms results in invisibility of the ball for the one or more frames, training a model based on the results of all of the blob detection algorithms to be the base detection algorithm.

11. A ball detection and tracking system, the system comprising:

a plurality of visual detectors each having one or more light sensors for sensing a plurality of frames of images; and

a mobile ball detection and tracking device coupled with the visual detectors for receiving the frames from the visual detection devices, the ball detection and tracking device having a processor and a non-transitory computer-readable memory storing a detection and tracking agent including:

a detection module that ranks a plurality of blob detection algorithms based on a detection metric, selects a best ranked detection algorithm to be a base detection algorithm, and uses the base detection algorithm to identify one or more first candidate blobs within a first frame of the frames and one or more second candidate blobs within a second frame of the frames; and

a tracking module that generates a track for each of the first candidate blobs including a predicted location of each of the tracks within the second frame, ranks each of the second candidate blobs based on a blob metric and each of the tracks based on a track metric, and assigns one of the second candidate blobs to a best ranked one of the tracks if the assignment satisfies a cost threshold as measured by a cost function except when the predicted location of one of the tracks lies within one of the second blobs, the tracking module assigns the one of the second blobs to the one of the tracks even if assigning a different one of the second blobs would be more cost efficient as measured by the cost function.

12. A mobile ball detection and tracking device, the device comprising:

a processor;

a plurality of visual detectors each having one or more light sensors for sensing a plurality of frames of images; and

a non-transitory computer-readable memory storing a detection and tracking agent including:

a detection module that ranks a plurality of blob detection algorithms based on a detection metric, selects the best ranked detection algorithm to be a base detection algorithm, and uses the base detection algorithm to identify one or more first candidate blobs within a first frame of the frames and one or more second candidate blobs within a second frame of the frames; and

a tracking module that generates a track for each of the first candidate blobs including a predicted location of each of the tracks within the second frame, ranks each of the second candidate blobs based on a blob metric and each of the tracks based on a track metric, and assigns one of the second candidate blobs to the best ranked one of the tracks if the assignment satisfies a cost threshold as measured by a cost function:

wherein the cost threshold is dynamically adjusted each time one of the blobs is associated with one of the tracks based on a cost of associating the one of the blobs with the one of the tracks as measured by the cost function.

13. The device of claim 12 , wherein the tracking module determines if any of the second candidate blobs satisfy the cost threshold for the best ranked one of the tracks, and of those that do satisfy the cost threshold, assigns the second blob having the best rank to the best ranked one of the tracks.

14. The device of claim 13 , wherein, at each iteration, after excluding any of the second candidate blobs that have already been assigned and any of the tracks that have been assigned to, the tracking module iteratively:

determines if any of the second candidate blobs satisfy the cost threshold for the best ranked one of the tracks; and

of those that do satisfy the cost threshold, assigns the second blob having the best rank to the best ranked one of the tracks until all of the tracks have been assigned to.

15. The device of claim 12 , wherein after using the base detection algorithm, the detection module:

uses one or more other detection algorithms of the plurality of detection algorithms to identify one or more other first candidate blobs within a first frame of the frames and one or more other second candidate blobs within a second frame of the frames;

classifies one of the second blobs as a ball blob, and for each of the other detection algorithms, classifies one of the other second blobs identified by the other detection algorithm as other ball blobs; and

combines the characteristics of the ball blob and each of the other ball blobs to generate a unionized blob.

16. The device of claim 12 , wherein the second frame has an initial exposure level, the detection module:

adjusts the second frame to have a second exposure level and uses the base detection algorithm to identify one or more other second candidate blobs within the second frame as adjusted;

classifies one of the second blobs as a ball blob and one of the other second blobs as another ball blob; and

creates a unionized ball blob that only includes pixels that are common to both the ball blob and the other ball blob.

17. The device of claim 12 , wherein the tracking module selects the cost function from a plurality of stored cost functions based on a cost score of each of the cost functions.

18. The device of claim 12 , wherein the tracking module determines if the assigning of the one of the second candidate blobs to the best ranked one of the tracks satisfies a second cost threshold as measured by a second cost function and refrains from making the assignment if the assignment does not satisfy the second cost threshold.

19. The device of claim 12 , wherein the tracking module refrains from generating a track for one or more of the first blobs if the blobs do not satisfy a blob metric threshold as measured by the blob metric.

20. The device of claim 12 , wherein the tracking module determines whether one of the tracks is a ball track based on one or more of a group consisting of an age of the one of the tracks, visibility of the one of the tracks, and a number of the frames in which the one of the tracks has had the best rank based on the track metric.

21. A mobile ball detection and tracking device, the device comprising:

a processor;

a plurality of visual detectors each having one or more light sensors for sensing a plurality of frames of images; and

a non-transitory computer-readable memory storing a detection and tracking agent including:

a detection module that ranks a plurality of blob detection algorithms based on a detection metric, selects the best ranked detection algorithm to be a base detection algorithm, and uses the base detection algorithm to identify one or more first candidate blobs within a first frame of the frames and one or more second candidate blobs within a second frame of the frames; and

a tracking module that generates a track for each of the first candidate blobs including a predicted location of each of the tracks within the second frame, ranks each of the second candidate blobs based on a blob metric and each of the tracks based on a track metric, and assigns one of the second candidate blobs to the best ranked one of the tracks if the assignment satisfies a cost threshold as measured by a cost function except when the predicted location of one of the tracks lies within one of the second blobs, the tracking module assigns the one of the second blobs to the one of the tracks even if assigning a different one of the second blobs would be more cost efficient as measured by the cost function.

22. A method of detecting and tracking a ball, the method comprising:

sensing a plurality of frames of images with a plurality of visual detectors each having one or more light sensors;

receiving the frames with a mobile ball detection and tracking device;

ranking a plurality of blob detection algorithms based on a detection metric, selecting the best ranked detection algorithm to be a base detection algorithm, and using the base detection algorithm to identify one or more first candidate blobs within a first frame of the frames and one or more second candidate blobs within a second frame of the frames with the ball detection and tracking device;

generating a track for each of the first candidate blobs including a predicted location of each of the tracks within the second frame, ranking each of the second candidate blobs based on a blob metric and each of the tracks based on a track metric, and assigning one of the second candidate blobs to the best ranked one of the tracks if the assignment satisfies a cost threshold as measured by a cost function with the ball detection and tracking device; and dynamically adjusting the cost threshold each time one of the blobs is associated with one of the tracks based on a cost of associating the one of the blobs with the one of the tracks as measured by the cost function.

23. The method of claim 22 , further comprising determining if any of the second candidate blobs satisfy the cost threshold for the best ranked one of the tracks, and of those that do satisfy the cost threshold, assigning the second blob having the best rank to the best ranked one of the tracks.

24. The method of claim 23 , further comprising, at each iteration, after excluding any of the second candidate blobs that have already been assigned and any of the tracks that have been assigned to, iteratively:

determining if any of the second candidate blobs satisfy the cost threshold for the best ranked one of the tracks; and

of those that do satisfy the cost threshold, assigning the second blob having the best rank to the best ranked one of the tracks until all of the tracks have been assigned to.

25. The method of claim 22 , further comprising, after using the base detection algorithm, using one or more other detection algorithms of the plurality of detection algorithms to identify one or more other first candidate blobs within a first frame of the frames and one or more other second candidate blobs within a second frame of the frames; classifying one of the second blobs as a ball blob, and for each of the other detection algorithms, classifies one of the other second blobs identified by the other detection algorithm as other ball blobs; and combining the characteristics of the ball blob and each of the other ball blobs to generate a unionized blob.

26. The method of claim 22 , wherein the second frame has an initial exposure level, further comprising:

adjusting the second frame to have a second exposure level and uses the base detection algorithm to identify one or more other second candidate blobs within the second frame as adjusted;

classifying one of the second blobs as a ball blob and one of the other second blobs as another ball blob; and

creating a unionized ball blob that only includes pixels that are common to both the ball blob and the other ball blob.

27. The method of claim 22 , further comprising selecting the cost function from a plurality of stored cost functions based on a cost score of each of the cost functions.

28. The method of claim 22 , further comprising determining if the assigning of the one of the second candidate blobs to the best ranked one of the tracks satisfies a second cost threshold as measured by a second cost function and refraining from making the assignment if the assignment does not satisfy the second cost threshold.

29. The method of claim 22 , further comprising refraining from generating a track for one or more of the first blobs if the blobs do not satisfy a blob metric threshold as measured by the blob metric.

30. The method of claim 22 , further comprising determining with the tracking module whether one of the tracks is a ball track based on one or more of a group consisting of an age of the one of the tracks, visibility of the one of the tracks, and a number of the frames in which the one of the tracks has had the best rank based on the track metric.

31. A method of detecting and tracking a ball, the method comprising:

sensing a plurality of frames of images with a plurality of visual detectors each having one or more light sensors;

receiving the frames with a mobile ball detection and tracking device;

ranking a plurality of blob detection algorithms based on a detection metric, selecting the best ranked detection algorithm to be a base detection algorithm, and using the base detection algorithm to identify one or more first candidate blobs within a first frame of the frames and one or more second candidate blobs within a second frame of the frames with the ball detection and tracking device;

generating a track for each of the first candidate blobs including a predicted location of each of the tracks within the second frame, ranking each of the second candidate blobs based on a blob metric and each of the tracks based on a track metric; and

assigning one of the second candidate blobs to the best ranked one of the tracks if the assignment satisfies a cost threshold as measured by a cost function with the ball detection and tracking device except when the predicted location of one of the tracks lies within one of the second blobs, assigning the one of the second blobs to the one of the tracks even if assigning a different one of the second blobs would be more cost efficient as measured by the cost function.