Granted Patent B2

US 12,361,570 · App. 18/641,880 · Granted Jul 15, 2025

Inventors: Daniel Forsgren (Enebyberg, SE); Anton Mikael Jansson (Stockholm, SE); Stein Norheim (Spånga, SE)

Assignee: Topgolf Sweden AB

G06T7/292G06T7/251G06T7/77G06V10/255G06V20/52G06V20/647G06T2207/10021G06T2207/10028G06T2207/20084G06T2207/30224G06T2207/30241

Abstract

Claims (103)

1. A method comprising:

obtaining three-dimensional positions of objects of interest registered by an object detection system configured to allow more false positives for the registered objects of interest so as to minimize false negatives;

forming hypotheses of objects in motion in three-dimensional space using a filter applied to the three-dimensional positions of the registered objects of interest, wherein the filter allows connections between specific ones of the registered objects of interest when estimated three-dimensional velocity vectors for the specific objects of interest roughly correspond to an object in motion in three-dimensional space across time;

eliminating a proper subset of the hypotheses that are not further extended by connections made, during the forming, with at least one additional object of interest registered by the object detection system;

specifying at least one three-dimensional track of at least one ball in motion in three-dimensional space by applying a full three-dimensional physics model to data for the three-dimensional positions of the registered objects of interest used in the forming of at least one hypothesis that survives the eliminating; and

outputting for display the at least one three-dimensional track of the at least one ball in motion in three-dimensional space;

wherein the obtaining comprises receiving or generating the three-dimensional positions of the registered objects of interest in which a majority of the registered objects of interest are false positives including false detections by individual sensors and false combinations of detections from respective ones of the individual sensors including combined detections that are not from a same object.

2. The method of claim 1 , wherein forming the hypotheses of objects in motion in three-dimensional space using the filter comprises:

using a first model of motion in three-dimensional space when a number of registered detections for a given hypothesis is less than a threshold value, wherein the first model provides a very loose filter for what constitutes an object in Newtonian motion; and

using a second model of motion in three-dimensional space when the number of registered detections for the given hypothesis is greater than or equal to the threshold value, wherein the second model incorporates some knowledge of 3D physics.

3. The method of claim 2 , wherein the first model of motion in three-dimensional space is a linear model, and the second model of motion in three-dimensional space is a recurrent neural network model.

4. The method of claim 2 , wherein forming the hypotheses of objects in motion in three-dimensional space comprises:

predicting a next three-dimensional position for a registered detection for the given hypothesis using the first model or the second model, in accordance with the number of registered detections for the given hypothesis and the threshold value;

searching a spatial data structure containing all of the three-dimensional positions of objects of interest registered by the object detection system for a next time slice to find a set of three-dimensional positions within a predefined distance of the predicted next three-dimensional position;

using the predicted next three-dimensional position for the given hypothesis when the set of three-dimensional positions is a null set;

using a single three-dimensional position for the given hypothesis when the set of three-dimensional positions contains only the single three-dimensional position; and

when the set of three-dimensional positions contains two or more three-dimensional positions,

sorting the two or more three-dimensional positions based on proximity to the predicted next three-dimensional position to form a sorted set,

removing any less proximate three-dimensional positions in the sorted set beyond a predefined threshold number greater than two, and

branching the given hypothesis into two or more hypotheses in a group of hypotheses using the two or more three-dimensional positions remaining in the sorted set.

5. The method of claim 2 , wherein forming the hypotheses of objects in motion in three-dimensional space when the number of registered detections for the given hypothesis is greater than or equal to the threshold value comprises:

identifying a data point in the given hypothesis that has a local minimum vertical position;

checking respective vertical components of estimated three-dimensional velocity vectors before and after the data point in the given hypothesis; and

designating the data point as a ground impact when a first of the respective vertical components is negative and a second of the respective vertical components is positive.

6. The method of claim 5 , wherein the respective vertical components are from averages of the estimated three-dimensional velocity vectors before and after the data point in the given hypothesis, and the method comprises:

selecting at least one time window based on a noise level associated with the estimated three-dimensional velocity vectors, a minimum expected flight time on one or both sides of the data point, or both the noise level and the minimum expected flight time; and

computing the averages of the estimated three-dimensional velocity vectors before and after the data point that fall within the at least one time window.

7. The method of claim 5 , wherein forming the hypotheses of objects in motion in three-dimensional space when the number of registered detections for the given hypothesis is greater than or equal to the threshold value comprises dividing the given hypothesis into distinct segments comprising a flight segment, one or more bounce segments, and a roll segment, and the dividing comprises the identifying, the checking, the designating, and:

classifying a first segment of the given hypothesis before a first designated ground impact as the flight segment;

classifying a next segment of the given hypothesis after each next designated ground impact as one of the one or more bounce segments when an angle between a first estimated velocity vector before the next designated ground impact and a second estimated velocity vector after the next designated ground impact is greater than a threshold angle; and

classifying the next segment of the given hypothesis after a next designated ground impact as the roll segment when the angle between the first estimated velocity vector before the next designated ground impact and the second estimated velocity vector after the next designated ground impact is less than or equal to the threshold angle.

8. The method of claim 7 , wherein the given hypothesis is the at least one hypothesis that survives the eliminating, and specifying the at least one three-dimensional track of the at least one ball in motion in three-dimensional space comprises:

generating the data for the three-dimensional positions of the registered objects used in the forming of the at least one hypothesis by triangulating a more accurate 3D path using sensor observations registered by the object detection system in at least the flight segment; and

confirming the at least one ball in motion by applying the full three-dimensional physics model to the generated data for at least the flight segment.

9. The method of claim 1 , wherein the individual sensors comprise three or more sensors, and the obtaining comprises generating the three-dimensional positions by making respective combinations of the detections from respective pairs of the three or more sensors for a current time slice.

10. The method of claim 9 , wherein the three or more sensors comprise two cameras and a radar device, and the making comprises:

combining detections of a single object of interest by the two cameras using a stereo pairing of the two cameras to generate a first three-dimensional position of the single object of interest; and

combining detections of the single object of interest by the radar device and at least one of the two cameras to generate a second three-dimensional position of the single object of interest.

11. The method of claim 10 , wherein the three or more sensors comprise an additional camera, and the making comprises combining detections of the single object of interest by the additional camera and the two cameras using stereo pairings of the additional camera with each of the two cameras to generate a third three-dimensional position of the single object of interest and a fourth three-dimensional position of the single object of interest.

12. A system comprising:

two or more sensors and one or more first computers of an object detection system configured to register objects of interest and to allow more false positives for the registered objects of interest so as to minimize false negatives; and

one or more second computers configured to perform operations comprising

obtaining three-dimensional positions of objects of interest registered by the object detection system,

forming hypotheses of objects in motion in three-dimensional space using a filter applied to the three-dimensional positions of the registered objects of interest, wherein the filter allows connections between specific ones of the registered objects of interest when estimated three-dimensional velocity vectors for the specific objects of interest roughly correspond to an object in motion in three-dimensional space across time,

eliminating a proper subset of the hypotheses that are not further extended by connections made, during the forming, with at least one additional object of interest registered by the object detection system,

specifying at least one three-dimensional track of at least one ball in motion in three-dimensional space by applying a full three-dimensional physics model to data for the three-dimensional positions of the registered objects of interest used in the forming of at least one hypothesis that survives the eliminating, and

outputting for display the at least one three-dimensional track of the at least one ball in motion in three-dimensional space;

wherein the obtaining comprises receiving or generating the three-dimensional positions of the registered objects of interest in which a majority of the registered objects of interest are false positives including false detections by individual ones of the two or more sensors and false combinations of detections from respective ones of the two or more sensors including combined detections that are not from a same object.

13. The system of claim 12 , wherein the two or more sensors comprise cameras and at least one radar device.

14. The system of claim 12 , wherein the two or more sensors comprise a hybrid camera-radar sensor.

15. The system of claim 12 , wherein forming the hypotheses of objects in motion in three-dimensional space using the filter comprises:

using a first model of motion in three-dimensional space when a number of registered detections for a given hypothesis is less than a threshold value, wherein the first model provides a very loose filter for what constitutes an object in Newtonian motion; and

using a second model of motion in three-dimensional space when the number of registered detections for the given hypothesis is greater than or equal to the threshold value, wherein the second model incorporates some knowledge of 3D physics.

16. The system of claim 15 , wherein the first model of motion in three-dimensional space is a linear model, and the second model of motion in three-dimensional space is a recurrent neural network model.

17. The system of claim 15 , wherein forming the hypotheses of objects in motion in three-dimensional space comprises:

predicting a next three-dimensional position for a registered detection for the given hypothesis using the first model or the second model, in accordance with the number of registered detections for the given hypothesis and the threshold value;

searching a spatial data structure containing all of the three-dimensional positions of objects of interest registered by the object detection system for a next time slice to find a set of three-dimensional positions within a predefined distance of the predicted next three-dimensional position;

using the predicted next three-dimensional position for the given hypothesis when the set of three-dimensional positions is a null set;

using a single three-dimensional position for the given hypothesis when the set of three-dimensional positions contains only the single three-dimensional position; and

when the set of three-dimensional positions contains two or more three-dimensional positions,

sorting the two or more three-dimensional positions based on proximity to the predicted next three-dimensional position to form a sorted set,

removing any less proximate three-dimensional positions in the sorted set beyond a predefined threshold number greater than two, and

branching the given hypothesis into two or more hypotheses in a group of hypotheses using the two or more three-dimensional positions remaining in the sorted set.

18. The system of claim 15 , wherein forming the hypotheses of objects in motion in three-dimensional space when the number of registered detections for the given hypothesis is greater than or equal to the threshold value comprises:

identifying a data point in the given hypothesis that has a local minimum vertical position;

checking respective vertical components of estimated three-dimensional velocity vectors before and after the data point in the given hypothesis; and

designating the data point as a ground impact when a first of the respective vertical components is negative and a second of the respective vertical components is positive.

19. The system of claim 12 , wherein the two or more sensors comprise three or more sensors, and the obtaining comprises generating the three-dimensional positions by making respective combinations of the detections from respective pairs of the three or more sensors for a current time slice.

20. The system of claim 19 , wherein the three or more sensors comprise two cameras and a radar device, and the making comprises:

combining detections of a single object of interest by the two cameras using a stereo pairing of the two cameras to generate a first three-dimensional position of the single object of interest; and

combining detections of the single object of interest by the radar device and at least one of the two cameras to generate a second three-dimensional position of the single object of interest.

21. The system of claim 20 , wherein the three or more sensors comprise an additional camera, and the making comprises combining detections of the single object of interest by the additional camera and the two cameras using stereo pairings of the additional camera with each of the two cameras to generate a third three-dimensional position of the single object of interest and a fourth three-dimensional position of the single object of interest.

22. A non-transitory computer-readable medium encoding instructions that cause data processing apparatus associated with an object detection system comprising two or more sensors to perform operations comprising:

obtaining three-dimensional positions of objects of interest registered by the object detection system, which is configured to allow more false positives for the registered objects of interest so as to minimize false negatives;

forming hypotheses of objects in motion in three-dimensional space using a filter applied to the three-dimensional positions of the registered objects of interest, wherein the filter allows connections between specific ones of the registered objects of interest when estimated three-dimensional velocity vectors for the specific objects of interest roughly correspond to an object in motion in three-dimensional space across time;

eliminating a proper subset of the hypotheses that are not further extended by connections made, during the forming, with at least one additional object of interest registered by the object detection system;

specifying at least one three-dimensional track of at least one ball in motion in three-dimensional space by applying a full three-dimensional physics model to data for the three-dimensional positions of the registered objects of interest used in the forming of at least one hypothesis that survives the eliminating; and

outputting for display the at least one three-dimensional track of the at least one ball in motion in three-dimensional space;

wherein the obtaining comprises receiving or generating the three-dimensional positions of the registered objects of interest in which a majority of the registered objects of interest are false positives including false detections by individual sensors and false combinations of detections from respective ones of the individual sensors including combined detections that are not from a same object.

23. The non-transitory computer-readable medium of claim 22 , wherein forming the hypotheses of objects in motion in three-dimensional space using the filter comprises:

using a first model of motion in three-dimensional space when a number of registered detections for a given hypothesis is less than a threshold value, wherein the first model provides a very loose filter for what constitutes an object in Newtonian motion; and

using a second model of motion in three-dimensional space when the number of registered detections for the given hypothesis is greater than or equal to the threshold value, wherein the second model incorporates some knowledge of 3D physics.

24. The non-transitory computer-readable medium of claim 23 , wherein the first model of motion in three-dimensional space is a linear model, and the second model of motion in three-dimensional space is a recurrent neural network model.

25. The non-transitory computer-readable medium of claim 23 , wherein forming the hypotheses of objects in motion in three-dimensional space comprises:

predicting a next three-dimensional position for a registered detection for the given hypothesis using the first model or the second model, in accordance with the number of registered detections for the given hypothesis and the threshold value;

searching a spatial data structure containing all of the three-dimensional positions of objects of interest registered by the object detection system for a next time slice to find a set of three-dimensional positions within a predefined distance of the predicted next three-dimensional position;

using the predicted next three-dimensional position for the given hypothesis when the set of three-dimensional positions is a null set;

using a single three-dimensional position for the given hypothesis when the set of three-dimensional positions contains only the single three-dimensional position; and

when the set of three-dimensional positions contains two or more three-dimensional positions,

sorting the two or more three-dimensional positions based on proximity to the predicted next three-dimensional position to form a sorted set,

removing any less proximate three-dimensional positions in the sorted set beyond a predefined threshold number greater than two, and

branching the given hypothesis into two or more hypotheses in a group of hypotheses using the two or more three-dimensional positions remaining in the sorted set.

26. The non-transitory computer-readable medium of claim 23 , wherein forming the hypotheses of objects in motion in three-dimensional space when the number of registered detections for the given hypothesis is greater than or equal to the threshold value comprises:

identifying a data point in the given hypothesis that has a local minimum vertical position;

checking respective vertical components of estimated three-dimensional velocity vectors before and after the data point in the given hypothesis; and

designating the data point as a ground impact when a first of the respective vertical components is negative and a second of the respective vertical components is positive.

27. The non-transitory computer-readable medium of claim 26 , wherein the respective vertical components are from averages of the estimated three-dimensional velocity vectors before and after the data point in the given hypothesis, and the operations comprise:

selecting at least one time window based on a noise level associated with the estimated three-dimensional velocity vectors, a minimum expected flight time on one or both sides of the data point, or both the noise level and the minimum expected flight time; and

computing the averages of the estimated three-dimensional velocity vectors before and after the data point that fall within the at least one time window.

28. The non-transitory computer-readable medium of claim 26 , wherein forming the hypotheses of objects in motion in three-dimensional space when the number of registered detections for the given hypothesis is greater than or equal to the threshold value comprises dividing the given hypothesis into distinct segments comprising a flight segment, one or more bounce segments, and a roll segment, and the dividing comprises the identifying, the checking, the designating, and:

classifying a first segment of the given hypothesis before a first designated ground impact as the flight segment;

classifying a next segment of the given hypothesis after each next designated ground impact as one of the one or more bounce segments when an angle between a first estimated velocity vector before the next designated ground impact and a second estimated velocity vector after the next designated ground impact is greater than a threshold angle; and

classifying the next segment of the given hypothesis after a next designated ground impact as the roll segment when the angle between the first estimated velocity vector before the next designated ground impact and the second estimated velocity vector after the next designated ground impact is less than or equal to the threshold angle.

Assignments (1)

Continuity (3)

References Cited (53)

Cited By (1)

US 12,582,870