Methods and systems for generating sports analytics with a mobile device
Methods and systems for real-time generation of ball shot analytics are disclosed. The methods and systems perform steps for ball and posture detection, ball and posture flow generation, shot event identification and classification, and shot analytics generation based on identified shot events and shooter posture flow. Embodiments of the present invention use computer vision techniques to enable a resource-limited mobile device such as a smartphone to conduct the aforementioned steps. Therefore, the present invention may be implemented using a processor on a single mobile computing device. Also disclosed are benefits of the new methods, and alternative embodiments of implementation.
1 . A method for generating ball shot analytics using a hardware processor on a single mobile computing device, comprising:
receiving, on the hardware processor on the single mobile computing device, an input video of a ball game, and a location of a shooter in a shooter identification frame of the input video;
detecting, on the hardware processor on the single mobile computing device, one or more balls and one or more player postures from the input video;
generating, on the hardware processor on the single mobile computing device, one or more ball flows and one or more posture flows by grouping the one or more balls and the one or more player postures along a time line;
identifying, on the hardware processor on the single mobile computing device, a generated player posture flow as a shooter posture flow, based on an input location of the shooter;
identifying, on the hardware processor on the single mobile computing device, a generated ball flow as a related ball flow to the shooter posture flow;
determining, on the hardware processor on the single mobile computing device, a ball-from-shooter time by backtracking the related ball flow from a shot attempt;
determining, on the hardware processor on the single mobile computing device, a shot event occurring before the ball-from-shooter time; and
generating, on the hardware processor on the single mobile computing device, one or more shot analytics based on the shot event, the shooter posture flow, and the related ball flow.
2 . The method of claim 1 , wherein the detecting of the one or more balls and the one or more player postures from the input video comprises using one or more Convolutional Neural Network (CNN) modules.
3 . The method of claim 1 , wherein the generating of the one or more shot analytics comprises determining a shot type of the shot attempt.
4 . The method of claim 1 , wherein the ball game is a basketball game.
5 . The method of claim 1 , wherein the input video is streamed.
6 . The method of claim 1 , wherein the detecting the one or more balls and the one or more player postures is applied on a skip frame basis.
7 . The method of claim 1 , wherein the detecting the one or more balls and the one or more player postures is limited to an image area close to the shooter.
8 . The method of claim 1 , wherein the generating the one or more ball flows and the one or more posture flows comprises applying a bipartite matching to the one or more balls and the one or more player postures, respectively, to existing ball flows and posture flows, wherein the bipartite matching of a detected ball and an existing ball flow comprises computing a matching score between the detected ball and the existing ball flow, and wherein the computing of the matching score comprises:
generating a predicted ball comprising a next ball location and a next ball size based on the existing ball flow; and
computing the matching score based on a location difference and a size difference between the predicted ball and the detected ball.
9 . The method of claim 1 , wherein the shooter posture flow is closest to the input location of the shooter in the shooter identification frame when compared to other posture flows.
10 . The method of claim 1 ,
wherein identifying the related ball flow comprises applying non-max-suppression to all generated ball flows,
wherein the related ball flow has a score against the shooter posture flow,
wherein the score is computed based on shooter movements, a distance to the shooter, and a confidence value, and
wherein the score is above a pre-defined threshold.
11 . The method of claim 1 , further comprising declaring the shot attempt by determining whether the related ball flow is thrown from an upper body of the shooter upward.
12 . The method of claim 1 , wherein the shot event occurs within a pre-defined time period before the ball-from-shooter time.
13 . The method of claim 1 , wherein the shot event is selected from the group consisting of dribble event, jump event, catch-ball event, ball-leave-hand event, one-two leg jump, and wherein a shot type is selected from the group consisting of layup, regular shot, dribble-pull-up, off-the-move, and catch-and-shoot.
14 . The method of claim 1 , wherein the shot analytics is selected from the group consisting of release time, back angle, leg bend ratio, leg power, moving speed, moving direction, and height of jump.
15 . The method of claim 2 , wherein each CNN module has been trained using one or more prior input videos.
16 . The method of claim 1 , wherein the input video is received from a camera on the single mobile computing device.
17 . A single mobile computing device for generating ball shot analytics, comprising:
a processor on the single mobile computing device; and
a non-transitory computer readable storage medium for storing program code and accessible by the processor, the program code when executed by the processor causes the processor to:
receive, on the processor on the single mobile computing device, an input video of a ball game, and a location of a shooter in a shooter identification frame of the input video;
detect, on the processor on the single mobile computing device, one or more balls and one or more player postures from the input video;
generate, on the processor on the single mobile computing device, one or more ball flows and one or more posture flows by grouping the one or more balls and the one or more player postures along a time line;
identify, on the processor on the single mobile computing device, a generated player posture flow as a shooter posture flow, based on an input location of the shooter;
identify, on the processor on the single mobile computing device, a generated ball flow as a related ball flow to the shooter posture flow;
determine, on the processor on the single mobile computing device, a ball-from-shooter time by backtracking the related ball flow from a shot attempt;
determine, on the processor on the single mobile computing device, a shot event occurring before the ball-from-shooter time; and
generate, on the processor on the single mobile computing device, one or more shot analytics based on the shot event, the shooter posture flow, and the related ball flow.
18 . The single mobile computing device of claim 17 , wherein the program code to detect the one or more balls and the one or more player postures from the input video comprises program code for one or more Convolutional Neural Network (CNN) modules.
19 . A non-transitory computer readable storage medium for generating ball shot analytics, the non-transitory computer readable storage medium comprising program code stored thereon, the program code when executed by a processor on a single mobile computing device causes the processor to:
receive, on the processor on the single mobile computing device, an input video of a ball game, and a location of a shooter in a shooter identification frame of the input video;
detect, on the processor on the single mobile computing device, one or more balls and one or more player postures from the input video;
generate, on the processor on the single mobile computing device, one or more ball flows and one or more posture flows by grouping the one or more balls and the one or more player postures along a time line;
identify, on the processor on the single mobile computing device, a generated player posture flow as a shooter posture flow, based on an input location of the shooter;
identify, on the processor on the single mobile computing device, a generated ball flow as a related ball flow to the shooter posture flow;
determine, on the processor on the single mobile computing device, a ball-from-shooter time by backtracking the related ball flow from a shot attempt;
determine, on the processor on the single mobile computing device, a shot event occurring before the ball-from-shooter time; and
generate, on the processor on the single mobile computing device, one or more shot analytics based on the shot event, the shooter posture flow, and the related ball flow.
20 . The non-transitory computer readable storage medium of claim 19 , wherein the program code to detect the one or more balls and the one or more player postures from the input video comprises program code for one or more Convolutional Neural Network (CNN) modules.