Method and apparatus for determining a frontal body orientation

1. A method for determining a three-dimensional (3D) orientation of a frontal body of a player, comprising:

detecting each of a plurality of players in each of a plurality of frames captured by a plurality of cameras, comprising:

for each of the plurality of frames:

detecting each of the plurality of players in the frame; and

determining a two-dimensional (2D) orientation of each of the plurality of players in the frame based on a machine learning model, wherein the machine learning model is trained based on construed 3D positions obtained from a plurality of 2D key points annotated on players in frames captured by the plurality of cameras;

for each of the plurality of cameras, tracking each of the plurality of players between continuous frames captured by the camera; and

associating the plurality of frames captured by the plurality of cameras to generate the 3D orientation of each of the plurality of players.

2. The method of claim 1 , wherein for a player of the plurality of players, associating the plurality of frames captured by the plurality of cameras comprises:

associating the plurality of frames to associate the player from the plurality of frames;

generating a 3D position of the player; and

generating a 3D orientation of the player based on the 2D orientation of the player in each of the plurality of frames.

3. The method of claim 2 , wherein the generating of the 3D orientation of the player comprises:

for each of the plurality of frames:

calculating a 3D orientation of the camera that captures the frame based on the 3D position of the player and a 3D position of the camera, and

combining the 3D orientation of the camera with the 2D orientation of the player in the frame to generate the 3D orientation of the player for the frame; and

combining the 3D orientations of the player for each of the plurality of frames to generate the 3D orientation of the player for the plurality of frames.

4. The method of claim 3 , wherein the generating of the 3D orientation of the player for the plurality of cameras further comprises at least one of:

replacing an outlier 3D orientation of the player for the plurality of frames among a plurality of 3D orientations of the player for continuous frames with a linear interpolation of 3D orientations of the player for neighboring frames; and

applying different weights to a plurality of 3D orientations of the player for continuous frames to generate an optimized 3D orientation of the player for the middle frame of the continuous frames.

5. The method of claim 1 , wherein training data for the machine learning model is generated by:

annotating three 2D key points on a player in a frame;

constructing a 3D position of each of the three 2D key points;

generating a 3D orientation of the player based on the 3D positions of the three 2D key points; and

generating a 2D orientation of the player in the frame based on the 3D orientation of the player.

6. The method of claim 1 , further comprising putting a virtual camera in the frontal body of a player to render a video from the player's view using the 3D orientation of the player.

7. The method of claim 1 , further comprising:

associating the plurality of frames captured by the plurality of cameras to generate tracking information of the plurality of players;

detecting a ball in each of the plurality of frames;

for each of the plurality of cameras, tracking the ball between continuous frames captured by the camera;

associating the plurality of frames captured by the plurality of cameras to generate a 3D position of the ball; and

determining a ball carrier based on the tracking information of the plurality of players and the 3D position of the ball.

8. The method of claim 7 , wherein the detecting of each of the plurality of players comprises:

for each of the plurality of frames:

detecting each of the plurality of players in the frame; and

recognizing a jersey number and a team class of each of the plurality of players.

9. The method of claim 8 , wherein the associating of the plurality of frames captured by the plurality of cameras to generate tracking information of the plurality of players comprises, for a player:

associating the plurality of frames to associate the player from the plurality of frames;

generating a 3D position of the player; and

calculating a velocity and a moving orientation of the player based on the 3D position of the player.

10. The method of claim 9 , wherein the determining of the ball carrier based on the tracking information of the plurality of players and the 3D position of the ball comprises:

building a graph including a plurality of nodes that correspond to the plurality of players and edges connecting the plurality of players for each frame, wherein each node is identified with a jersey number and a team class;

extracting a feature vector for each player based on the tracking information of the plurality of players and the 3D position of the ball;

determining a single frame confidence for each of the plurality of players based on a node classification model with the graph and the feature vector for each node in the graph; and

calculating a multiple frame confidence for a player based on single frame confidences in multiple frames, wherein a player with a maximum multiple frame confidence is determined as the ball carrier.

11. The method of claim 10 , wherein training data for the node classification model is generated by:

identifying, for each of the plurality of players in each of the plurality of frames, a corresponding bounding box, a corresponding jersey number, a corresponding team class and a corresponding confidence;

associating the plurality of frames for each corresponding bounding box by the corresponding jersey number and the corresponding team class; calculating a corresponding 3D position of each player; and

calculating the velocity and the moving orientation of each player based on the corresponding 3D position of the player.

12. A computing system for determining a three-dimensional (3D) orientation of a frontal body of a player, comprising:

a processor to:

detect each of a plurality of players in each of a plurality of frames captured by a plurality of cameras, comprising:

for each of the plurality of frames, to detect each of the plurality of players in the frame, and determine a two-dimensional (2D) orientation of each of the plurality of players in the frame based on a machine learning model, wherein the machine learning model is trained based on construed 3D positions obtained from a plurality of 2D key points annotated on players in frames captured by the plurality of cameras;

for each of the plurality of cameras, track each of the plurality of players between continuous frames captured by the camera; and

associate the plurality of frames captured by the plurality of cameras to generate the 3D orientation of each of the plurality of players.

13. The computing system of claim 12 , wherein for a player of the plurality of players, associating the plurality of frames captured by the plurality of cameras comprises:

associating the plurality of frames to associate the player from the plurality of frames;

generating a 3D position of the player; and

generating a 3D orientation of the player based on the 2D orientation of the player in each of the plurality of frames.

14. The computing system of claim 13 , wherein the generating of the 3D orientation of the player comprises:

for each of the plurality of frames:

calculating a 3D orientation of the camera that captures the frame based on the 3D position of the player and a 3D position of the camera, and

combining the 3D orientation of the camera with the 2D orientation of the player in the frame to generate the 3D orientation of the player for the frame; and

combining the 3D orientations of the player for each of the plurality of frames to generate the 3D orientation of the player for the plurality of frames.

15. The computing system of claim 14 , wherein the generating of the 3D orientation of the player for the plurality of cameras further comprises at least one of:

replacing an outlier 3D orientation of the player for the plurality of frames among a plurality of 3D orientations of the player for continuous frames with a linear interpolation of 3D orientations of the player for neighboring frames; and

applying different weights to a plurality of 3D orientations of the player for continuous frames to generate an optimized 3D orientation of the player for the middle frame of the continuous frames.

16. The computing system of claim 12 , wherein training data for the machine learning model is generated by:

annotating three 2D key points on a player in a frame;

constructing a 3D position of each of the three 2D key points;

generating a 3D orientation of the player based on the 3D positions of the three 2D key points; and

generating a 2D orientation of the player in the frame based on the 3D orientation of the player.

17. The computing system of claim 12 , wherein the processor is further to put a virtual camera in the frontal body of a player to render a video from the player's view using the 3D orientation of the player.

18. The computing system of claim 12 , wherein the processor is further to:

associate the plurality of frames captured by the plurality of cameras to generate tracking information of the plurality of players;

detect a ball in each of the plurality of frames;

for each of the plurality of cameras, track the ball between continuous frames captured by the camera;

associate the plurality of frames captured by the plurality of cameras to generate a 3D position of the ball; and

determine a ball carrier based on the tracking information of the plurality of players and the 3D position of the ball.

19. The computing system of claim 18 , wherein the detecting of each of the plurality of players comprises:

for each of the plurality of frames:

detecting each of the plurality of players in the frame; and

recognizing a jersey number and a team class of each of the plurality of players.

20. The computing system of claim 19 , wherein the associating of the plurality of frames captured by the plurality of cameras to generate tracking information of the plurality of players comprises, for a player:

associating the plurality of frames to associate the player from the plurality of frames;

generating a 3D position of the player; and

calculating a velocity and a moving orientation of the player based on the 3D position of the player.

21. The computing system of claim 20 , wherein the determining of the ball carrier based on the tracking information of the plurality of players and the 3D position of the ball comprises:

building a graph including a plurality of nodes that correspond to the plurality of players and edges connecting the plurality of players for each frame, wherein each node is identified with a jersey number and a team class;

extracting a feature vector for each player based on the tracking information of the plurality of players and the 3D position of the ball;

determining a single frame confidence for each of the plurality of players based on a node classification model with the graph and the feature vector for each node in the graph; and

calculating a multiple frame confidence for a player based on single frame confidences in multiple frames, wherein a player with a maximum multiple frame confidence is determined as the ball carrier.