Sports neural network codec
A computing system receives a broadcast video stream of a game. A codec module of the computing system extracts image level features from the broadcast video stream. The codec module includes an object detection portion configured to detect players in the broadcast video stream and a subnet portion attached to the object detection portion. The subnet portion is configured to identify foreground information of the detected players. The codec module provides the image level features to a plurality of task specific modules for analysis. The plurality of task specific modules generates a plurality of outputs based on the image level features.
1. A method comprising:
receiving, by a computing system, a broadcast video stream of a game;
extracting, via a codec module of the computing system, image level features from the broadcast video stream, the codec module comprising an object detection portion that detects players in the broadcast video stream and a subnet portion attached to and downstream of the object detection portion, wherein the subnet portion identifies foreground information of the detected players;
providing, by the codec module, the image level features to a plurality of task specific modules for analysis; and
generating, by the plurality of task specific modules, a plurality of outputs based on the image level features.
2. The method of claim 1 , wherein the object detection portion comprises:
a backbone that extracts image level features from the broadcast video stream;
a neck downstream of the backbone, wherein the neck aggregates the extracted image level features; and
a head downstream of the neck, wherein the head identifies locations of players in the broadcast video stream based on the extracted image level features.
3. The method of claim 2 , wherein the head comprises a plurality of convolutions, wherein each convolution identifies a location of a player at varying resolutions.
4. The method of claim 3 , wherein the codec module further comprises:
a non-maximum suppression function downstream of the head, wherein the non-maximum suppression function combines the identified locations of the player at varying resolutions to generate a single location for the player.
5. The method of claim 2 , wherein the subnet portion is attached to the neck.
6. The method of claim 2 , wherein the subnet portion receives input from the neck, wherein the input from the neck is output generated by the neck, the output comprising floating point values indicating a likely position of players in the broadcast video stream.
7. The method of claim 1 , further comprising:
training, by the computing system, the object detection portion independent of the subnet portion; and
after training the object detection portion, training, by the computing system, the object detection portion with the subnet portion attached thereto.
8. A non-transitory computer readable medium comprising one or more sequences of instructions, which, when executed by a processor, causes a computing system to perform operations comprising:
receiving, by the computing system, a broadcast video stream of a game;
extracting, via a codec module of the computing system, image level features from the broadcast video stream, the codec module comprising an object detection portion that detects players in the broadcast video stream and a subnet portion attached to and downstream of the object detection portion, wherein the subnet portion identifies foreground information of the detected players;
providing, by the codec module, the image level features to a plurality of task specific modules for analysis; and
generating, by the plurality of task specific modules, a plurality of outputs based on the image level features.
9. The non-transitory computer readable medium of claim 8 , wherein the object detection portion comprises:
a backbone that extracts image level features from the broadcast video stream;
a neck downstream of the backbone, wherein the neck aggregates the extracted image level features; and
a head downstream of the neck, wherein the head identifies locations of players in the broadcast video stream based on the extracted image level features.
10. The non-transitory computer readable medium of claim 9 , wherein the head comprises a plurality of convolutions, wherein each convolution identifies a location of a player at varying resolutions.
11. The non-transitory computer readable medium of claim 10 , wherein the codec module further comprises:
a non-maximum suppression function downstream of the head, wherein the non-maximum suppression function combines the identified locations of the player at varying resolutions to generate a single location for the player.
12. The non-transitory computer readable medium of claim 9 , wherein the subnet portion is attached to the neck.
13. The non-transitory computer readable medium of claim 9 , wherein the subnet portion receives input from the neck, wherein the input from the neck is output generated by the neck, the output comprising floating point values indicating a likely position of players in the broadcast video stream.
14. The non-transitory computer readable medium of claim 8 , further comprising:
training, by the computing system, the object detection portion independent of the subnet portion; and
after training the object detection portion, training, by the computing system, the object detection portion with the subnet portion attached thereto.
15. A system comprising:
a processor; and
a memory having programming instructions stored thereon, which, when executed by the processor, cause the system to perform operations comprising:
receiving a broadcast video stream of a game;
extracting, via a codec module, image level features from the broadcast video stream, the codec module comprising an object detection portion that detects players in the broadcast video stream and a subnet portion attached to and downstream of the object detection portion, wherein the subnet portion identifies foreground information of the detected players;
providing, by the codec module, the image level features to a plurality of task specific modules for analysis; and
generating, by the plurality of task specific modules, a plurality of outputs based on the image level features.
16. The system of claim 15 , wherein the object detection portion comprises:
a backbone that extracts image level features from the broadcast video stream;
a neck downstream of the backbone, wherein the neck aggregates the extracted image level features; and
a head downstream of the neck, wherein the head identifies locations of players in the broadcast video stream based on the extracted image level features.
17. The system of claim 16 , wherein the head comprises a plurality of convolutions, wherein each convolution identifies a location of a player at varying resolutions.
18. The system of claim 17 , wherein the codec module further comprises:
a non-maximum suppression function downstream of the head, wherein the non-maximum suppression function combines the identified locations of the player at varying resolutions to generate a single location for the player.
19. The system of claim 16 , wherein the subnet portion receives input from the neck, wherein the input from the neck is output generated by the neck, the output comprising floating point values indicating a likely position of players in the broadcast video stream.
20. The system of claim 15 , further comprising:
training the object detection portion independent of the subnet portion; and
after training the object detection portion, training the object detection portion with the subnet portion attached thereto.