Systems and methods for the presentation of graphical overlays
Systems and methods for the presentation of visual overlays are provided. Particularly, the systems and methods described herein provide for time synchronization between different data sources associated with a live broadcast of an event (such as the video and/or audio feed of the live event and metadata including data captured by sensors included at the live event). The systems and methods also provide for player identification and tracking within the video feed such that the information in the metadata may be correlated to the players and events shown in the video feed. Performing this time synchronization and player identification and tracking, allows for graphical overlays including relevant information to be displayed over the live broadcast of the event. For example, in an American football game, information about players or plays occurring on the field may be displayed in real-time.
1 . A method comprising:
receiving first data associated with a live event, the first data including sensor data obtained from one or more sensors at a live sporting event, wherein the one or more sensors are located at least one of: player equipment, game equipment, and a playing field;
receiving second data associated with the live event, the second data including one or more video frames depicting the live event;
determining, using a first machine learning model and a first temporal window, a first probability that a first event associated with a play of the live sporting event is occurring within a first video frame of the one or more video frames, the first temporal window including a first number of video frames located sequentially before and after the first video frame within the one or more video frames;
determining, using a second machine learning model and a second temporal window, a second probability that the first event associated with the play of the live sporting event is occurring within the first video frame, the second temporal window including a second number of video frames located sequentially before and after the first video frame within the one or more video frames;
determining a third probability that the first event is occurring within the first video frame based on the first probability and the second probability;
determining, based on the third probability surpassing a threshold probability, that the first event is occurring within the first video frame;
determining a time stamp associated with the first event within the first data;
performing a time synchronization between the first data and the second data based on the time stamp; and
presenting, using the first data, a graphical overlay within a live broadcast of the live sporting event.
2 . The method of claim 1 , further comprising:
determining, using a third machine learning model and a third temporal window, a fourth probability that the first event associated with the play of the live sporting event is occurring within the first video frame, the third temporal window including a third number of video frames located sequentially before and after the first video frame within the one or more video frames, wherein determining the third probability is further based on the fourth probability.
3 . The method of claim 1 , wherein performing the time synchronization is further based on a first frequency at which the one or more video frames are presented and a second frequency at which the first data is received.
4 . The method of claim 1 , wherein the first machine learning model is a deep learning action recognition model that is trained using prior video frames in which the first event is occurring.
5 . A method comprising:
receiving first data associated with a live event, the first data being metadata including information about the live event;
receiving second data associated with the live event, the second data including one or more video frames depicting the live event;
determining, using a first computing model and using a first temporal window, that a first event is occurring within a first video frame of the one or more video frames comprising:
determining a first probability that the first event is occurring within the first video frame;
determining that the first probability is greater than a threshold probability;
determining, using a second computing model and a second temporal window, a second probability that the first event is occurring within the first video frame; and
determining a third probability that the first event is occurring within the first video frame based on the first probability and the second probability, wherein determining that the first event is occurring within the first video frame is based on the third probability;
determining a time stamp associated with the first event within the first data; and
performing a time synchronization between the first data and the second data based on the time stamp.
6 . The method of claim 5 , wherein the first temporal window including a first number of video frames located sequentially before and after the first video frame within the one or more video frames.
7 . The method of claim 5 , further comprising:
presenting, using the first data, a graphical overlay within a broadcast of the live event.
8 . The method of claim 5 , wherein performing the time synchronization is further based on a first frequency at which the one or more video frames are presented and a second frequency at which the first data is received.
9 . The method of claim 5 , wherein the live event is a live sporting event, wherein the metadata includes sensor data captured from one or more sensors, and wherein the one or more sensors are located at least one of: player equipment, game equipment, and a playing field.
10 . The method of claim 5 , wherein the first computing model is a deep learning action recognition model that is trained using prior video frames in which the first event is occurring.
11 . A system comprising:
memory that stores computer-executable instructions; and
one or more processors configured to access the memory and execute the computer-executable instructions to:
receive first data associated with a live event, the first data being metadata including information about the live event;
receive second data associated with the live event, the second data including one or more video frames depicting the live event;
determine, using a first computing model, that a first event is occurring within a first video frame of the one or more video frames comprising:
determine a first probability that the first event is occurring within the first video frame;
determine that the first probability is greater than a threshold probability;
determine, using a second computing model and a second temporal window, a second probability that the first event is occurring within the first video frame; and
determine a third probability that the first event is occurring within the first video frame based on the first probability and the second probability, wherein determining that the first event is occurring within the first video frame is based on the third probability;
determine a time stamp associated with the first event within the first data; and
perform a time synchronization between the first data and the second data based on the time stamp.
12 . The system of claim 11 , wherein the first temporal window including a first number of video frames located sequentially before and after the first video frame within the one or more video frames.
13 . The system of claim 11 , further comprising:
presenting, using the first data, a graphical overlay within a broadcast of the live event.
14 . The system of claim 11 , wherein performing the time synchronization is further based on a first frequency at which the one or more video frames are presented and a second frequency at which the first data is received.
15 . The system of claim 11 , wherein the live event is a live sporting event, wherein the metadata includes sensor data captured from one or more sensors, and wherein the one or more sensors are located at least one of: player equipment, game equipment, and a playing field.
16 . The system of claim 11 , wherein the first computing model is a deep learning action recognition model that is trained using prior video frames in which the first event is occurring.