Systems and methods for the presentation of graphical overlays
Systems and methods for the presentation of visual overlays are provided. Particularly, the systems and methods described herein provide for time synchronization between different data sources associated with a live broadcast of an event (such as the video and/or audio feed of the live event and metadata including data captured by sensors included at the live event). The systems and methods also provide for player identification and tracking within the video feed such that the information in the metadata may be correlated to the players and events shown in the video feed. Performing this time synchronization and player identification and tracking, allows for graphical overlays including relevant information to be displayed over the live broadcast of the event. For example, in an American football game, information about players or plays occurring on the field may be displayed in real-time.
1 . A method comprising:
receiving sensor data associated with a live sporting event, the sensor data indicating a position of a first player on a field at the live sporting event;
receiving, from a video capture device, a video stream of the live sporting event, the video stream including a first video frame;
identifying, using a machine learning model, a first location of the first player within the first video frame;
mapping the sensor data to a second location within the first video frame using a first type of homography and to a third location within the first video frame using a second type of homography;
determining a first distance between the first location and the second location;
determining a second distance between the first location and the third location; and
determining that the first distance is less than the second distance;
determining, using the second location and based on the determination that the first distance is less than the second distance, that the first player in the first video frame is associated with the sensor data; and
presenting, using the sensor data, a graphical overlay within a broadcast of the live sporting event.
2 . The method of claim 1 , further comprising:
determining, using the machine learning model, that a first event is occurring within the first video frame.
3 . The method of claim 2 , wherein determining that a first event is occurring within the first video frame is further based on a first temporal window, the first temporal window including a first number of video frames located sequentially before and after the first video frame within the one or more video frames.
4 . A method comprising:
receiving sensor data associated with a live event, the sensor data indicating a position of a first person at the live event;
receiving, from a video capture device, a first video frame of the live event;
identifying, using a computing model, a first location of the first person within the first video frame;
mapping the sensor data to a second location within the first video frame using a first type of homography and to a third location within the first video frame using a second type of homography;
determining a first distance between the first location and the second location;
determining a second distance between the first location and the third location; and
determining that the first distance is less than the second distance; and
determining, using the second location and based on the determination that the first distance is less than the second distance, that the first person in the first video frame is associated with the sensor data.
5 . The method of claim 4 , further comprising:
presenting, using the sensor data, a graphical overlay within a broadcast of the live event.
6 . The method of claim 4 , further comprising:
determining, using the computing model, a first probability that a first event is occurring within the first video frame.
7 . The method of claim 6 , wherein determining that a first event is occurring within the first video frame is further based on a first temporal window, the first temporal window including a first number of video frames located sequentially before and after the first video frame within the one or more video frames.
8 . The method of claim 7 , wherein determining that a first event is occurring within a first video frame of the one or more video frames further comprises:
determining, using a second computing model and a second temporal window, a second probability that the first event is occurring within the first video frame; and
determining a third probability that the first event is occurring within the first video frame based on the first probability and the second probability, wherein determining that the first event is occurring within the first video frame is based on the third probability.
9 . The method of claim 4 , wherein the live event is a live sporting event, wherein the sensor data is received from one or more sensors at the live event, and wherein the one or more sensors are located at least one of: player equipment, game equipment, and a playing field, and wherein the method further comprises tracking a location of a helmet worn by the first person in the first video frame.
10 . The method of claim 4 , wherein the first type of homography is performed using a location within the first video frame.
11 . A system comprising:
memory that stores computer-executable instructions; and
one or more processors configured to access the memory and execute the computer-executable instructions to:
receive sensor data associated with a live event, the sensor data indicating a position of a first person at the live event;
receive, from a video capture device, a first video frame of the live event;
identify, using a computing model, a first location of the first person within the first video frame;
map the sensor data to a second location within the first video frame using a first type of homography and to a third location within the first video frame using a second type of homography; and
determine a first distance between the first location and the second location;
determine a second distance between the first location and the third location; and
determine that the first distance is less than the second distance; and
determine, using the second location and based on the determination that the first distance is less than the second distance, based on a distance between the first location and the second location, that the first person in the first video frame is associated with the sensor data.
12 . The system of claim 11 , wherein the one or more processors are further configured to execute the computer-executable instructions to:
present, using the sensor data, a graphical overlay within a broadcast of the live event.
13 . The system of claim 11 , wherein the one or more processors are further configured to execute the computer-executable instructions to:
determine, using the computing model, a first probability that a first event is occurring within the first video frame.
14 . The system of claim 13 , wherein determining that a first event is occurring within the first video frame is further based on a first temporal window, the first temporal window including a first number of video frames located sequentially before and after the first video frame within the one or more video frames.
15 . The system of claim 14 , wherein determining that a first event is occurring within a first video frame of the one or more video frames further comprises:
determine, using a second computing model and a second temporal window, a second probability that the first event is occurring within the first video frame; and
determine a third probability that the first event is occurring within the first video frame based on the first probability and the second probability, wherein determining that the first event is occurring within the first video frame is based on the third probability.
16 . The system of claim 11 , wherein the live event is a live sporting event, wherein the sensor data is received from a sensor at the live event, and wherein the sensor is located at least one of: player equipment, game equipment, and a playing field, and wherein the method further comprises tracking a location of a helmet worn by the first person in the first video frame.
17 . The system of claim 11 , wherein the first type of homography is performed using a location within the first video frame.