Granted Patent B2

US 12,725,424 · App. 18/015,609 · Granted Sep 1, 2026

Inventors: Stefan Schiffer (Munich, DE); Marcel Naujeck (Munich, DE)

Assignee: Fujitsu Germany GmbH

G06V20/54G06T7/0002G06T7/70G06V10/25G06V10/26G06V10/764G06V10/774G06V10/82G06V20/625G06T2207/10016G06T2207/30236G06T2207/30241G06V2201/08

Abstract

Claims (88)

1 . A method of monitoring a race track comprising:

obtaining at least one sequence of images from a track-side camera capturing at least one section of the race track;

segmenting images of the sequence of images into different areas associated with the race track;

using automatic object recognition to detect vehicles in the sequence of images;

once at least one vehicle has been detected, performing:

mapping the at least one detected vehicle to at least one of the different areas associated with the race track;

comparing a first image of the at least one sequence of images taken before a passage of the at least one detected vehicle with a second image of the at least one sequence of images taken after passage of the at least one detected vehicle to detect anomalies along the race track;

classifying a detected anomaly based on automatic pattern recognition;

mapping the detected anomaly to at least one of the different areas associated with the race track;

activating at least one warning device based on at least one set of rules, wherein the at least one set of rules comprises a first rule triggering a first warning if the at least one detected vehicle is mapped to a first predefined area of the race track, a crash barrier or an out-of-bounds area, and a second rule triggering a warning if the detected anomaly is mapped to a second predefined area of the race track;

re-identifying the at least one detected vehicle as a specific vehicle of a predetermined set of vehicles using embedding; and

mapping the re-identified vehicle to a corresponding digital twin in a digital representation of the race track,

wherein the predetermined set of vehicles corresponds to a subset of all vehicles having corresponding digital twins in the digital representation of the race track, and

the subset is selected based on a set of rules providing a likelihood of re-identifying a given vehicle in a sequence of images corresponding to the at least one section of the race track based on the data of the corresponding digital twin.

2 . The method of claim 1 , wherein the areas of the race track comprise at least one of: a driving surface, a lane, a track boundary, a crash barrier, a run-out area, an out-of-bounds area, a pit area, a viewing area, a tarmac area, a gravel area, a dirt area, a grassy area, and a forested area.

3 . The method of claim 1 , wherein

the detected anomaly is classified as acceptable, if it is classified as one or more rain drops, leaves, reflections, shadows, and/or light beams; and/or

the detected anomaly is classified as unacceptable, if it is classified as a vehicle part, oil, and/or gravel.

4 . The method of claim 1 , wherein the first image is a last image in a time sequence of images taken before a bounding box surrounding the at least one detected vehicle entered the section of the race track and the second image is a first image in the time sequence of images taken after the bounding box surrounding the at least one detected vehicle left the section of the race track.

5 . The method of claim 1 , further comprising:

locating a first position of the at least one detected vehicle;

locating a second position of the detected anomaly; and

displaying the first position and the second position on a visual representation of the race track.

6 . The method of claim 1 , further comprising:

computing at least one reference embedding vector for the embedding based on at least one image taken when at least one vehicle enters the race track or a monitored part of the race track.

7 . The method of claim 6 , further comprising:

extracting at least one characteristic feature, a number plate or other registration number of the at least one vehicle, from the at least one image taken when the at least one vehicle enters the race track, wherein the at least one characteristic feature is used in the step of re-identifying the at least one vehicle.

8 . The method of claim 1 , further comprising:

selecting a plurality of sequences of images from a plurality of cameras capturing different sections of the race track based on the re-identification of at least one specific vehicle in each one of the plurality of sequences; and

cutting the plurality of sequences to generate footage of the at least one specific vehicle driving along the race track.

9 . The method of claim 1 , further comprising:

determining a first real-world position of at least one re-identified vehicle based on a mapping relationship; and/or

determining a second real-world position of the at least one anomaly detected based on the mapping relationship, wherein

the mapping relationship maps a plurality of pixel areas in the images of the at least one sequence of images to a corresponding plurality of real-world positions of the corresponding section of the race track captured by the camera.

10 . The method of claim 9 , further comprising:

mapping the re-identified vehicle to a corresponding digital twin in a digital representation of the race track; and

adding first position and timestamp information to the corresponding digital twin each time a first real-world position of a re-identified vehicle is determined to store a trajectory of the respective vehicle in the digital representation of the race track.

11 . The method of claim 10 , further comprising:

adding second position and timestamp information to a corresponding digital representation of at least one unacceptable anomaly detected along the race track; and

correlating the first and second position and timestamp information by comparing the trajectories of re-identified vehicles with a position and a first occurrence of the detected anomaly in the sequence of images to identify a vehicle likely to have caused the at least one unacceptable anomaly.

12 . A monitoring system for a race track comprising:

one or more track-side cameras, each camera having a field of view covering at least one section of the race track;

an image capturing system configured to obtain at least one sequence of images from at least one of the track-side cameras;

one or more warning devices configured to be activated when a first warning and/or a second warning is triggered; and

an image processing system comprising at least one processor configured to:

segment images of the sequence of images into different areas associated with the race track;

use automatic object recognition to detect vehicles in the sequence of images;

map any detected vehicle to at least one of the different areas associated with the race track;

compare a first image of the at least one sequence of images taken before a passage of at least one detected vehicle with a second image of the at least one sequence of images taken after the passage of the at least one detected vehicle to detect anomalies along the race track;

classify a detected anomaly based on automatic pattern recognition;

map the detected anomaly to at least one of the different areas associated with the race track;

trigger the first warning if the at least one detected vehicle is mapped to a first predefined area of the race track, and/or trigger the second warning, if the detected anomaly is mapped to a second predefined area of the race track;

re-identifying the at least one detected vehicle as a specific vehicle of a predetermined set of vehicles using embedding; and

mapping the re-identified vehicle to a corresponding digital twin in a digital representation of the race track,

wherein the predetermined set of vehicles corresponds to a subset of all vehicles having corresponding digital twins in the digital representation of the race track, and

the subset is selected based on a set of rules providing a likelihood of re-identifying a given vehicle in a sequence of images corresponding to the at least one section of the race track based on the data of the corresponding digital twin.

13 . A method of monitoring a race track comprising:

obtaining at least one sequence of images from a camera capturing at least one section of the race track;

detecting at least one vehicle in the sequence of images using automatic object recognition;

re-identifying at least one detected vehicle as a specific vehicle of a predetermined set of vehicles using embedding, comprising computing at least one reference embedding vector for the embedding based on at least one image taken when the at least one detected vehicle entered the race track or the monitored part of the race track;

mapping the re-identified vehicle to a corresponding digital twin in a digital representation of the race track; and

showing an estimated position of the re-identified vehicle in the digital representation of the race track;

wherein the predetermined set of vehicles corresponds to a subset of all vehicles having corresponding digital twins in the digital representation of the race track, and

the subset is selected based on a third set of rules providing a likelihood of re-identifying a given vehicle in a sequence of images corresponding to the at least one section of the race track based on the data of the corresponding digital twin;

determining a first real-world position of at least one re-identified vehicle based on a mapping relationship; and

determining a second real-world position of the at least one anomaly detected along the race track based on a comparison of a first image of the at least one sequence of images taken before a passage of the at least one detected vehicle with a second image of the at least one sequence of images taken after the passage of the at least one detected vehicle based on the mapping relationship,

wherein the mapping relationship maps a plurality of pixel areas in the images of the at least one sequence of images to a corresponding plurality of real-world positions of the corresponding section of the race track captured by the camera.

14 . The method of claim 13 , further comprising:

segmenting images of the sequence of images into different areas associated with the race track;

mapping the at least one detected vehicle to at least one of the different areas associated with the race track; and

activating at least one warning device based on a first set of rules, wherein the first set of rules comprises at least one first rule triggering a first warning if the at least one detected vehicle is mapped to a first predefined area of the race track, a crash barrier or an out-of-bounds area.

15 . The method of claim 13 , wherein the predetermined set of vehicles comprises individual vehicles taking part in a race, and re-identification of the at least one vehicle is implemented using a neural network that has been trained offline or before the use of the monitoring system to detect the at least one vehicle in the sequence of images in the race, using an encoder/decoder model to identify specific vehicles from a given class of objects, Formula 1 cars, normal road cars, or motorcycles.

16 . The method of claim 15 , wherein training of the neural network is performed unsupervised, and comprises:

receiving images on an input side of the neural network and simplifying the received images by nodes of the neural network to form or encode an embedding vector;

decode information of the embedding vector to recreate an image on a decoder or output side of the neural network;

varying weights and other settings of the neural network until a difference between the images received on the input side and the recreated-images becomes very small or minimal, based on an automatic comparison of the input side and output side using a similarity metric.

17 . The method of claim 15 , further comprising:

in a training stage, providing a high number of training images of different vehicles to an encoder or input side of the neural network, including training images selected or confirmed manually and taken on the race track;

during an initial registration stage before the at least one vehicle enters the race track, taking and processing, by the previously trained neural network the least one image of at least one vehicle to compute the at least one reference embedding vector; and

during re-identification of the at least one vehicle once the at least one vehicle is on the race track, feeding parts of an image corresponding to the at least one detected vehicle to the neural network to determine a new embedding vector, comparing the new embedding vector with a set of previously registered embedding vectors, comprising the at least one reference embedding vector, and identifying the at least vehicle as the vehicle corresponding to the closest one of the set of previously registered embedding vectors.

18 . The method of claim 17 , wherein

if the new embedding vector differs from the closest previously registered embedding vectors by more than a first pre-set threshold value, the new embedding vector is stored in an array of vectors corresponding to a given vehicle; and/or

if the new embedding vector differs from each one of the previously registered embedding vectors by more than a second pre-set threshold value, failing the identification and/or not including the new embedding vector in the array of vectors.

19 . The method of claim 13 , further comprising:

extracting at least one characteristic feature, a number plate or other registration number of the at least one vehicle, from the at least one image taken when the at least one vehicle entered the race track or the monitored part of the race track, wherein the at least one characteristic feature is used in the step of re-identifying the at least one vehicle.

20 . The method of claim 13 , further comprising:

selecting a plurality of sequences of images from a plurality of cameras capturing different sections of the race track based on the re-identification of at least one specific vehicle in each one of the plurality of sequences; and

cutting the plurality of sequences to generate footage of the at least one specific vehicle driving along the race track.

Assignments (3)

Priority Claims (2)

Continuity (1)

References Cited (31)