Systems and methods to account for latency associated with remote driving applications
Systems and methods to account for latency associated with remote driving applications may include a vehicle having an imaging device and a teleoperator station in communication with each other via a network. Imaging data that is captured by the imaging device may be transmitted to the teleoperator station for presentation to a teleoperator. In order to account for latency in the transmission, receipt, processing, and presentation of the imaging data, one or more visualizations of the vehicle, with various visual characteristics, may be rendered within or overlaid onto the imaging data, in order to facilitate safe and reliable remote operation of the vehicle by the teleoperator at the teleoperator station.
1 . A method to account for video latency in a remote driving system, comprising:
receiving, by a processor at a teleoperator station via a communication network, video data from an imaging device associated with a vehicle, the vehicle being positioned within an environment remote from the teleoperator station;
receiving, by the processor via the communication network, drive state information associated with the vehicle;
determining, by the processor, a video latency associated with the video data;
determining, by the processor, a position of a visualization of the vehicle within the video data based on the video latency and the drive state information;
determining, by the processor, a first characteristic associated with the visualization based on the video latency, wherein the first characteristic comprises at least one of a shape, size, color, highlighting, outlining, sharpness, focus, or resolution associated with the visualization of the vehicle;
determining, by the processor, an uncertainty associated with the position of the visualization based on the drive state information;
determining, by the processor, a second characteristic associated with the visualization based on the uncertainty;
generating, by the processor, the visualization of the vehicle within the video data based on the position, the first characteristic, and the second characteristic associated with the visualization; and
presenting, by the processor via a presentation device of the teleoperator station, the visualization of the vehicle within the video data.
2 . The method of claim 1 , wherein the drive state information comprises at least one of a speed, an acceleration, or a steering angle associated with the vehicle.
3 . The method of claim 1 , wherein the video latency is between approximately Oms and approximately 200 ms.
4 . The method of claim 1 , wherein the second characteristic associated with the visualization comprises at least one of a size, sharpness, focus, resolution, a bounding box, or a maximum envelope associated with the visualization of the vehicle.
5 . A method, comprising:
receiving, by a processor associated with a teleoperator station via a communication network, imaging data from an imaging device associated with a vehicle, the vehicle being positioned within an environment remote from the teleoperator station;
receiving, by the processor via the communication network, drive state information associated with the vehicle;
determining, by the processor, a latency associated with the imaging data, the latency comprising video latency and control latency, and the control latency based at least in part on a time of execution of a drive control command by the vehicle;
determining, by the processor, a position of a visualization of the vehicle within the imaging data based on the latency and the drive state information;
determining, by the processor, a first characteristic associated with the visualization based on the latency, wherein the first characteristic comprises at least one of a shape, size, color, highlighting, outlining, sharpness, focus, or resolution associated with the visualization of the vehicle; and
generating, by the processor, the visualization of the vehicle for presentation within the imaging data based on the position of the visualization and the first characteristic associated with the visualization.
6 . The method of claim 5 , wherein the imaging data comprises video data associated with the environment around the vehicle.
7 . The method of claim 5 , wherein the drive state information comprises at least one of a speed, an acceleration, or a steering angle associated with the vehicle.
8 . The method of claim 5 , wherein determining the latency associated with the imaging data comprises:
decoding, by the processor, a timestamp embedded within the imaging data;
comparing, by the processor, the timestamp with a current time; and
determining the latency based on a difference between the timestamp and the current time.
9 . The method of claim 5 , wherein the visualization of the vehicle comprises a representation of at least a portion of the vehicle within the imaging data.
10 . The method of claim 5 , further comprising:
determining, by the processor, an uncertainty associated with the position of the visualization based on the drive state information; and
determining, by the processor, a second characteristic associated with the visualization based on the uncertainty;
wherein the second characteristic comprises at least one of a size, sharpness, focus, resolution, a bounding box, or a maximum envelope associated with the visualization of the vehicle.
11 . The method of claim 5 , further comprising:
presenting, by the processor via a presentation device of the teleoperator station, the visualization of the vehicle within the imaging data.
12 . The method of claim 5 , further comprising:
comparing, by the processor, the video latency with a threshold value for video latency;
determining, by the processor, that the video latency is greater than the threshold value for video latency; and
causing, by the processor, the vehicle to slow or stop based on the video latency;
wherein the threshold value for video latency is between approximately 0 ms and 300 ms.
13 . The method of claim 5 , further comprising:
comparing, by the processor, an end-to-end latency with a threshold value for end-to-end latency, the end-to-end latency including the video latency and the control latency;
determining, by the processor, that the end-to-end latency is greater than the threshold value for end-to-end latency; and
causing, by the processor, the vehicle to slow or stop based on the end-to-end latency;
wherein the threshold value for end-to-end latency is between approximately 300 ms and 400 ms.
14 . The method of claim 5 , wherein the time of execution of the drive control command is determined based on a timestamp associated with a portion of the drive state information that corresponds to the drive control command.
15 . The method of claim 5 , wherein the time of execution of the drive control command is determined based on processing of video data that includes a representation of execution of the drive control command by the vehicle.
16 . The method of claim 5 , wherein the control latency is determined based on a comparison between the time of execution of the drive control command and at least one of a transmission time or a reception time of the drive control command from the teleoperator station.
17 . A remote driving system, comprising:
a vehicle within an environment, the vehicle comprising an imaging device; and
a teleoperator station that is remote from the vehicle, the teleoperator station in communication with the vehicle via a communication network, the teleoperator station comprising a control interface, a presentation device, and a processor;
wherein the processor is configured to at least:
receive imaging data from the imaging device associated with the vehicle;
receive drive state information associated with the vehicle;
determine a latency associated with the imaging data, the latency comprising video latency and control latency, and the control latency based at least in part on a time of execution of a drive control command by the vehicle;
determine a position of a visualization of the vehicle within the imaging data based on the latency and the drive state information;
determine a first visual characteristic associated with the visualization based on the latency, the first visual characteristic comprising at least one of a shape, size, color, highlighting, outlining, sharpness, focus, or resolution; and
generate the visualization of the vehicle for presentation within the imaging data based on the position of the visualization and the first visual characteristic associated with the visualization.
18 . The remote driving system of claim 17 , wherein the processor is further configured to at least:
present, via the presentation device, the visualization of the vehicle within the imaging data.
19 . The remote driving system of claim 17 , wherein the visualization of the vehicle comprises a representation of at least a portion of the vehicle that is presented via the presentation device within the imaging data.
20 . The remote driving system of claim 17 , wherein the processor is further configured to at least:
determine a second visual characteristic associated with the visualization based on the drive state information, the second visual characteristic comprising at least one of a size, sharpness, focus, resolution, a bounding box, or a maximum envelope; and
wherein the visualization of the vehicle comprises the second visual characteristic that is presented via the presentation device within the imaging data.