LOW LATENCY WIRELESS COMMUNICATION SYSTEM FOR TELEOPERATED VEHICLE ENVIRONMENTS
A vehicle remote support system includes a communication system that operates over a plurality of parallel wireless network connections to provide low-latency video from vehicle to a remote support server that provides remote support to the vehicle dependent on real-time video. The vehicle includes a source that encodes multiple versions of the original video segments (e.g., one per wireless network connection) and transmits the multiple versions of the segments to a sink at the remote support server over the respective wireless connections. This redundant multi-path communication system rationally allocates network resources to the managed video streams and balances bandwidth against latency in order to avoid network congestion and safety issues associated with single-path transmissions. In other embodiments, a similar communication system that transmits video or other real-time messages between a source and a sink may be utilized in cloud robotics applications.
1 . A method for wirelessly receiving video at a remote support server supporting a vehicle during a remote support session, the method comprising:
establishing the remote support session with the vehicle;
receiving at a video processor, a first encoded video segment from a first wireless network connection;
determining a first video performance indicator associated with the first encoded video segment;
receiving at the video processor from a second wireless network connection different than the first wireless network connection, a second encoded video segment that represents redundant underlying video content as the first encoded video segment;
determining a second video performance indicator associated with second encoded video segment; and
determining a selected video segment selected between the first encoded video segment and the second encoded video segment based on the first and second video performance indicators; and
outputting the selected video segment to a video device.
2 . The method of claim 1 , wherein the first and second video performance indicators comprise respective quality metrics associated with the first and second video segments.
3 . The method of claim 1 , wherein determining the selected video segment based on the first and second video performance indicators comprises:
determining a threshold switching video performance indicator as a function of the first video performance indicator; and
selecting the second video segment as the selected video segment when the second video performance indicator associated with the second video segment exceeds the threshold switching video performance indicator.
4 . The method of claim 1 , wherein determining the selected video segment based on the first and second video performance indicators comprises:
performing a detection indicating that the first video segment comprises a key frame and the second video segment comprises a non-key frame; and
selecting the first video segment in response to the detection.
5 . The method of claim 1 , wherein the first wireless network connection comprises a first cellular network connection to a first cellular network operated by a first wireless carrier, and wherein the second wireless network connection comprises a second cellular network connection to a second cellular network operated by a second wireless carrier different than the first wireless carrier.
6 . The method of claim 1 , wherein the first wireless network connection comprises a first WiFi network connection and the second wireless network connection comprises a second WiFi network connection.
7 . The method of claim 1 , wherein the first wireless network connection comprises a cellular network connection to a cellular network and the second wireless network connection comprises a WiFi network connection to a WiFi network.
8 . The method of claim 1 , wherein the first wireless network connection operates according to a first communication protocol and wherein the second wireless network connection operates according to a second communication protocol different than the first communication protocol.
9 . The method of claim 1 , wherein the first wireless network connection is facilitated by a first wireless network includes first networking infrastructure and wherein the second wireless network connection is facilitated by a second wireless network including second networking infrastructure different than the first networking infrastructure.
10 . The method of claim 1 , wherein communications over the first wireless network connection comprise different performance parameters than communications over the second wireless network connection.
11 . The method of claim 1 , wherein the second encoded video segment is encoded according to a different encoding format or encoding parameters than the first encoded video segment.
12 . A non-transitory computer-readable storage medium storing instructions for wirelessly receiving video at a remote support server supporting a vehicle during a remote support session, the instructions when executed by one or more processors causing the one or more processors to perform steps comprising:
establishing the remote support session with the vehicle;
receiving at a video processor, a first encoded video segment from a first wireless network connection;
determining a first video performance indicator associated with the first encoded video segment;
receiving at the video processor from a second wireless network connection different than the first wireless network connection, a second encoded video segment that represents redundant underlying video content as the first encoded video segment;
determining a second video performance indicator associated with second encoded video segment; and
determining a selected video segment selected between the first encoded video segment and the second encoded video segment based on the first and second video performance indicators; and
outputting the selected video segment to a video device.
13 . The non-transitory computer-readable storage medium of claim 12 , wherein the first and second video performance indicators comprise respective quality metrics associated with the first and second video segments.
14 . The non-transitory computer-readable storage medium of claim 12 , wherein determining the selected video segment based on the first and second video performance indicators comprises:
determining a threshold switching video performance indicator as a function of the first video performance indicator; and
selecting the second video segment as the selected video segment when the second video performance indicator associated with the second video segment exceeds the threshold switching video performance indicator.
15 . The non-transitory computer-readable storage medium of claim 12 , wherein determining the selected video segment based on the first and second video performance indicators comprises:
performing a detection indicating that the first video segment comprises a key frame and the second video segment comprises a non-key frame; and
selecting the first video segment in response to the detection.
16 . The non-transitory computer-readable storage medium of claim 12 , wherein the first wireless network connection comprises a first cellular network connection to a first cellular network operated by a first wireless carrier, and wherein the second wireless network connection comprises a second cellular network connection to a second cellular network operated by a second wireless carrier different than the first wireless carrier.
17 . The non-transitory computer-readable storage medium of claim 12 , wherein the first wireless network connection comprises a first WiFi network connection and the second wireless network connection comprises a second WiFi network connection.
18 . The non-transitory computer-readable storage medium of claim 12 , wherein the first wireless network connection comprises a cellular network a connection to a cellular network and the second wireless network connection comprises a WiFi network connection to a WiFi network.
19 . The non-transitory computer-readable storage medium of claim 12 , wherein the first wireless network connection operates according to a first communication protocol and wherein the second wireless network connection operates according to a second communication protocol different than the first communication protocol.
20 . A computer system comprising:
one or more processors; and
a non-transitory computer-readable storage medium storing instructions for wirelessly receiving video at a remote support server supporting a vehicle during a remote support session, the instructions when executed by the one or more processors causing the one or more processors to perform steps comprising:
establishing the remote support session with the vehicle;
receiving at a video processor, a first encoded video segment from a first wireless network connection;
determining a first video performance indicator associated with the first encoded video segment;
receiving at the video processor from a second wireless network connection different than the first wireless network connection, a second encoded video segment that represents redundant underlying video content as the first encoded video segment;
determining a second video performance indicator associated with second encoded video segment; and
determining a selected video segment selected between the first encoded video segment and the second encoded video segment based on the first and second video performance indicators; and
outputting the selected video segment to a video device.