Low latency wireless communication system for teleoperated vehicle environments
A vehicle remote support system includes a communication system that operates over a plurality of parallel wireless network connections to provide low-latency video from vehicle to a remote support server that provides remote support to the vehicle dependent on real-time video. The vehicle includes a source that encodes multiple versions of the original video segments (e.g., one per wireless network connection) and transmits the multiple versions of the segments to a sink at the remote support server over the respective wireless connections. This redundant multi-path communication system rationally allocates network resources to the managed video streams and balances bandwidth against latency in order to avoid network congestion and safety issues associated with single-path transmissions. In other embodiments, a similar communication system that transmits video or other real-time messages between a source and a sink may be utilized in cloud robotics applications.
1. A method for wirelessly communicating low latency video from a vehicle to a remote support server during a remote support session, the method comprising:
receiving a first video feed from a first camera;
encoding, by a first encoder, a first video segment of the first video feed to generate a first encoded video segment, wherein the first encoder adjusts a set of encoding parameters for encoding the first video segment in response to network feedback indicating limited bandwidth conditions are present on a first wireless network connection;
encoding, by a second encoder, the first video segment to generate a second encoded video segment, the second encoded video segment including redundant video data to the first encoded video segment;
generating, by a network orchestrator, an assignment of the first encoded video segment for transmission by a first communication unit over the first wireless network connection, and an assignment of the second encoded video segment for transmission by a second communication unit over a second wireless network connection different than the first wireless network connection;
transmitting, by the first communication unit, the first encoded video segment over the first wireless network connection;
transmitting, by the second communication unit, the second encoded video segment over the second wireless network connection.
2. The method of claim 1 , wherein encoding the first video segment by the first encoder comprises:
receiving at the first encoder, feedback parameters relating to transmission of prior encoded video segments over the first wireless network connection; and
adjusting encoding parameters of the first encoder based on the feedback parameters.
3. The method of claim 1 , wherein encoding the first video segment by the first encoder comprises:
receiving location data associated with the vehicle;
determining encoding parameters from a geospatial database based on the location data; and
applying the encoding parameters when encoding the first video segment.
4. The method of 1 , wherein encoding the first video segment by the first encoder comprises:
responsive to network feedback indicating limited bandwidth conditions are present on the first wireless network connection, encoding chromatic components of the first video segment at lower resolution than luminosity components of the first video segment.
5. The method of 1 , wherein encoding the first video segment by the first encoder comprises:
responsive to network feedback indicating limited bandwidth conditions are present on the first wireless network, encoding the first video segment at a higher resolution if the first video frame is a key frame than if the first video segment is a non-key frame.
6. The method of claim 1 , further comprising:
receiving a second video feed from a second camera;
encoding, by a third encoder, the second video segment to generate a third encoded video segment;
encoding, by a fourth encoder, the video segment to generate a fourth encoded video segment;
transmitting, by the first communication unit, the third encoded video segment over the first wireless network;
transmitting, by the second communication unit, the fourth encoded video segment over the second wireless network, the fourth encoded video segment including redundant video data to the third encoded video segment.
7. The method of claim 1 , wherein encoding the first video segment by the first encoder comprises applying a first encoding format with first encoding parameters, and wherein encoding the first video segment by the second encoder comprises applying a second code with second encoding parameters different than the first encoding parameters.
8. The method of claim 1 , wherein generating the assignments by the network orchestrator comprises:
applying an optimization algorithm to optimize the assignments according to a predefined optimization criterion.
9. The method of claim 8 , wherein the optimization criterion comprises at least one of minimizing predicting latency, minimizing predicted network congestion, and maximizing predicted quality.
10. A non-transitory computer-readable storage medium storing instructions for wirelessly communicating low latency video from a vehicle to a remote support server during a remote support session, the instructions when executed by one or more processors causing the one or more processors to perform steps comprising:
receiving a first video feed from a first camera;
encoding, by a first encoder, a first video segment of the first video feed to generate a first encoded video segment, wherein the first encoder adjusts a set of encoding parameters for encoding the first video segment in response to network feedback indicating limited bandwidth conditions are present on a first wireless network connection;
encoding, by a second encoder, the first video segment to generate a second encoded video segment, the second encoded video segment including redundant video data to the first encoded video segment;
generating, by a network orchestrator, an assignment of the first encoded video segment for transmission by a first communication unit over the first wireless network connection, and an assignment of the second encoded video segment for transmission by a second communication unit over a second wireless network connection different than the first wireless network connection;
transmitting, by the first communication unit, the first encoded video segment over the first wireless network connection;
transmitting, by the second communication unit, the second encoded video segment over the second wireless network connection.
11. The non-transitory computer-readable storage medium of claim 10 , wherein encoding the first video segment by the first encoder comprises:
receiving at the first encoder, feedback parameters relating to transmission of prior encoded video segments over the first wireless network connection; and
adjusting encoding parameters of the first encoder based on the feedback parameters.
12. The non-transitory computer-readable storage medium of claim 10 , wherein encoding the first video segment by the first encoder comprises:
receiving location data associated with the vehicle;
determining encoding parameters from a geospatial database based on the location data; and
applying the encoding parameters when encoding the first video segment.
13. The non-transitory computer-readable storage medium of 10 , wherein encoding the first video segment by the first encoder comprises:
responsive to network feedback indicating limited bandwidth conditions are present on the first wireless network connection, encoding chromatic components of the first video segment at lower resolution than luminosity components of the first video segment.
14. The non-transitory computer-readable storage medium of 10 , wherein encoding the first video segment by the first encoder comprises:
responsive to network feedback indicating limited bandwidth conditions are present on the first wireless network, encoding the first video segment at a higher resolution if the first video segment is a key frame than if the first video segment is a non-key frame.
15. The non-transitory computer-readable storage medium of claim 10 , further comprising:
receiving a second video feed from a second camera;
encoding, by a third encoder, the second video segment to generate a third encoded video segment;
encoding, by a fourth encoder, the video segment to generate a fourth encoded video segment;
transmitting, by the first communication unit, the third encoded video segment over the first wireless network;
transmitting, by the second communication unit, the fourth encoded video segment over the second wireless network, the fourth encoded video segment including redundant video data to the third encoded video segment.
16. The non-transitory computer-readable storage medium of claim 10 , wherein encoding the first video segment by the first encoder comprises applying a first encoding format with first encoding parameters, and wherein encoding the first video segment by the second encoder comprises applying a second code with second encoding parameters different than the first encoding parameters.
17. The non-transitory computer-readable storage medium of claim 10 , wherein generating the assignments by the network orchestrator comprises:
applying an optimization algorithm to optimize the assignments according to a predefined optimization criterion.
18. The non-transitory computer-readable storage medium of claim 17 , wherein the optimization criterion comprises at least one of minimizing predicting latency, minimizing predicted network congestion, and maximizing predicted quality.
19. A vehicle system for wirelessly communicating low latency video from a vehicle to a remote support server during a remote support session, the vehicle system comprising:
a drive system for controlling operation of the vehicle;
a camera for generating a first video feed;
one or more processors;
a non-transitory computer-readable storage medium storing instructions that when executed by one or more processors causing the one or more processors to perform steps comprising:
receiving the first video feed from the camera;
encoding, by a first encoder, a first video segment of the first video feed to generate a first encoded video segment, wherein the first encoder adjusts a set of encoding parameters for encoding the first video segment in response to network feedback indicating limited bandwidth conditions are present on a first wireless network connection;
encoding, by a second encoder, the first video segment to generate a second encoded video segment, the second encoded video segment including redundant video data to the first encoded video segment;
generating, by a network orchestrator, an assignment of the first encoded video segment for transmission by over the first wireless network connection, and an assignment of the second encoded video segment for transmission over a second wireless network connection different than the first wireless network connection;
a first communication unit to transmit the first encoded video segment over the first wireless network connection; and
a second communication unit to transmit the second encoded video segment over the second wireless network connection.
20. The vehicle system of claim 19 , wherein generating the assignments by the network orchestrator comprises:
applying an optimization algorithm to optimize the assignments according to a predefined optimization criterion.