Transmitting sideband data to enable tele-operation of a vehicle
A system may generate visual data from a vehicle in a transportation network. The system may also generate sideband data from the vehicle. The sideband data may encapsulate a serialization of vehicle information collected using one or more sensors of the vehicle. In some implementations, the sideband data may be formatted according to JavaScript Object Notation (JSON) to enable reconstruction of the vehicle information by a server. The sideband data may be associated with timing information used to correlate the vehicle information to the visual data. The system may transmit a stream including the visual data and the sideband data to the server to enable tele-operation of the vehicle.
1 . A method, comprising:
generating visual data from a vehicle in a transportation network, wherein the visual data is associated with visual timing information corresponding to when the visual data was collected;
generating sideband data from the vehicle, the sideband data encapsulating a serialization of vehicle information collected using one or more sensors of the vehicle, wherein the sideband data is associated with sideband timing information corresponding to when the vehicle information was collected;
encoding the visual data and the visual timing information into one or more visual data packets;
encoding the sideband data and the sideband timing information into one or more sideband data packets;
generating a single stream comprising the one or more visual data packets and the one or more sideband data packets; and
transmitting the single stream including the visual data and the sideband data to enable tele-operation of the vehicle, wherein the visual timing information and the sideband timing information are used to correlate the vehicle information to the visual data.
2 . The method of claim 1 , wherein the visual data comprises one or more images collected from a camera of the vehicle.
3 . The method of claim 1 , wherein the visual data comprises point cloud data collected from a Lidar sensor of the vehicle.
4 . The method of claim 1 , wherein the sideband data is formatted according to at least one of JavaScript Object Notation (JSON), YAML, Binary JSON (BSON), or MessagePack to enable reconstruction of the vehicle information by a server.
5 . The method of claim 1 ,
wherein the one or more visual data packets and the one or more sideband data packets are Real-time Transport Protocol (RTP) packets in the single stream.
6 . The method of claim 1 , wherein the single stream is transmitted, via cellular communication, to a server using Real-Time Streaming Protocol (RTSP).
7 . The method of claim 1 , further comprising:
applying the sideband timing information, based on a time from a global positioning system and before transmitting the single stream, to the sideband data to enable a server to correlate the vehicle information to the visual data.
8 . The method of claim 1 , further comprising:
receiving the single stream that includes the visual data and the sideband data;
de-serializing the sideband data to re-construct the vehicle information; and
displaying the visual data and the vehicle information in a web browser to perform the tele-operation.
9 . The method of claim 1 , wherein the vehicle information indicates an object detection and a vehicle location in the transportation network.
10 . The method of claim 1 , wherein the visual timing information and the sideband timing information are used to correlate the vehicle information to the visual data by determining that the visual timing information and the sideband timing information are within a threshold.
11 . An apparatus, comprising:
a memory; and
a processor configured to execute instructions stored in the memory to:
generate visual data from a vehicle in a transportation network, wherein the visual data includes a visual data timestamp corresponding to when the visual data was collected;
generate sideband data from the vehicle, the sideband data encapsulating a serialization based on JSON of vehicle information collected using one or more sensors of the vehicle, wherein the sideband data includes a sideband data timestamp corresponding to when the vehicle information was collected;
encode the visual data and the visual data timestamp into a visual data packet;
encode the sideband data and the sideband data timestamp into a sideband data packet;
generate a single stream comprising the visual data packet and the sideband data packet; and
transmit the single stream including the visual data and the sideband data to enable tele-operation of the vehicle, wherein the visual data timestamp and the sideband data timestamp are used to correlate the vehicle information to the visual data.
12 . The apparatus of claim 11 , wherein the visual data comprises one or more images collected from a camera of the vehicle.
13 . The apparatus of claim 11 , wherein
the visual data packet and the sideband data packet are each Real-time Transport Protocol (RTP) packets in the single stream.
14 . The apparatus of claim 11 , wherein the single stream is transmitted, via Campus Wi-Fi, to a server using at least one of Real-Time Messaging Protocol (RTMP) or Reliable Internet Stream Transport (RIST).
15 . The apparatus of claim 11 , wherein the processor is further configured to execute instructions stored in the memory to:
apply the sideband data timestamp, based on a time from a global positioning system and before transmitting the single stream, to the sideband data to enable a server to correlate the vehicle information to the visual data.
16 . A non-transitory computer readable medium storing instructions operable to cause one or more processors to perform operations comprising:
receiving a single stream from a vehicle in a transportation network, the single stream including packets of visual data and packets of sideband data, wherein each packet of visual data comprises visual data generated by the vehicle and visual timing information corresponding to when the visual data was collected, and wherein each packet of sideband data comprises sideband data encapsulating a serialization of vehicle information collected using one or more sensors of the vehicle and sideband timing information corresponding to when the vehicle information was collected;
de-serializing the sideband data to re-construct the vehicle information and correlate the vehicle information to the visual data based on the visual timing information and the sideband timing information; and
displaying the visual data and the vehicle information in a web browser to enable tele-operation of the vehicle.
17 . The non-transitory computer readable medium of claim 16 , wherein the serialization includes human-readable text.
18 . The non-transitory computer readable medium of claim 16 , wherein the packets of visual data and the packets of sideband data are RTP packets in the single stream, the operations further comprising:
decoding the RTP packets in the single stream to obtain the visual data and the sideband data.
19 . The non-transitory computer readable medium of claim 16 , wherein the single stream is received, via a cellular telephone network, by a server using a port configured for real-time video transport.
20 . The non-transitory computer readable medium of claim 16 , wherein the vehicle is an autonomous vehicle (AV) and the vehicle information indicates vehicle velocity; vehicle location; vehicle operational state; vehicle destination; vehicle route; and vehicle sensor data.