Remote operation of a vehicle using virtual representations of a vehicle state
In various examples, at least partial control of a vehicle may be transferred to a control system remote from the vehicle. Sensor data may be received from a sensor(s) of the vehicle and the sensor data may be encoded to generate encoded sensor data. The encoded sensor data may be transmitted to the control system for display on a virtual reality headset of the control system. Control data may be received by the vehicle and from the control system that may be representative of a control input(s) from the control system, and actuation by an actuation component(s) of the vehicle may be caused based on the control input.
1. A processor comprising processing circuitry to:
initiate a remote control session to cause transfer of at least partial control of a vehicle to a control system remote from the vehicle;
generate, using one or more sensors of the vehicle, vehicle state data representative of a state of one or more components of the vehicle; and
transmit the vehicle state data to the control system to cause the control system to generate, based at least in part on the vehicle state data, a virtual display of a representation of at least one component of the one or more components of the vehicle in the state.
2. The processor of claim 1 , further comprising processing circuitry to:
receive control data representative of at least one control input from the control system; and
cause actuation by at least one actuation component of the vehicle based at least in part on the at least one control input.
3. The processor of claim 1 , wherein the transmission of the vehicle state data includes:
encoding the vehicle state data to generate encoded data; and
transmitting the encoded data to the control system.
4. The processor of claim 1 , further comprising processing circuitry to:
generate image data representative of one or more video streams; and
transmit the image data to the control system,
wherein the generation of the virtual display is further based at least in part on the image data.
5. The processor of claim 4 , wherein the virtual display includes at least one video stream of the one or more video streams.
6. The processor of claim 1 , wherein the one or more components include at least one of a steering wheel, a wheel, a gear control, or a tire.
7. The processor of claim 1 , wherein the virtual display includes a representation of the vehicle within a virtual environment from a vantage point interior to the vehicle, and one or more portions of the representation of the vehicle are removed or made at least partially translucent such that at least one component of the one or more components are visible in the virtual display from the vantage point.
8. The processor of claim 1 , further comprising processing circuitry to:
generate calibration data corresponding to at least one of a sensitivity of a steering mechanism of the vehicle, a braking sensitivity of a braking mechanism of the vehicle, or an acceleration sensitivity of an acceleration mechanism of the vehicle; and
transmit the calibration data to the control system,
wherein the calibration data is used by the control system, at least in part, to set a sensitivity of a steering mechanism of the control system, a braking sensitivity of a braking mechanism of the control system, or an acceleration sensitivity of an acceleration mechanism of the control system.
9. A system comprising:
one or more processing units; and
one or more memory units storing instructions that, when executed using the one or more processing units, cause the one or more processing units to perform operations comprising:
receiving vehicle state information corresponding to one or more components of a vehicle;
based at least in part on the vehicle state information, generating a virtual environment including a representation of at least one component of the one or more components at a current state determined based at least in part on the vehicle state information; and
causing display of the virtual environment using a control system.
10. The system of claim 9 , wherein the operations further comprise:
generating control data representative of at least one control for the vehicle based at least in part on receiving at least one control input to the control system; and
transmitting the control data to the vehicle to cause the vehicle to execute at least one control.
11. The system of claim 9 , wherein the virtual environment includes a representation of at least a portion of a virtual vehicle corresponding to the vehicle and at least one video stream representing at least a portion of sensor data received from the vehicle.
12. The system of claim 9 , wherein the display is of at least a portion of a virtual representation of the vehicle from a vantage point within the vehicle, and one or more portions of the representation of a virtual vehicle are removed or made at least partially translucent such that the at least one component of the one or more components are visible in the virtual environment from the vantage point.
13. The system of claim 9 , wherein the display is of at least a portion of a virtual representation of the vehicle from a vantage point outside of the vehicle, and the display further includes one or more video streams displayed using image data generated using one or more image sensors of the vehicle.
14. The system of claim 9 , wherein the control system includes a virtual reality headset and at least one physical control component, the at least one physical control component including one or more of a steering component, a braking component, or an acceleration component.
15. The system of claim 9 , wherein the control system includes at least one of a steering component, a braking component, or an acceleration component, and the operations further comprise calibrating the at least one of the steering component, the braking component, or the acceleration component based at least in part on calibration data received from the vehicle or obtained from a data store accessible by the control system.
16. A method comprising:
initiating a remote control session to cause transfer of at least partial control of a vehicle to a control system remote from the vehicle;
generating, using one or more sensors of the vehicle, vehicle state data representative of a state of one or more components of the vehicle; and
transmitting the vehicle state data to the control system to cause the control system to generate, based at least in part on the vehicle state data, a virtual display of a representation of at least one component of the one or more components of the vehicle in the state.
17. The method of claim 16 , further comprising:
receiving control data representative of at least one control input from the control system; and
causing actuation by at least one actuation component of the vehicle based at least in part on the at least one control input.
18. The method of claim 16 , wherein the transmitting the vehicle state data includes:
encoding the vehicle state data to generate encoded data; and
transmitting the encoded data to the control system.
19. The method of claim 16 , further comprising:
generating image data representative of one or more video streams; and
transmitting the image data to the control system,
wherein the generation of the virtual display is further based at least in part on the image data.
20. The method of claim 16 , wherein the virtual display includes a representation of the vehicle within a virtual environment from a vantage point interior to the vehicle, and one or more portions of the representation of the vehicle are removed or made at least partially translucent such that at least one component of the one or more components are visible in the virtual display from the vantage point.