System and method for remote intervention of vehicles
Technologies disclosed relate to a remote intervention system for the operation of a vehicle, which can be an autonomous vehicle, a vehicle that includes driver assist features, a vehicle used for ride sharing services or the like. The system includes a remote operator receiving a request for remote intervention from a vehicle when the operation of the vehicle is suspended and sending a response to the vehicle. The vehicle can transmit visual data detected by one or more sensors on the vehicle to the remote operator. The remote operator can output a response after analyzing the visual data transmitted by the vehicle. The remote operator can be a human operator or an AI operator. The response can result in an update of the vehicle operation.
1. A vehicle configured to receive remote intervention, the vehicle comprising:
an onboard hardware processor;
at least one sensor in electronic communication with the onboard hardware processor, the at least one sensor configured to detect visual data about a road on which the vehicle is traveling and transmitting the visual data to the onboard hardware processor;
a wireless transmitter; and
a wireless receiver,
wherein the onboard hardware processor is configured to:
search for a remote server from a plurality of available remote servers,
verify an identity of the remote server,
in response to the identity of the remote server being verified, transmit a remote intervention request to the remote server via the wireless transmitter in response to an operation of the vehicle being suspended due to the onboard hardware processor detecting that the visual data is missing a marker object or includes an indeterminate object along a planned route,
output a command to stop the vehicle at its current location while waiting for a response from the remote server,
transmit the visual data to the remote server,
receive the response from the remote server in response to the remote intervention request via the wireless receiver, wherein the remote server is configured to generate and output the response by connecting to one or more external traffic knowledge bases providing current local traffic information so as to aid the remote server in locating the marker object or identifying the indeterminate object, and
act upon the response, the acting resulting in a control signal to update the operation of the vehicle.
2. The vehicle of claim 1 , wherein the visual data comprises three- dimensional image data.
3. The vehicle of claim 1 , wherein the vehicle is an autonomous driving vehicle.
4. The vehicle of claim 1 , wherein the response comprises a direct command to the onboard hardware processor.
5. The vehicle of claim 4 , wherein the onboard hardware processor is configured to act upon the direct command by relinquishing control of the vehicle to the remote server, wherein the remote server is configured to output the control signal to the vehicle.
6. The vehicle of claim 1 , wherein the operation of the vehicle is suspended due to the indeterminate object, and wherein the response comprises a classification of the indeterminate object or a confirmation of a decision suggested by the onboard hardware processor.
7. The vehicle of claim 6 , wherein the onboard hardware processor is configured to output a control signal to update the operation of the vehicle based at least in part on the response.
8. The vehicle of claim 7 , wherein the onboard hardware processor outputs the control signal based at least in part on additional signal inputs from the at least one sensor.
9. The vehicle of claim 1 , wherein the at least one sensor comprises LiDAR, camera, infrared camera, ultrasonic transducer, Radar, or a combination thereof.
10. The vehicle of claim 1 , wherein the response from the remote server is provided by an artificial intelligence (AI) operator based on a prior response provided by a human operator.
11. The vehicle of claim 1 , wherein the vehicle moves to an adjacent free lane or in its current lane in response to the control signal.
12. The vehicle of claim 1 , wherein the vehicle remains stopped in response to the control signal.
13. A method of providing remote invention by a remote server to a vehicle, the method comprising:
using an onboard hardware processor of the vehicle:
receiving signal inputs from at least one sensor of the vehicle in electronic communication with the onboard hardware processor, the at least one sensor configured to detect visual data about a road on which the vehicle is traveling;
in response to an operation of the vehicle being suspended due to the onboard hardware processor detecting that the visual data is missing a marker object or includes an indeterminate object along a planned route:
searching for a remote server from a plurality of available remote servers, verifying an identity of the remote server,
in response to the identity of the remote server being verified, outputting a remote intervention request to the remote server via a wireless transmitter;
outputting a command to stop the vehicle at its current location while waiting for a response from the remote server,
transmitting the visual data to the remote server;
receiving the response from the remote server in response to the remote intervention request via a wireless receiver, wherein the remote server is configured to generate and output the response by connecting to one or more external traffic knowledge bases providing current local traffic information so as to aid the remote server in locating the marker object or identifying the indeterminate object; and
acting upon the response, the acting resulting in a control signal to update the operation of the vehicle.
14. The method of claim 13 , wherein the visual data comprises three- dimensional image data.
15. The method of claim 13 , wherein the vehicle is an autonomous driving vehicle.
16. The method of claim 13 , wherein the response comprises a direct command to the onboard hardware processor.
17. The method of claim 16 , wherein the acting comprises relinquishing control of the vehicle to the remote server, wherein the remote server is configured to output the control signal to the vehicle.
18. The method of claim 13 , wherein the operation of the vehicle is suspended due to the indeterminate object, and wherein the response comprises a classification of the indeterminate object or a confirmation of a decision suggested by the onboard hardware processor.
19. The method of claim 18 , wherein the acting comprises outputting a control signal to update the operation of the vehicle based at least in part on the response.
20. The method of claim 19 , further comprises outputting the control signal based at least in part on additional signal inputs from the at least one sensor.
21. The method of claim 13 , wherein the at least one sensor comprises LiDAR, camera, infrared camera, ultrasonic transducer, Radar, or a combination thereof.
22. The method of claim 13 , wherein the response from the remote server is provided by an artificial intelligence (AI) operator based on a prior response provided by a human operator.
23. The method of claim 13 , wherein the vehicle moves to an adjacent free lane or in its current lane in response to the control signal.
24. The method of claim 13 , wherein the vehicle remains stopped in response to the control signal.