Electronic device for controlling driving of vehicle and operation method thereof
Provided is an electronic device that determines a final steering value and a final throttle value for controlling driving of a vehicle through a pre-trained first and second neural networks, and a method for operating the same.
1. A method for operating an electronic device for controlling driving of a vehicle, the method comprising:
acquiring first sensed data and second sensed data;
determining a first reliability level, a first steering value, a first throttle value based on an output of a first neural network to which the first sensed data is input, and determining a second reliability level, a second steering value and a second throttle value based on an output of a second neural network to which the second sensed data is input; and
determining a final steering value for controlling driving of the vehicle from the first steering value and the second steering value based on the first reliability level and the second reliability level, and determining a final throttle value for controlling driving of the vehicle from the first throttle value and the second throttle value based on the first reliability level and the second reliability level,
wherein the first neural network is a convolutional neural network trained to output a steering value and a throttle value for driving the vehicle in a drivable area based on first sensed data,
wherein the second neural network is a convolutional neural network trained to output a steering value and a throttle value for driving the vehicle in a drivable area based on second sensed data,
wherein the first reliability level is calculated based on outputs of a softmax layer of the first neural network, and
wherein the second reliability level is calculated based on outputs of a softmax layer of the second neural network.
2. The method of claim 1 , wherein the determining the final steering value and the final throttle value includes:
weighing each of the first steering value and the second steering value according to a ratio of the first reliability level and the second reliability level, and determining the final steering value from a weighed first steering value and a weighed second steering value; and
weighing each of the first throttle value and the second throttle value according to a ratio of the first reliability level and the second reliability level, and determining the final throttle value from a weighed first throttle value and a weighed second throttle value.
3. The method of claim 1 , wherein the determining the final steering value and the final throttle value includes:
determining any one of the first steering value and the second steering value as the final steering value through a magnitude comparison between the first reliability level and the second reliability level, and determining one of the first throttle value and the second throttle value as the final throttle value through the magnitude comparison.
4. The method of claim 1 , wherein the first reliability level includes a 1-1 reliability level with respect to the first steering value and a 1-2 reliability level with respect to the first throttle value,
wherein the second reliability level includes a 2-1 reliability level with respect to the second steering value and a 2-2 reliability level with respect to the second throttle value, and
wherein the determining the final steering value and the final throttle value includes determining the final steering value from the first steering value and the second steering value based on the 1-1 reliability level and the 2-1 reliability level, and determining the final throttle value from the first throttle value and the second throttle value based on the 1-2 reliability level and the 2-2 reliability level.
5. The method of claim 1 , wherein the first sensed data is data acquired from an image sensor, and the second sensed data is data acquired from a Lidar sensor.
6. The method of claim 5 , wherein the second sensed data is two-dimensional image data acquired by processing Lidar data acquired from the Lidar sensor through a road detection network.
7. The method of claim 5 , wherein the first sensed data is data acquired by processing image data acquired from an image sensor through segmentation.
8. The method of claim 1 , wherein the first sensed data is image data acquired from an image sensor, and the second sensed data is data acquired by processing the image data through segmentation.
9. An electronic device for controlling driving of a vehicle, the electronic device comprising:
a memory in which at least one program is stored; and
a processor configured to:
by executing the at least one program, acquire first sensed data and second sensed data,
determine a first reliability level, a first steering value and a first throttle value based on an output of a first neural network to which the first sensed data is input, and determine a second reliability level, a second steering value and a second throttle value based on an output of a second neural network to which the second sensed data is input, and
determine a final steering value for controlling driving of the vehicle from the first steering value and the second steering value based on the first reliability level and the second reliability level, and determine a final throttle value for controlling driving of the vehicle from the first throttle value and the second throttle value based on the first reliability level and the second reliability level,
wherein the first neural network is a convolutional neural network trained to output a steering value and a throttle value for driving the vehicle in a drivable area based on first sensed data,
wherein the second neural network is a convolutional neural network trained to output a steering value and a throttle value for driving the vehicle in a drivable area based on second sensed data,
wherein the first reliability level is calculated based on outputs of a softmax layer of the first neural network, and
wherein the second reliability level is calculated based on outputs of a softmax layer of the second neural network.
10. A non-transitory computer-readable recording medium on which a program for executing a method for operating an electronic device for controlling driving of a vehicle on a computer is recorded, wherein the method includes:
acquiring first sensed data and second sensed data;
determining a first reliability level, a first steering value and a first throttle value based on an output of a first neural network to which the first sensed data is input, and determining a second reliability level, a second steering value and a second throttle value based on an output of a second neural network to which the second sensed data is input; and
determining a final steering value for controlling driving of the vehicle from the first steering value and the second steering value based on the first reliability level and the second reliability level, and determining a final throttle value for controlling driving of the vehicle from the first throttle value and the second throttle value based on the first reliability level and the second reliability level,
wherein the first neural network is a convolutional neural network trained to output a steering value and a throttle value for driving the vehicle in a drivable area based on first sensed data,
wherein the second neural network is a convolutional neural network trained to output a steering value and a throttle value for driving the vehicle in a drivable area based on second sensed data,
wherein the first reliability level is calculated based on outputs of a softmax layer of the first neural network, and
wherein the second reliability level is calculated based on outputs of a softmax layer of the second neural network.