Method for image processing for controlling vehicle and electronic device for performing same method

1 . An image processing method performed by an electronic device, the image processing method comprising:

receiving an original image generated by using a camera mounted on a vehicle;

generating first detail feature information by processing the original image in a preset first scheme that applies one or more first convolutional layers to the original image and generating second detail feature information by processing the first detail feature information in a preset second scheme that applies one or more second convolutional layers to the first detail feature information;

generating lane type contextual information by processing the original image in a preset third scheme that applies one or more third convolution layers, parameters of which are pre-trained to perform stem, gather and expansion (GE) and context embedding (CE) on the original image;

determining a type of a target line in the original image based on the second detail feature information and the lane type contextual information;

generating lane instance contextual information by processing the first detail feature information in a preset fourth scheme that applies one or more fourth convolutional layers, parameters of which are pretrained to perform gather and expansion (GE) and context embedding (CE); and

determining an instance of the target line based on the second detail feature information and the lane instance contextual information wherein the instance of the target line is information on a relative position of the target line based on a position of the vehicle, and

wherein the determining of the type of the target line comprises:

generating upsampled lane type contextual information by upsampling the lane type contextual information;

generating first reference information based on the upsampled lane type contextual information and the second detail feature information;

generating downsampled second detail feature information by downsampling the second detail feature information;

generating second reference information based on the downsampled second detail feature information and the lane type contextual information; and

determining the type of the target line in the original image based on the first reference information and the second reference information.

2 . The image processing method of claim 1 ,

wherein the type of the target line comprises at least one of a solid line, a dotted line, a single line, a double line, and a color.

3 . The image processing method of claim 1 ,

wherein the instance of the target line comprises at least one of a left line of a target lane on which the vehicle drives, a right line of the target lane, a line of a left lane of the target lane, and a line of a right lane of the target lane.

4 . The image processing method of claim 1 , wherein the determining of the instance of the target line comprises:

generating downsampled second detail feature information by downsampling the second detail feature information;

generating third reference information based on the downsampled second detail feature information and the lane instance contextual information;

generating upsampled lane instance contextual information by upsampling the lane instance contextual information;

generating fourth reference information based on the upsampled lane instance contextual information and the second detail feature information; and

determining the instance of the target line in the original image based on the third reference information and the fourth reference information.

5 . The image processing method of claim 4 ,

wherein the determining of the instance of the target line in the original image based on the third reference information and the fourth reference information comprises:

generating line existence information based on the line instance contextual information;

generating fusion information by fusing the third reference information and the fourth reference information; and

determining the instance of the target line based on the line existence information and the fusion information.

6 . The image processing method of claim 1 ,

wherein the image processing method is performed by a model pre-trained based on a neural network.

7 . The image processing method of claim 1 , further comprising:

controlling the vehicle based on at least one of the type and the instance of the target line.

8 . A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the image processing method of claim 1 .

9 . An electronic device comprising:

a memory configured to store a program for processing an image; and

a processor configured to execute the program,

wherein the processor is configured to:

receive an original image generated by using a camera mounted on a vehicle,

generate first detail feature information by processing the original image in a preset first scheme that applies one or more first convolutional layers to the original image and generate second detail feature information by processing the first detail feature information in a preset second scheme that applies one or more second convolutional layers to the first detail feature information,

generate lane type contextual information by processing the original image in a preset third scheme that applies one or more third convolution layers, parameters of which are pre-trained to perform stem, gather and expansion (GE) and context embedding (CE) on the original image,

determine a type of a target line in the original image based on the second detail feature information and the lane type contextual information,

generate lane instance contextual information by processing the first detail feature information in a preset fourth scheme that applies one or more fourth convolutional layers, parameters of which are pretrained to perform gather and expansion (GE) and context embedding (CE), and

determine an instance of the target line based on the second detail feature information and the lane instance contextual information wherein the instance of the target line is information on a relative position of the target line based on a position of the vehicle, and

wherein the processor is configured to:

generate upsampled lane type contextual information by upsampling the lane type contextual information,

generate first reference information based on the upsampled lane type contextual information and the second detail feature information,

generate downsampled second detail feature information by downsampling the second detail feature information,

generate second reference information based on the downsampled second detail feature information and the lane type contextual information, and

determine the type of the target line in the original image based on the first reference information and the second reference information.

10 . The electronic device of claim 9 ,

wherein the type of the target line comprises at least one of a solid line, a dotted line, a single line, a double line, and a color.

11 . The electronic device of claim 9 ,

wherein the instance of the target line comprises at least one of a left line of a target lane on which the vehicle drives, a right line of the target lane, a line of a left lane of the target lane, and a line of a right lane of the target lane.

12 . The electronic device of claim 9 ,

wherein the processor is configured to:

generate downsampled second detail feature information by downsampling the second detail feature information,

generate third reference information based on the downsampled second detail feature information and the lane instance contextual information,

generate upsampled lane instance contextual information by upsampling the lane instance contextual information,

generate fourth reference information based on the upsampled lane instance contextual information and the second detail feature information, and

determine the instance of the target line in the original image based on the third reference information and the fourth reference information.

13 . The electronic device of claim 12 ,

wherein the processor is configured to:

generate line existence information based on the line instance contextual information,

generate fusion information by fusing the third reference information and the fourth reference information, and

determine the instance of the target line based on the line existence information and the fusion information.

14 . The electronic device of claim 9 ,

wherein the electronic device is comprised in the vehicle.

15 . The electronic device of claim 9 ,

wherein the processor is configured to:

control the vehicle based on the type and the instance of the target line.

16 . A vehicle control method performed by an electronic device, the vehicle control method comprising:

receiving an original image generated by using a camera mounted on a vehicle;

generating first detail feature information by processing the original image in a preset first scheme that applies one or more first convolutional layers to the original image and generating second detail feature information by processing the first detail feature information in a preset second scheme that applies one or more second convolutional layers to the first detail feature information;

generating lane type contextual information by processing the original image in a preset third scheme that applies one or more third convolution layers, parameters of which are pre-trained to perform stem, gather and expansion (GE) and context embedding (CE) on the original image;

determining a type of a target line in the original image based on the second detail feature information and the lane type contextual information;

generating lane instance contextual information by processing the first detail feature information in a preset fourth scheme that applies one or more fourth convolutional layers, parameters of which are pretrained to perform gather and expansion (GE) and context embedding (CE);

determining an instance of the target line based on the second detail feature information and the lane instance contextual information wherein the instance of the target line is information on a relative position of the target line based on a position of the vehicle; and

controlling the vehicle based on at least one of the type and the instance of the target line, and

wherein the determining of the type of the target line comprises:

generating upsampled lane type contextual information by upsampling the lane type contextual information;

generating first reference information based on the upsampled lane type contextual information and the second detail feature information;

generating downsampled second detail feature information by downsampling the second detail feature information;

generating second reference information based on the downsampled second detail feature information and the lane type contextual information; and

determining the type of the target line in the original image based on the first reference information and the second reference information.