Device and method for monitoring vessel and harbor
The present invention relates to a method by which a computing means monitors a harbor, and a harbor monitoring method, according to one aspect of the present invention, comprises the steps of: acquiring a harbor image; generating a segmentation image corresponding to the harbor image; generating a display image corresponding to the harbor image and having a first view attribute; generating a conversion segmentation image, which corresponds to the segmentation image and has a second view attribute different from the first view attribute; matching the display image so as to generate a panoramic image; matching the conversion segmentation image so as to generate a matching segmentation image; calculating ship mooring guide information on the basis of the matching segmentation image; and outputting the mooring guide information together with the panoramic image.
1. A method for monitoring a harbor performed by one or more processors, the method comprising:
obtaining a harbor image of a first view;
generating, by using an artificial neural network trained to output information related to a type of an object included in an input image from the input image, a segmentation image of the first view, from the harbor image, including a target region indicating a target object, wherein the target region corresponds to one or more pixels of the harbor image and one or more identification values reflecting a type of the target object are assigned to one or more pixels in the target region corresponding to specific pixels in the segmentation image;
generating a transformed segmentation image of a second view by transforming a viewpoint of the segmentation image of the first view, wherein the transformed segmentation image includes a transformed target region corresponding to the target region of the segmentation image, and the identification values are assigned to one or more pixels in the transformed target region;
calculating monitoring information based on a location of the one or more pixels in the transformed target region and the identification values assigned to the one or more pixels in the transformed target region of the transformed segmentation image, wherein the monitoring information includes at least one of a distance between the target object and a wharf wall or a relative speed of the target object to the wharf wall; and
outputting the monitoring information.
2. The method of claim 1 ,
wherein the second view is a view overlooking a sea level from a direction perpendicular to the sea level.
3. The method of claim 1 ,
wherein the distance includes a first distance and a second distance which are distances to the wharf wall from a first point and a second point, respectively, corresponding to both ends of the target object contacting with a sea level.
4. The method of claim 1 , wherein the method further comprises:
generating a display image of a third view by transforming a viewpoint of the harbor image of the first view; and
outputting the display image.
5. The method of claim 4 ,
wherein the harbor image includes a first harbor image and a second harbor image whose monitoring area is at least partially overlapped with the first harbor image,
wherein the segmentation image includes a first segmentation image and a second segmentation image corresponding to the first harbor image and the second harbor image, respectively, wherein at least one of the first segmentation image or the second segmentation image includes the target region,
wherein the display image is generated by matching a first display image and a second display image, wherein the first display image and the second display image corresponds to the first harbor image and the second harbor image, respectively, and
wherein the transformed segmentation image is generated by matching a first transformed segmentation image corresponding to the first segmentation image and a second transformed segmentation image corresponding to the second segmentation image, wherein at least one of the first transformed segmentation image or the second transformed segmentation image includes the transformed target region.
6. The method of claim 5 ,
wherein the display image is generated by matching the first display image and the second display image using first matching information, wherein the first matching information is calculated based on a first image of the third view and a second image of the third view, and
wherein the transformed segmentation image is generated by matching the first transformed segmentation image and the second transformed segmentation image using second matching information, wherein the second matching information is calculated based on a third image of the second view and a fourth image of the second view.
7. The method of claim 6 ,
wherein the first image and the second image are the first display image and the second display image, respectively, and
wherein the third image and the fourth image are an image generated by transforming a viewpoint of the first harbor image to the second view and an image generated by transforming a viewpoint of the second harbor image to the second view, respectively.
8. The method of claim 1 ,
wherein the harbor image is an image that has been preprocessed on an original image to remove noise included in the original image.
9. The method of claim 1 , wherein the method further comprises:
determining a collision risk based on the monitoring information; and
outputting the collision risk to a user.
10. A non-transitory computer-readable medium storing a program for executing the method according to claim 1 on a computer.
11. A device for monitoring a harbor, the device comprising:
a camera generating a harbor image of a first view by capturing a harbor; and
a controller configured to calculate monitoring information, wherein the monitoring information includes at least one of a distance between a target object and a wharf wall or a relative speed of the target object to the wharf wall,
wherein the controller further configured to:
obtain the harbor image from the camera,
generate, by using an artificial neural network trained to output information related to a type of an object included in an input image from the input image, a segmentation image of the first view, from the harbor image, including a target region indicating a target object, wherein the target region corresponds to one or more pixels of the harbor image and one or more identification values reflecting a type of the target object are assigned to one or more pixels in the target region corresponding to specific pixels in the segmentation image,
generate a transformed segmentation image of a second view by transforming a viewpoint of the segmentation image of the first view, wherein the transformed segmentation image includes a transformed target region corresponding to the target region of the segmentation image, and the identification values are assigned to one or more pixels in the transformed target region, and
calculate the monitoring information based on a location of the one or more pixels in the transformed target region and the identification values assigned to the one or more pixels in the transformed target region of the transformed segmentation image.
12. The device of claim 11 ,
wherein the second view is a view overlooking a sea level from a direction perpendicular to the sea level.
13. The device of claim 11 ,
wherein the distance includes a first distance and a second distance which are distances to the wharf wall from a first point and a second point, respectively, corresponding to both ends of the target object contacting with a sea level.
14. The device of claim 11 , wherein the controller is further configured to:
generate a display image of a third view by transforming a viewpoint of the harbor image of the first view, and
output the display image.
15. The device of claim 14 ,
wherein the harbor image includes a first harbor image and a second harbor image whose monitoring area is at least partially overlapped with the first harbor image,
wherein the segmentation image includes a first segmentation image and a second segmentation image corresponding to the first harbor image and the second harbor image, respectively, wherein at least one of the first segmentation image or the second segmentation image includes the target region,
wherein the display image is generated by matching a first display image and a second display image, wherein the first display image and the second display image corresponds to the first harbor image and the second harbor image, respectively, and
wherein the transformed segmentation image is generated by matching a first transformed segmentation image corresponding to the first segmentation image and a second transformed segmentation image corresponding to the second segmentation image, wherein at least one of the first transformed segmentation image or the second transformed segmentation image includes the transformed target region.
16. The device of claim 15 ,
wherein the display image is generated by matching the first display image and the second display image using first matching information, wherein the first matching information is calculated based on a first image of the third view and a second image of the third view, and
wherein the transformed segmentation image is generated by matching the first transformed segmentation image and the second transformed segmentation image using second matching information, wherein the second matching information is calculated based on a third image of the second view and a fourth image of the second view.
17. The device of claim 16 ,
wherein the first image and the second image are the first display image and the second display image, respectively, and
wherein the third image and the fourth image are an image generated by transforming a viewpoint of the first harbor image to the second view and an image generated by transforming a viewpoint of the second harbor image to the second view, respectively.
18. The device of claim 11 ,
wherein the harbor image is an image that has been preprocessed on an original image to remove noise included in the original image.