System and method for estimating vegetation coverage in a real-world environment
Computer-implemented method and system ( 100 ) for estimating vegetation coverage in a real-world environment. The system receives an RGB image ( 91 ) of a real-world scenery ( 1 ) with one or more plant elements ( 10 ) of one or more plant species. At least one channel of the RGB image ( 91 ) is provided to a semantic regression neural network ( 120 ) which is trained to estimate at least a near-infrared channel (NIR) from the RGB image. The system obtains an estimate of the near-infrared channel (NIR) by applying the semantic regression neural network ( 120 ) to the at least one RGB channel ( 91 ). A multi-channel image ( 92 ) comprising at least one of the R-, G-, B-channels (R, G, B) of the RGB image and the estimated near-infrared channel (NIR), is provided as test input (TI 1 ) to a semantic segmentation neural network ( 130 ) trained with multi-channel images to segment the test input (TI 1 ) into pixels associated with plant elements and pixels not associated with plant elements. The system segments the test input (TI 1 ) using the semantic segmentation neural network ( 130 ) resulting in a vegetation coverage map ( 93 ) indicating pixels of the test input associated with plant elements ( 10 ) and indicating pixels of the test input not associated with plant elements.
1 . A computer-implemented method for estimating vegetation coverage in a real-world environment, the method comprising:
receiving an RGB image of a real-world scenery with one or more plant elements of one or more plant species;
providing at least one channel of the RGB image to a semantic regression neural network, the semantic regression neural network trained to estimate at least a near-infrared channel (NIR) from the at least one channel of the RGB image, the semantic regression neural network having a topology based on a convolutional segmentation neural network with its last layer being substituted by a monotonic activation function and its loss function being substituted by a regression loss function, to learn a pixel-wise regression transformation that transforms any RGB channel from an RGB domain to a target domain comprising at least a near-infrared domain;
obtaining an estimate of the near-infrared channel (NIR) by applying the semantic regression neural network to the RGB image;
deriving at least one infrared-dark channel of a multi-channel image, wherein infrared-dark channel values for each pixel are based on: the value of the near-infrared channel and a minimum value of available R-, G-, B-channels (R, G, B) and the near-infrared channel (NIR) for the respective pixel,
wherein the at least one infrared-dark channel is derived either (i) by applying respective mathematical operations to the estimated near-infrared channel or (ii) by configuring the semantic regression neural network to derive the at least one infrared-dark channel together with the near-infrared channel;
providing the multi-channel image comprising at least one of the R-, G-, B-channels (R, G, B) of the RGB image, and the near-infrared channel (NIR), as test input (TI 1 ) to a semantic segmentation neural network, the semantic segmentation neural network trained with a training data set comprising multi-channel images of the test input type to segment the test input (TI 1 ) into pixels associated with the plant elements and pixels not associated with the plant elements; and
segmenting the test input (TI 1 ) using the semantic segmentation neural network resulting in a vegetation coverage map indicating pixels of the test input associated with the plant elements and indicating pixels of the test input not associated with the plant elements.
2 . The method of claim 1 , wherein the at least one infrared-dark channel comprises an infrared-dark subtractive channel (IDS) representing for each pixel the difference between the near-infrared channel value and the minimum value of the available R-, G-, B-channels, and the near-infrared channel of the respective pixel.
3 . The method of claim 1 , wherein the at least one infrared-dark channel comprises an infrared-dark ratio channel (IDR) representing for each pixel the ratio of the near-infrared channel value to the minimum value of the available R-, G-, B-channels, and the near-infrared channel of the respective pixel, the minimum value added to a predefined offset.
4 . The method of claim 1 , further comprising:
prior to providing the multi-channel image to the segmentation neural network, deriving a normalized difference vegetation channel (NDVI) from the R-and estimated NIR channels, the normalized difference vegetation channel forming part of the multi-channel image.
5 . The method of claim 1 , wherein the monotonic activation function is selected from the group of: linear, sigmoid or hyperbolic tangent, and wherein the regression loss function is selected from the group of: Mean Square Error, Mean Absolute Error, Huber Loss, Log-Cosh Loss, and Quantile Loss.
6 . The method of claim 1 , wherein the regression loss function is combined with an adversarial perceptual loss function following a pix2pix architecture where the loss is determined as a weighted sum of the regression loss and the adversarial perceptual loss.
7 . The method of claim 1 , wherein the segmentation neural network is extended to classify the pixels associated with the plant elements into multiple plant classes with each plant representing a particular plant species.
8 . The method of claim 1 , wherein the semantic regression neural network is based on a fully convolutional DenseNet architecture.
9 . The method of claim 1 , wherein the semantic segmentation neural network has an input layer of size M×N×K, where M and N represent the width and height of a test input multi-channel image, and K represents the number of channels of the multi-channel image, and further has a final layer which is composed by two output channels with each output channel of size M and N, one output channel containing the estimate for the vegetation coverage segmentation class and the other output channel containing the estimate for non-vegetation classes, the final layer followed by a softmax activation layer to ensure mutually exclusiveness of the vegetation coverage segmentation class and the class containing the non-vegetation classes.
10 . The method of claim 9 , wherein the semantic segmentation neural network is minimized over any of the loss functions: categorical cross-entropy loss, weighted categorical cross-entropy, balanced cross-entropy, Focal loss, Focal loss plus Dice loss, Tversky loss, Lovász-Softmax.
11 . A computer program product for estimating the vegetation coverage in the real-world environment, the computer program product, when loaded into a memory of a computing device and executed by at least one processor of the computing device, causing the at least one processor to execute the steps of the computer-implemented method according to claim 1 .
12 . A computer system for estimating the vegetation coverage in the real-world environment, the computer system comprising a memory and at least one processor and further comprising software modules that, when executed by the at least one processor, cause the computer system to perform the steps of the computer-implemented method according to claim 1 .