Complementary learning for multi-modal saliency detection

1. A method of saliency detection on images including both appearance and depth information, the method comprising:

processing appearance information of an image with a first saliency encoder to extract appearance features from the image;

processing depth information of the image with a second saliency encoder to extract depth features from the image;

processing the extracted appearance features with a first latent feature module to generate a latent appearance feature;

processing the extracted depth features with a second latent feature module to generate a latent depth feature;

in a first appearance complimentary learning branch, fusing at least a portion of the extracted appearance features with the latent depth feature to generate an appearance saliency prediction;

in a second appearance complimentary learning branch, fusing at least a portion of the extracted depth features with the latent appearance feature to generate a depth saliency prediction; and

processing the extracted appearance features, extracted depth features, appearance saliency prediction and depth saliency prediction to obtain a combined saliency prediction providing an indication of salient locations within the image.

2. The method of claim 1 , wherein the first latent feature module and the second latent feature module are trained based on mutual information between latent appearance features of training images and latent depth features of the training images.

3. The method of claim 2 , wherein the mutual information measures a difference between entropy of the latent appearance features of the training images and the latent depth features of the training images.

4. The method of claim 3 , wherein an objective function used in training the first and second latent feature modules is given by:

= ce ( P,Y )+λ 1 ce ( P f ,Y )+λ 2 ce ( P a ,Y )+λ 3 ce ( P g ,Y )+λ mi ( z a ,z g ),

wherein , ce , and mi are objective functions,

wherein P, P f , P a , and P g are saliency predictions,

wherein z a and z g are latent features;

wherein Y is a parameter set of the saliency decoder, and

wherein λ 1 , λ 2 , λ 3 , and λ are constants.

5. The method of claim 1 , wherein the first latent feature module and the second latent feature module each comprise five convolution layers and two fully connected layers.

6. The method of claim 5 , wherein each of the five convolution layers of the first latent feature module and the second latent feature module have a kernel size of 4×4 and a stride size of 2.

7. The method of claim 1 , wherein the first and second saliency encoders have a common network architecture.

8. The method of claim 7 , wherein the common network architecture is based on a ResNet 50 network.

9. The method of claim 7 , wherein the common network architecture comprises four convolutional groups.

10. The method of claim 9 , wherein the common network architecture further comprises an additional convolutional layer of kernel size 3×3 after each convolutional group.

11. A non-transitory computer readable medium having stored thereon instructions, which when executed by a processor of a computing system, configure the computing system to perform a method of saliency detection on images including both appearance and depth information, the method comprising:

processing appearance information of an image with a first saliency encoder to extract appearance features from the image;

processing depth information of the image with a second saliency encoder to extract depth features from the image;

processing the extracted appearance features with a first latent feature module to generate a latent appearance feature;

processing the extracted depth features with a second latent feature module to generate a latent depth feature;

in a first appearance complimentary learning branch, fusing at least a portion of the extracted appearance features with the latent depth feature to generate an appearance saliency prediction;

in a second appearance complimentary learning branch, fusing at least a portion of the extracted depth features with the latent appearance feature to generate a depth saliency prediction; and

processing the extracted appearance features, extracted depth features, appearance saliency prediction and depth saliency prediction to obtain a combined saliency prediction providing an indication of salient locations within the image.

12. The non-transitory computer readable medium of claim 11 , wherein the first latent feature module and the second latent feature module are trained based on mutual information between latent appearance features of training images and latent depth features of the training images.

13. The non-transitory computer readable medium of claim 12 , wherein the mutual information measures a difference between entropy of the latent appearance features of the training images and the latent depth features of the training images.

14. The non-transitory computer readable medium of claim 13 , wherein an objective function used in training the first and second latent feature modules is given by:

= ce ( P,Y )+λ 1 ce ( P f ,Y )+λ 2 ce ( P a ,Y )+λ 3 ce ( P g ,Y )+λ mi ( z a ,z g ),

wherein , ce , and mi are objective functions,

wherein P, P f , P a , and P g are saliency predictions,

wherein z a and z g are latent features,

wherein Y is a parameter set of the saliency decoder, and

wherein λ 1 , λ 2 , λ 3 , and λ are constants.

15. The non-transitory computer readable medium of claim 14 , wherein the first latent feature module and the second latent feature module each comprise five convolution layers and two fully connected layers.

16. The non-transitory computer readable medium of claim 15 , wherein each of the five convolution layers of the first latent feature module and the second latent feature module have a kernel size of 4×4 and a stride size of 2.

17. The non-transitory computer readable medium of claim 15 , wherein the first and second saliency encoders have a common network architecture.

18. The non-transitory computer readable medium of claim 17 , wherein the common network architecture is based on a ResNet 50 network.

19. The non-transitory computer readable medium of claim 17 , wherein the common network architecture comprises four convolutional groups.

20. The non-transitory computer readable medium of claim 19 , wherein the common network architecture further comprises an additional convolutional layer of kernel size 3×3 after each convolutional group.

21. A system saliency detection on images including both appearance and depth information, the system comprising:

a processor for executing instructions; and

a memory for storing instructions, which when executed by the processor configure the system to implement a method of saliency detection on images including both appearance and depth information, the method comprising:

processing appearance information of an image with a first saliency encoder to extract appearance features from the image;

processing depth information of the image with a second saliency encoder to extract depth features from the image;

processing the extracted appearance features with a first latent feature module to generate a latent appearance feature;

processing the extracted depth features with a second latent feature module to generate a latent depth feature;

in a first appearance complimentary learning branch, fusing at least a portion of the extracted appearance features with the latent depth feature to generate an appearance saliency prediction;

in a second appearance complimentary learning branch, fusing at least a portion of the extracted depth features with the latent appearance feature to generate a depth saliency prediction; and

processing the extracted appearance features, extracted depth features, appearance saliency prediction and depth saliency prediction to obtain a combined saliency prediction providing an indication of salient locations within the image.