Saliency-based apparatus and methods for visual prostheses
The present invention relates to a saliency-based apparatus and methods for visual prostheses. A saliency-based component processes video data output by a digital signal processor before the video data are input to the retinal stimulator. In a saliency-based method, an intensity stream is extracted from an input image, feature maps based on the intensity stream are developed, plural most salient regions of the input image are detected and one of the regions is selected as a highest saliency region.
1. A method of operating an image processor suitable to convert an input image to stimulation data to be provided to a subject through a visual stimulator adapted to be implanted, the image processor performing the steps, comprising:
extracting only an intensity stream from the input image;
developing feature maps based on the intensity stream only;
detecting a plurality of most salient regions of the input image based on the feature maps;
selecting one salient region of the most salient regions as a highest saliency region;
producing an output image based on the input image and the highest saliency region by highlighting the highest saliency region in the output image;
selecting a field of view from the output image;
downsampling the field of view to derive the stimulation data; and
providing the stimulation data to the visual stimulator.
2. The method of claim 1 , wherein the plurality of most salient regions are three most salient regions.
3. The method of claim 1 , further comprising constructing an image pyramid for the intensity stream after extracting only an intensity stream from the input image and before developing feature maps based on the intensity stream.
4. The method of claim 1 , further comprising constructing conspicuity maps after developing feature maps based on the intensity stream and before detecting a plurality of most salient regions of the input image based on the feature maps.
5. The method of claim 1 , further comprising averaging each region of the plurality of most salient regions after detecting the plurality of most salient regions and before selecting one salient region of the most salient regions as a highest saliency region.
6. The method of claim 1 , the method being performed in absence of video data normalization.
7. The method of claim 1 , wherein also information in the infrared, ultraviolet, or X-ray spectra, or ultrasound information is provided.
8. The method of claim 1 , wherein the method is applied to the input image inside and outside the field of view of the subject.
9. The method of claim 1 , further comprising separating the input image into multiple regions based on saliency, and processing video data separately in each region.
10. The method of claim 9 , wherein separating the input image into multiple regions generates high saliency regions, medium saliency regions and low saliency regions.
11. The method of claim 10 , wherein video data in low saliency regions are processed together, while video data in high saliency regions and medium saliency regions are processed independently of each other by the image processor.
12. The method of claim 10 , wherein the video data in high saliency regions are processed before the medium saliency regions and the video data in medium saliency regions are processed before the low saliency regions.
13. The method of claim 1 , wherein regions of the input image falling below a saliency threshold are not included in the stimulation data.