Saliency-based apparatus and methods for visual prostheses
The present invention is a saliency-based apparatus and methods for visual prostheses. A saliency-based component processes video data output by a digital signal processor before the video data are input to the retinal stimulator. In a saliency-based method, an intensity stream is extracted from an input image, feature maps based on the intensity stream are developed, plural most salient regions of the input image are detected and one of the regions is selected as a highest saliency region.
1. A visual prosthesis comprising:
an image processor to process video data from an imager, including a saliency module for separating the video data into a plurality of video streams, including the saliency module selecting at least one video stream from a group containing luminance, color, texture, differently oriented structure, and motion and producing a selected at least one video stream;
a visual stimulator to stimulate visual neurons on a patient;
the saliency module further including a saliency-based component to process the selected at least one video stream before the video data are input to the visual stimulator, the saliency-based component being configured to apply a saliency algorithm to the selected at least one video stream and alter the video data according to the saliency algorithm; and
means for selectively activating electrodes to provide a cue to a visual prosthesis subject as to which region, including regions outside the field of view of the subject, is most salient.
2. The visual prosthesis of claim 1 , wherein the image processor and the saliency-based component are part of a video data processing unit.
3. The visual prosthesis of claim 1 , wherein the image processor and the saliency-based component operate in parallel.
4. The visual prosthesis of claim 1 , wherein the image processor is a digital signal processor (DSP), a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC).
5. The visual prosthesis of claim 1 , wherein the saliency-based component operates on the video data after or before the image processor.
6. The visual prosthesis of claim 1 , wherein the saliency-based component is a software component.
7. The visual prosthesis of claim 1 , wherein the image processor runs on a limited amount of power compliant with power limits of devices operating on visual prostheses.
8. The visual prosthesis of claim 1 , wherein the image processor has a processing rate of 3 to 30 frames per second.
9. The visual prosthesis of claim 8 , wherein the image processor has a processing rate of 3 to 5 frames per second.
10. The visual prosthesis of claim 1 , wherein the saliency-based component, only processes an intensity stream of the video data.
11. The visual prosthesis of claim 1 , wherein the saliency-based component is adapted to be turned on and turned off on demand.
12. The visual prosthesis of claim 11 , wherein turning on and turning off the saliency-based component on demand is effected by a patient using the visual prosthesis.
13. The visual prosthesis of claim 1 , wherein the video data pertain to multiple images.
14. The visual prosthesis of claim 13 , wherein the saliency-based component cues on motion.
15. The visual prosthesis of claim 1 , wherein the imager is sensitive to visible light, infrared, ultraviolet, X-ray spectra, or ultrasound.
16. The visual prosthesis of claim 1 , further comprising a pattern recognition component.
17. The visual prosthesis of claim 16 , wherein the pattern recognition component comprises a table of salient stored patterns for comparison with the results of the saliency component.
18. The visual prosthesis according to claim 1 , wherein the plurality of video streams includes intensity.
19. The visual prosthesis according to claim 1 , wherein the plurality of video streams includes color.
20. The visual prosthesis according to claim 1 , wherein the plurality of video streams includes orientation.
21. The visual prosthesis according to claim 1 , wherein the plurality of video streams includes movement.