IP Library Granted Patent US 11,640,681
Granted Patent B2
US 11,640,681 · App. 17/011,651 · Granted May 2, 2023

Retinal encoder for machine vision

Inventors: Sheila Nirenberg (New York, NY); Iliya Bomash (Brooklyn, NY)
Assignee: CORNELL UNIVERSITY
G06V10/449G06N3/049G06T7/0012G06T9/00G06T9/002G06T9/007G06V10/454H04N19/60H04N19/62H04N19/85G06T2207/20024G06T2207/20048G06T2207/20084G06T2207/30041G06V2201/03
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,640,681
App. No.
17/011,651
Granted
May 2, 2023
Kind
B2
Abstract

A method is disclosed including: receiving raw image data corresponding to a series of raw images; processing the raw image data with an encoder to generate encoded data, where the encoder is characterized by an input/output transformation that substantially mimics the input/output transformation of one or more retinal cells of a vertebrate retina; and applying a first machine vision algorithm to data generated based at least in part on the encoded data.

Claims (53)

1. A method including:

applying, by an encoding module, a spatiotemporal transformation to image data to generate retinal output cell response values;

generating, by the encoding module, encoded data based on the retinal output cell response values;

generating, by the encoding module, a series of images based on the encoded data; and

applying, by a machine vision module, a machine vision algorithm to the encoded data, wherein applying the machine vision algorithm comprises:

segmenting each image of the series of images into a plurality of regions;

classifying a parameter of each region of the plurality of regions; and

performing a behavioral task based on the classified parameter of each region.

2. The method of claim 1 , further comprising:

monitoring, by a controller, performance of the machine vision algorithm; and

adjusting, by the controller, the machine vision algorithm based on the monitored performance.

3. The method of claim 2 , wherein adjusting the machine vision algorithm comprises iteratively adjusting one or more parameters until the error rate of the machine vision algorithm satisfies a threshold level.

4. The method of claim 3 , wherein iteratively adjusting the one or more parameters comprises iteratively adjusting the one or more parameters of the machine vision algorithm until an incremental increase in performance per iteration falls below a threshold level.

5. The method of claim 2 , wherein the machine vision algorithm comprises an artificial neural network, and wherein adjusting the machine vision algorithm comprises changing a plurality of connections in an artificial neural network.

6. The method of claim 1 , wherein the plurality of regions do not overlap.

7. The method of claim 1 , further comprising determining pixel values in the images based on the encoded data, wherein determining the pixel values includes determining a pixel intensity or color indicative of a retinal cell response, and wherein the data indicative of a retinal cell response is indicative of at least one of a retinal cell firing rate, a retinal cell output pulse train, and a generator potential.

8. The method of claim 1 , wherein the parameter comprises an optical flow speed of a given region.

9. The method of claim 1 , wherein classifying the parameter of each region comprises:

transmitting a pair of consecutive retinal images for the region to the machine vision module; and

applying the pair of consecutive retinal images to a neural network.

10. The method of claim 1 , wherein the behavioral task comprises a navigational determination.

11. The method of claim 1 , wherein the behavioral task comprises at least one of a pattern recognition task, a motion analysis task, and a modeling task.

12. The method of claim 1 , wherein the behavioral task comprises an event detection task.

13. The method of claim 1 , wherein the behavioral task comprises a facial recognition task.

14. An apparatus including:

a memory storage device configured to store image data corresponding to a series of images; and

a processor operably coupled with the memory and programmed to:

receive the image data corresponding to the series of images;

generate encoded data from the image data, wherein, to generate the encoded data, the processor is configured to:

apply a spatiotemporal transformation to the image data to generate retinal output cell response values;

generate the encoded data based on the retinal output cell response values;

generate a series of images based on the encoded data; and

a machine vision module configured to apply a machine vision algorithm to the encoded data, wherein to apply the machine vision algorithm the machine vision module is configured to:

segment each image of the series of images into a plurality of regions;

classify a parameter of each region of the plurality of regions; and

perform a behavioral task based on the classified parameter of each region.

15. The apparatus of claim 14 , further comprising a controller configured to:

monitor performance of the machine vision algorithm; and

adjust the machine vision algorithm based on the monitored performance.

16. The apparatus of claim 15 , wherein, to adjust the machine vision algorithm, the controller is configured to iteratively adjust one or more parameters until the error rate of the machine vision algorithm satisfies a threshold level.

17. The apparatus of claim 14 , wherein the plurality of regions do not overlap.

18. The apparatus of claim 14 , wherein the parameter comprises an optical flow speed of a given region.

19. A non-transitory computer-readable medium having computer-executable instructions for implementing operations comprising:

applying a spatiotemporal transformation to image data to generate retinal output cell response values;

generating encoded data based on the retinal output cell response values;

generating a series of images based on the encoded data; and

applying a machine vision algorithm to the encoded data, wherein applying the machine vision algorithm comprises:

segmenting each image of the series of images into a plurality of regions;

classifying a parameter of each region of the plurality of regions; and

performing a behavioral task based on the classified parameter of each region.

20. The non-transitory computer-readable medium of claim 19 , further comprising computer-executable instructions for implementing operations comprising:

monitoring performance of the machine vision algorithm; and

adjusting the machine vision algorithm based on the monitored performance, and wherein adjusting the machine vision algorithm comprises iteratively adjusting one or more parameters until the error rate of the machine vision algorithm satisfies a threshold level.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 25, 2022
From: NIRENBERG, SHEILA; BOMASH, ILLYA
To: CORNELL UNIVERSITY
Reel/Frame 060605/0935 →
Continuity (6)
Continuation 16421147 · May 23, 2019
Continuation 15408178 · Jan 17, 2017
Continuation 14239828
Provisional Application 61657406 · Jun 8, 2012
Provisional Application 61527493 · Aug 25, 2011
Related Publication 20200401837A1 · Dec 24, 2020