IP Library Granted Patent US 9,626,566
Granted Patent B2
US 9,626,566 · App. 14/662,657 · Granted Apr 18, 2017

Methods and apparatus for autonomous robotic control

Inventors: Massimiliano Versace (Boston, MA); Anatoly Gorshechnikov (Newton, MA); Gennady Livitz (Belmont, MA); Jesse Palma (Somerville, MA)
Assignee: Neurala, Inc.
G06K9/00664B25J9/1697G06K9/3241G06K9/4628G06T7/004G06T7/0044G06T7/0081G06T7/2093G05B2219/39082G06N3/008G06N3/049G06T2207/20141G06T2207/20144G06T2207/30244
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,626,566
App. No.
14/662,657
Granted
Apr 18, 2017
Kind
B2
Abstract

Sensory processing of visual, auditory, and other sensor information (e.g., visual imagery, LIDAR, RADAR) is conventionally based on “stovepiped,” or isolated processing, with little interactions between modules. Biological systems, on the other hand, fuse multi-sensory information to identify nearby objects of interest more quickly, more efficiently, and with higher signal-to-noise ratios. Similarly, examples of the OpenSense technology disclosed herein use neurally inspired processing to identify and locate objects in a robot's environment. This enables the robot to navigate its environment more quickly and with lower computational and power requirements.

Claims (46)

1. A system comprising:

an image sensor to acquire a plurality of images of at least a portion of an environment surrounding a robot; and

a processor, operably coupled to the image sensor, to:

translate each image in the plurality of images from a frame of reference of the image sensor to an allocentric frame of reference;

identify a position, in the allocentric frame of reference, of an object appearing in at least one image in the plurality of images; and

determine if the object appears in at least one other image in the plurality of images based on the position, in the allocentric frame of reference, of the object.

2. The system of claim 1 , wherein the processor is configured to translate the at least one image from the frame of reference of the image sensor to an allocentric frame of reference by:

translating each image in the plurality of images from the frame of reference of the image sensor to an egocentric frame of reference based on a position and/or an orientation of the image sensor in the egocentric frame of reference, the egocentric frame of reference being defined with respect to the robot; and

translating each image in the plurality of images from the egocentric frame of reference to the allocentric frame of reference.

3. The system of claim 1 , wherein the processor is configured to identify the position in the allocentric frame of reference of the object by:

generating a segmented version of the at least one image in the plurality of images; and

determining at least one spatial shroud fitting a form of the object based at least in part on the segmented version of the at least one image.

4. The system of claim 3 , wherein the processor is configured to determine if the object appears in at least one other image in the plurality of images at least in part on by:

applying the at least one spatial shroud to the other image in the plurality of images.

5. The system of claim 1 , wherein the processor is configured to:

map the position, in the allocentric frame of reference, of the object to coordinates in the frame of reference of the image sensor; and

determine a change to a position and/or an orientation of the image sensor based at least in part on the coordinates in the frame of reference of the image sensor.

6. The system of claim 5 , further comprising:

an actuator, operably coupled to the processor and to the image sensor, to adjust a field of view of the image sensor based at least in part on the change to the position and/or the orientation of the image sensor, and

wherein the image sensor is configured to acquire a subsequent image in the plurality of images in response to adjustment of the field of view.

7. A method of locating an object with respect to a robot, the method comprising:

(A) acquiring, with a image sensor coupled to the robot, a plurality of images of at least a portion of an environment surrounding the robot;

(B) automatically translating each image in the plurality of images from a frame of reference of the image sensor to an allocentric frame of reference;

(C) identifying a position, in the allocentric frame of reference, of an object appearing in at least one image in the plurality of images; and

(D) determining if the object appears in at least one other image in the plurality of images based on the position, in the allocentric frame of reference, of the object.

8. The method of claim 7 , wherein (B) comprises:

(B1) translating each image in the plurality of images from the frame of reference of the image sensor to an egocentric frame of reference based on a position and/or an orientation of the image sensor in the egocentric frame of reference, the egocentric frame of reference being defined with respect to the robot; and

(B2) translating each image in the plurality of images from the egocentric frame of reference to the allocentric frame of reference.

9. The method of claim 7 , wherein (C) comprises:

(C1) generating a segmented version of a first image in the plurality of images; and

(C2) determining a spatial shroud fitting a form of the object based at least in part on the segmented version of the first image.

10. The method of claim 9 , further comprising:

(E) mapping the position, in the allocentric frame of reference, of the object to coordinates in the frame of reference of the image sensor; and

(F) determining a change to a position and/or an orientation of the image sensor based at least in part on the coordinates in the frame of reference of the image sensor.

11. The method of claim 10 , wherein (F) further comprises positioning and/or orienting the image sensor away from the object.

12. The method of claim 10 , wherein (F) further comprises positioning and/or orienting the image sensor to acquire another image of the object.

13. The method of claim 10 , wherein:

(D) comprises translating and/or transforming the spatial shroud based at least in part on the change in the position and/or the orientation of the image sensor determined in (F), and

(C) comprises determining if the spatial shroud fits the form of the object in a segmented version of a second image in the plurality of images.

14. The method of claim 13 , wherein (D) further comprises:

(D3) identifying at least one feature of the object in the first image; and

(D4) comparing the at least one feature to a plurality of features identified in other images in the plurality of images.

15. The method of claim 10 , further comprising:

(G) adjusting a field of view of the image sensor based at least in part on the change to the position and/or the orientation of the image sensor.

16. The method of claim 15 , wherein (A) comprises:

acquiring a subsequent image in the plurality of images in response to adjustment of the field of view.

Assignments (7)
CHANGE OF NAME Recorded May 28, 2026
From: NEURAL HOLDINGS LLC
To: NEURAL NAVIGATION LLC
Reel/Frame 075641/0922 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 26, 2026
From: NEURALA INC.
To: NEURAL HOLDINGS LLC
Reel/Frame 074758/0527 →
LICENSE Recorded Feb 2, 2018
From: NEURALA, INC.
To: UNITES STATES OF AMERICA AS REPRESENTED BY THE ADMINISTRATOR OF NASA
Reel/Frame 045241/0059 →
LICENSE Recorded Aug 17, 2017
From: TRUSTEES OF BOSTON UNIVERSITY
To: UNITED STATES OF AMERICA AS REPRESENTED BY THE ADMINISTRATOR OF NASA
Reel/Frame 043581/0682 →
CONFIRMATORY LICENSE Recorded Jan 14, 2016
From: NEURALA, INC.
To: AFRL/RIJ
Reel/Frame 037486/0964 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 13, 2015
From: LIVITZ, GENNADY; PALMA, JESSE
To: TRUSTEES OF BOSTON UNIVERSITY
Reel/Frame 036343/0707 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 13, 2015
From: VERSACE, MASSIMILIANO; GORCHETCHNIKOV, ANATOLI
To: NEURALA LLC
Reel/Frame 036318/0497 →
Continuity (2)
Provisional Application 61955756 · Mar 19, 2014
Related Publication 20150269439A1 · Sep 24, 2015