IP Library Granted Patent US 10,349,035
Granted Patent B2
US 10,349,035 · App. 14/941,826 · Granted Jul 9, 2019

Automatically scanning and representing an environment having a plurality of features

Inventors: Jianjun Wang (West Hartford, CT); Biao Zhang (West Hartford, CT); Carlos Martinez (South Windsor, CT); Carlos W. Morato (Avon, CT); Remus Boca (Simsbury, CT)
Assignee: ABB Schweiz AG
H04N13/204G06K9/00791G06T7/75H04N5/232G06K9/00201G06T2207/10016G06T2207/10028G06T2207/30252
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,349,035
App. No.
14/941,826
Granted
Jul 9, 2019
Kind
B2
Abstract

Automatic scanning and representing an environment having a plurality of features, for example, includes scanning the environment along a scanning path, interspersing a plurality of localized scanning of the plurality of features in the environment during the scanning along the scanning path of the environment wherein the interspersed localized scanning of the plurality of features in the environment being different from the scanning the environment along the scanning path, and obtaining a representation of at least a portion of the environment based on the scanning of the environment and the interspersed localized scanning of the plurality of features in the environment.

Claims (47)

1. A computer-implemented method comprising:

scanning an environment along a scanning path;

interspersing a plurality of localized scanning of a plurality of candidate features in the environment during the scanning along the scanning path of the environment, the interspersed localized scanning of the plurality of candidate features in the environment being different from the scanning the environment along the scanning path; and

obtaining a representation of at least a portion of the environment based on the scanning of the environment and the interspersed localized scanning of the plurality of candidate features in the environment,

wherein interspersing the plurality of localized scanning of the plurality of candidate features includes:

generating a set of data corresponding to one candidate feature of the plurality of candidate features by scanning one candidate of the plurality of candidate features a plurality of times,

determining the one candidate of the plurality of candidate features exceeds a certainty threshold using the set of data,

recording the candidate location, a robot location, and a robot end effector orientation in response to the determining, and

updating a robot forward kinematic model for the robot end effector using the recorded robot location and robot end effector orientation.

2. The computer-implemented method of claim 1 wherein the interspersing the plurality of localized scanning of the plurality of candidate features in the environment comprises identifying one of the plurality of candidate features during the scanning of the scanning path of a first portion of the environment, scanning a localized portion of the environment having the identified one of the plurality of candidate features in the first portion of the environment, continuing the scanning along the scanning path of the environment another portion of the environment, identifying another one of the plurality of candidate features during the scanning of the scanning path of the another portion of the environment, and scanning another localized portion of the environment having the identified another of the plurality of candidate features in the another portion of the environment.

3. The computer-implemented method of claim 1 wherein the scanning along the scan path of the environment is at a first resolution or speed and the scanning of the localized portions of the environment is at a second resolution or speed different from the first resolution or speed.

4. The computer-implemented method of claim 1 wherein the scanning of the environment along the scanning path is performed using a first scanning parameter, the interspersing comprises using at least one criteria for identifying the plurality of candidate features along the first scanning path, and the scanning of the localized portions is based on the plurality of identified candidate features and performed using a second scanning parameter different from the first scanning parameter.

5. The computer-implemented method of claim 1 wherein the scanning along the scanning path employs a 3D imager and the scanning the localized portions employs the 3D imager.

6. The computer-implemented method of claim 1 wherein the obtaining the representation comprises obtaining a first plurality of frames of data of the scanning along the scan path of the environment and a plurality of frames of data of the localized portions of the plurality of candidate features of the environment, and merging together data from the first plurality of frames of data of the scanning along the scan path of the environment and the plurality of frames of data of the localized portions of the plurality of candidate features of the environment.

7. The computer-implemented method of claim 1 wherein the scanning the environment along the scanning path comprises updating the statistical model of feature location prediction based on prior feature detection results, and estimating a certainty of a current detected feature using the updated statistical model of feature location prediction and a current feature detection result, and wherein the obtaining the representation of the interspersed localized scanning of the plurality of candidate features in the environment is based on the estimated certainty of the current detected feature being above a threshold.

8. The computer-implemented method of claim 1 wherein the scanning along the scan path comprises scan across a plane.

9. The computer-implemented method of claim 1 wherein the scanning the localized portions comprises scanning the localized portions from a plurality of orientations.

10. The computer-implemented method of claim 1 wherein the scanning along a scan path and the scanning of the localized portions comprises employing a robotically supported movable imager.

11. The computer-implemented method of claim 1 wherein the environment comprises a rock wall and the plurality of candidate features comprise a plurality of elongated holes defining axes for receiving charges for mining.

12. The computer-implemented method of claim 11 wherein the scanning the wall along the scanning path comprises updating the statistical model of feature location prediction and a drilling map matching based on prior hole detection results, and estimating a certainty of a current detected hole using the updated statistical model of feature location prediction, the updated drilling map matching, and a current hole detection result, and wherein the obtaining the representation of the interspersed localized scanning of the plurality of holes in the wall is based on the estimated certainty of the current detected feature being above a threshold.

13. The computer-implemented method of claim 11 further comprising determining orientations of the axes of the plurality of holes based on the scanning of the localized portions.

14. A system comprising:

a memory; and

a physical processor in communications with the memory,

wherein the system is configured to perform a method, said method comprising:

scanning an environment along a scanning path;

interspersing a plurality of localized scanning of a plurality of candidate features in the environment during the scanning along the scanning path of the environment, the interspersed localized scanning of the plurality of candidate features in the environment being different from the scanning the environment along the scanning path; and

obtaining a representation of at least a portion of the environment based on the scanning of the environment and the interspersed localized scanning of the plurality of candidate features in the environment,

wherein interspersing the plurality of localized scanning of the plurality of candidate features includes:

generating a set of data corresponding to one candidate feature of the plurality of candidate features by rescanning around one candidate of the plurality of candidate features,

determining the one candidate of the plurality of candidate features exceeds a certainty threshold using the set of data,

recording the candidate location, a robot location, and a robot end effector orientation in response to the determining, and

updating a robot forward kinematic model for the robot end effector using the recorded robot location and robot end effector orientation.

15. A non-transitory computer readable storage medium readable by a processor comprising:

instructions stored on the non-transitory computer readable storage medium for execution by the processor for performing a method comprising:

scanning an environment along a scanning path;

interspersing a plurality of localized scanning of a plurality of candidate features in the environment during the scanning along the scanning path of the environment, the interspersed localized scanning of the plurality of candidate features in the environment being different from the scanning the environment along the scanning path; and

obtaining a representation of at least a portion of the environment based on the scanning of the environment and the interspersed localized scanning of the plurality of candidate features in the environment,

wherein interspersing the plurality of localized scanning of the plurality of candidate features includes:

generating a set of data corresponding to one candidate feature of the plurality of candidate features by reorienting around one candidate of the plurality of candidate features,

determining the one candidate of the plurality of candidate features exceeds a certainty threshold using the set of data,

recording the candidate location, a robot location, and a robot end effector orientation in response to the determining, and

updating a robot forward kinematic model for the end effector using the recorded robot location and robot end effector orientation.

16. The computer-implemented method of claim 1 wherein interspersing the plurality of localized scanning of the plurality of candidate features includes:

generating a second set of data corresponding to a second candidate feature of the plurality of candidate features by scanning the second candidate of the plurality of candidate features a plurality of times,

determining the second candidate of the plurality of candidate features is a false positive detection using the second set of data, and

updating a statistical model of feature location prediction using feedback including the second set of data corresponding to the false positive detection in response to determining the second candidate of the plurality of candidate features is a false positive detection.

Assignments (3)
MERGER Recorded Nov 15, 2016
From: ABB TECHNOLOGY LTD.
To: ABB SCHWEIZ AG
Reel/Frame 040621/0792 →
MERGER Recorded Oct 20, 2016
From: ABB TECHNOLOGY AG
To: ABB SCHWEIZ AG
Reel/Frame 040441/0836 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 7, 2015
From: WANG, JIANJUN; ZHANG, BIAO; MARTINEZ, CARLOS; MORATO, CARLOS W.; BOCA, REMUS
To: ABB TECHNOLOGY AG
Reel/Frame 037227/0362 →
Continuity (1)
Related Publication 20170142403A1 · May 18, 2017
Cited By (1)
US 12,674,386