IP Library Granted Patent US 12,204,314
Granted Patent B2
US 12,204,314 · App. 17/454,217 · Granted Jan 21, 2025

Method and apparatus for improved auto-calibration of a robotic cell

Inventors: Ronald Poelman (Wassenaar, NL); Barrett Clark (Seattle, WA); Oytun Akman (Oakland, CA); Matthew Brown (Seattle, WA)
Assignee: Bright Machines, Inc.
G05B19/41815B25J9/1653B25J9/1671B25J9/1692B25J19/023G06T7/80G05B2219/39016
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,204,314
App. No.
17/454,217
Granted
Jan 21, 2025
Kind
B2
Abstract

A robotic cell calibration method comprising a robotic cell system having elements comprising: one or more cameras, one or more sensors, components, and a robotic arm. The method comprises localizing positions of the one or more cameras and components relative to a position of the robotic arm using a common coordinate frame, moving the robotic arm in a movement pattern, and using the cameras and sensors to determine robotic arm position at multiple times during the movement. The method includes identifying a discrepancy in robotic arm position between a predicted position and the determined position in real time, and computing, by an auto-calibrator, a compensation for the identified discrepancy, the auto-calibrator solving for the elements in the robotic cell system as a system. The method includes modifying actions of the robotic arm in real time during the movement based on the compensation.

Claims (60)

1. A robotic cell calibration method comprising:

providing a robotic cell system having elements comprising: one or more cameras, one or more sensors, components, and a robotic arm;

localizing positions of the one or more cameras and components relative to a position of the robotic arm within the robotic cell using a common coordinate frame;

moving the robotic arm in a movement pattern under a first load, and using the cameras and sensors to determine robotic arm position at multiple times during the movement;

moving the robotic arm in the movement pattern under a second load and using the cameras and sensors to determine the robotic arm position at multiple times during the movement;

identifying a discrepancy in robotic arm position between a predicted position in the movement pattern and the determined robotic arm position in real time, for the first load and the second load;

computing, by an auto-calibrator, a compensation for the identified discrepancy, the auto-calibrator solving for the elements in the robotic cell system as a system; and

modifying actions of the robotic arm in real time during the movement based on the compensation to reduce the identified discrepancy in the robotic arm position.

2. The robotic cell calibration method of claim 1 , further comprising:

forming a virtual representation of the robotic cell system that tracks at least some actions of the robotic cell using the localized position of the cameras, the sensors, and the robotic arm, the virtual representation used by the auto-calibrator.

3. The robotic cell calibration method of claim 1 , further comprising:

determining if robotic arm position is within a determined accuracy range; and

initiating a recalibration by the auto-calibrator when the robotic arm is outside the determined accuracy range.

4. The robotic cell calibration method of claim 1 , further comprising providing continuous calibration of robotic arm position as the robotic arm moves through movement patterns.

5. The robotic cell calibration method of claim 1 , further comprising performing a pre-use calibration comprising:

selecting robotic arm movement patterns to improve position localization of components visible in an observable volume of the robotic cell; and

performing the selected robotic arm movement patterns while an object including one or more fiducials is attached to the robotic arm.

6. The robotic cell calibration method of claim 1 , further comprising calibrating the one or more cameras, the calibrating accounting for camera lens distortion.

7. The robotic cell calibration method of claim 1 , further comprising performing a pre-calibration comprising:

building a position accuracy map with position metrics for the robotic arm; and

adjusting the movement patterns based on the position accuracy map.

8. The robotic cell calibration method of claim 1 , wherein the computing comprises modelling dynamic and static components in the robotic cell system.

9. The robotic cell calibration method of claim 1 , further comprising:

performing idle time calibration by performing calibration refinement steps on idle cycles of a processor while the robotic cell is in use, the idle time calibration determining whether the robotic cell remains calibrated.

10. The robotic cell calibration method of claim 1 , wherein the method comprises an initial calibration when the robotic cell is not in use.

11. The robotic cell calibration method of claim 10 , wherein the method comprises an in-use calibration, applied while the robotic cell is in use, to update calibration to address changes in the robotic cell during the use.

12. A robotic cell system comprising:

one or more cameras;

one or more sensors;

a robotic arm;

a processor configured to:

perform a pre-calibration comprising:

building a position accuracy map with position metrics for the robotic arm; and

adjusting a movement pattern based on the position accuracy map;

localize positions of cameras and components relative to a position of the robotic arm within the robotic cell using a common coordinate frame;

control movement of the robotic arm in the movement pattern under a first load and using the cameras and the sensors to determine robotic arm position at multiple times during the movement;

control movement of the robotic arm in the movement pattern under a second load and using the cameras and the sensors to determine robotic arm position at multiple times during the movement;

identify a discrepancy in robotic arm position between a predicted position in the movement pattern and the determined robotic arm position in real time, for the first load and the second load;

compute, by an auto-calibrator, a compensation for the identified discrepancy, the auto-calibrator solving for elements in the robotic cell system as a system; and

modify actions of the robotic arm in real time during the movement based on the compensation to reduce the identified discrepancy in the robotic arm position.

13. The robotic cell system of claim 12 , further comprising:

the processor further configured to form a virtual representation of the robotic cell system that tracks at least some actions of the robotic cell using the localized position of the cameras, the sensors, and the robotic arm, the virtual representation used by the auto-calibrator.

14. The robotic cell system of claim 12 , further comprising the processor configured to determine if robotic arm position is within a determined accuracy range, and initiate a recalibration by the auto-calibrator when the robotic arm is outside the determined accuracy range.

15. The robotic cell system of claim 12 , wherein the system provides a continuous calibration of robotic arm position as the robotic arm moves through movement patterns.

16. The robotic cell system of claim 12 , further comprising:

robotic controls configured to move the robotic arm under varying loads, at varying speeds, and moving along varying paths identify the discrepancy.

17. The robotic cell system of claim 12 , further comprising:

an auto-calibrator configured to perform a pre-use calibration including:

selecting robotic arm movement patterns to improve position localization of components visible in an observable volume of the robotic cell; and

performing the selected robotic arm movement patterns while an object including one or more fiducials is attached to the robotic arm.

18. The robotic cell system of claim 12 , wherein the processor is configured to perform idle time calibration by performing calibration refinement steps on idle cycles of a processor while the robotic cell is in use, the idle time calibration determining whether the robotic cell remains calibrated.

19. A method to calibrate a robotic cell system including a robotic arm and one or more cameras, the method comprising:

moving the robotic arm in a movement pattern under a first load and using the cameras to determine robotic arm positions at multiple times during the movement pattern;

moving the robotic arm in the movement pattern under a second load and using the cameras to determine robotic arm positions at multiple times during the movement pattern;

identifying a discrepancy in robotic arm position between a predicted position in the movement pattern and the determined robotic arm position in real time, for the first load and the second load;

computing, by an auto-calibrator, a compensation for the identified discrepancy, the auto-calibrator solving for elements in the robotic cell system as a system; and

modifying actions of the robotic arm during use based on the compensation to reduce the identified discrepancy in the robotic arm position.

20. The method of claim 19 , further comprising:

creating one or more movement patterns to such that a path traced by the movement pattern is visible to the one or more cameras; and

performing the movement patterns while an object including one or more fiducials is attached to the robotic arm.

Assignments (6)
RELEASE OF SECURITY INTEREST IN INTELLECTUAL PROPERTY COLLATERAL AT REEL/FRAME NO. 67201/0797 Recorded Jun 2, 2025
From: JPMORGAN CHASE BANK, N.A.
To: BRIGHT MACHINES, INC.
Reel/Frame 071851/0097 →
SECURITY INTEREST Recorded Jun 2, 2025
From: BRIGHT MACHINES, INC.; BRIGHT MACHINES AUTOMATION CORP.
To: STIFEL BANK
Reel/Frame 071467/0419 →
TERMINATION AND RELEASE OF INTELLECTUAL PROPERTY SECURITY AGREEMENT AT REEL 061361 FRAME 0151 Recorded Apr 25, 2024
From: SILICON VALLEY BANK, A DIVISION OF FIRST CITIZENS BANK & TRUST COMPANY (SUCCESSOR TO SILICON VALLEY BANK)
To: BRIGHT MACHINES, INC.
Reel/Frame 067238/0943 →
SECURITY INTEREST Recorded Apr 23, 2024
From: BRIGHT MACHINES, INC.
To: JPMORGAN CHASE BANK, N.A.
Reel/Frame 067201/0797 →
SECURITY INTEREST Recorded Oct 10, 2022
From: BRIGHT MACHINES AUTOMATION CORP.; BRIGHT MACHINES, INC.
To: SILICON VALLEY BANK
Reel/Frame 061361/0151 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 10, 2021
From: POELMAN, RONALD; CLARK, BARRETT; AKMAN, OYTUN; BROWN, MATTHEW
To: BRIGHT MACHINES, INC.
Reel/Frame 058077/0724 →
Continuity (2)
Provisional Application 63198757 · Nov 10, 2020
Related Publication 20220147026A1 · May 12, 2022
References Cited (123)
US 7403835B2 · Sandner et al. · 2008 [cited by applicant]
US 7996113B2 · Yuan et al. · 2011 [cited by applicant]
US 8346392B2 · Walser et al. · 2013 [cited by applicant]
US 9508148B2 · Zhang et al. · 2016 [cited by applicant]
US 9643314B2 · Guerin et al. · 2017 [cited by applicant]
US 9671777B1 · Aichele et al. · 2017 [cited by applicant]
US 9707680B1 · Jessen et al. · 2017 [cited by applicant]
US 9796083B2 · Atohira et al. · 2017 [cited by applicant]
US 9811074B1 · Aichele et al. · 2017 [cited by applicant]
US 9840007B1 · Kuffner · 2017 [cited by applicant]
US 9844882B2 · Takizawa et al. · 2017 [cited by applicant]
US 9971914B2 · Maturana et al. · 2018 [cited by applicant]
US 10279480B1 · Holson et al. · 2019 [cited by applicant]
US 10456915B1 · Nikolaev · 2019 [cited by applicant]
US 10475240B2 · Evans et al. · 2019 [cited by applicant]
US 10576635B2 · Ogawa et al. · 2020 [cited by applicant]
US 10596700B2 · Corkum et al. · 2020 [cited by applicant]
US 10773386B2 · Yamauchi et al. · 2020 [cited by applicant]
US 10782668B2 · Bank et al. · 2020 [cited by applicant]
US 10875176B2 · Radrich et al. · 2020 [cited by applicant]
US 11023934B1 · Jacobs et al. · 2021 [cited by applicant]
US 11209798B1 · Michalowski et al. · 2021 [cited by applicant]
US 20060107507A1 · Brose · 2006 [cited by examiner]
US 20070150100A1 · Saraliev · 2007 [cited by examiner]
US 20080013825A1 · Nagatsuka et al. · 2008 [cited by applicant]
US 20090118864A1 · Eldridge et al. · 2009 [cited by applicant]
US 20120265345A1 · Nakahara · 2012 [cited by applicant]
US 20140046635A1 · Kersh et al. · 2014 [cited by applicant]
US 20140148949A1 · Graca · 2014 [cited by examiner]
US 20140297244A1 · Maturana et al. · 2014 [cited by applicant]
US 20150134317A1 · Maturana et al. · 2015 [cited by applicant]
US 20150239127A1 · Barajas et al. · 2015 [cited by applicant]
US 20150321352A1 · Debain et al. · 2015 [cited by applicant]
US 20160239013A1 · Troy · 2016 [cited by examiner]
US 20160243704A1 · Vakanski et al. · 2016 [cited by applicant]
US 20170079725A1 · Hoffman et al. · 2017 [cited by applicant]
US 20180029232A1 · Ouchi et al. · 2018 [cited by applicant]
US 20180126553A1 · Corkum · 2018 [cited by examiner]
US 20180304467A1 · Matsuura et al. · 2018 [cited by applicant]
US 20180315237A1 · Byers et al. · 2018 [cited by applicant]
US 20180356895A1 · Dailey et al. · 2018 [cited by applicant]
US 20190001573A1 · Gulbrandsen et al. · 2019 [cited by applicant]
US 20190015163A1 · Abhari et al. · 2019 [cited by applicant]
US 20190056928A1 · Smal et al. · 2019 [cited by applicant]
US 20190084160A1 · Liu et al. · 2019 [cited by applicant]
US 20190254754A1 · Johnson et al. · 2019 [cited by applicant]
US 20190280405A1 · Iwai et al. · 2019 [cited by applicant]
US 20190291277A1 · Oleynik · 2019 [cited by applicant]
US 20190308320A1 · Konishi · 2019 [cited by applicant]
US 20190339966A1 · Moondhra et al. · 2019 [cited by applicant]
US 20190339967A1 · Moondhra et al. · 2019 [cited by applicant]
US 20190350685A1 · Saghatchi · 2019 [cited by examiner]
US 20190380794A1 · Al Jewad · 2019 [cited by examiner]
US 20190389060A1 · Roy Chaudhuri et al. · 2019 [cited by applicant]
US 20200016758A1 · Keller · 2020 [cited by examiner]
US 20200024853A1 · Furrer et al. · 2020 [cited by applicant]
US 20200030979A1 · Bank et al. · 2020 [cited by applicant]
US 20200078948A1 · Krause · 2020 [cited by examiner]
US 20200086483A1 · Li et al. · 2020 [cited by applicant]
US 20200147794A1 · Kerrick et al. · 2020 [cited by applicant]
US 20200156251A1 · Huang et al. · 2020 [cited by applicant]
US 20200159648A1 · Ghare et al. · 2020 [cited by applicant]
US 20200171668A1 · Lin · 2020 [cited by examiner]
US 20200171671A1 · Huang et al. · 2020 [cited by applicant]
US 20200206913A1 · Kaehler · 2020 [cited by applicant]
US 20200206919A1 · Nakayama · 2020 [cited by examiner]
US 20200225655A1 · Cella et al. · 2020 [cited by applicant]
US 20200265329A1 · Thomsen et al. · 2020 [cited by applicant]
US 20200276714A1 · Declerck et al. · 2020 [cited by applicant]
US 20200301510A1 · Birchfield et al. · 2020 [cited by applicant]
US 20200306977A1 · Islam · 2020 [cited by examiner]
US 20200310394A1 · Wouhaybi et al. · 2020 [cited by applicant]
US 20200342067A1 · Yu et al. · 2020 [cited by applicant]
US 20200398435A1 · Okura · 2020 [cited by examiner]
US 20210018903A1 · Muneta et al. · 2021 [cited by applicant]
US 20210046639A1 · Ravi et al. · 2021 [cited by applicant]
US 20210096824A1 · Stump et al. · 2021 [cited by applicant]
US 20210101282A1 · Yoneyama · 2021 [cited by examiner]
US 20210133670A1 · Cella et al. · 2021 [cited by applicant]
US 20210138651A1 · McGregor et al. · 2021 [cited by applicant]
US 20210141870A1 · McGregor et al. · 2021 [cited by applicant]
US 20210178575A1 · Riek et al. · 2021 [cited by applicant]
US 20210323167A1 · Hemes et al. · 2021 [cited by applicant]
US 20210369353A1 · Nikou et al. · 2021 [cited by applicant]
US 20210375439A1 · McKinnon et al. · 2021 [cited by applicant]
US 20220147026A1 · Poelman et al. · 2022 [cited by applicant]
US 20220219327A1 · Riek et al. · 2022 [cited by applicant]
US 20230019873A1 · Landon et al. · 2023 [cited by applicant]
CN 106426158A · 2017 [cited by applicant]
CN 110795874A · 2020 [cited by applicant]
EP 2498156A2 · 2012 [cited by applicant]
EP 2871544A2 · 2015 [cited by applicant]
EP 3476545A1 · 2019 [cited by applicant]
EP 3476549A1 · 2019 [cited by applicant]
EP 3705239A1 · 2020 [cited by examiner]
JP 2006003236A · 2006 [cited by applicant]
WO 2012027541A1 · 2012 [cited by applicant]
WO 2018176025A1 · 2018 [cited by applicant]
WO 2018196232A1 · 2018 [cited by applicant]
WO 2019064916A1 · 2019 [cited by applicant]
WO 2019113618A1 · 2019 [cited by applicant]
WO 2019113619A1 · 2019 [cited by applicant]
WO 2020053083A1 · 2020 [cited by applicant]
WO 2020190272A1 · 2020 [cited by applicant]
Alexei Vassiliev, Designing The Built-In Microcontroller Control Systems of Executive Robotic Devices Using the Digital Twins Technology, 2019, pp. 1-5. https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8843814 (… [cited by applicant]
Igor Azkarate Fernandez, Virtual commissioning of a robotic cell: an educational case study, 2019, pp. 1-6. https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8869373 (Year: 2019). [cited by applicant]
Final Office Action, U.S. Appl. No. 16/949,753, Feb. 17, 2022, 37 pages. [cited by applicant]
Fuchs et al., “Cooperative bin-picking with Time-of-Flight camera and impedance controlled DLR lightweight robot III”, Intelligent Robots and Systems (IROS), 2010 IEEE/RSJ International Conference ON, IEEE, Oct. 18, 201… [cited by applicant]
International Search Report and Written Opinion, PCT App. No. PCT/US21/72308, Mar. 7, 2022, 17 pages. [cited by applicant]
Lu et al., An on-line relative position and orientation error calibration methodology for workcell robot operations, Robotics and Computer Integrated Manufacturing, vol. 13, No. 2, Jun. 1, 1997, pp. 89-99. [cited by applicant]
Middelplaats et al., “Automatic Extrinsic Calibration and Workspace Mapping Algorithms to Shorten the Setup time of Camera-guided Industrial Robots”, Jun. 11, 2014, pp. 1-144. [cited by applicant]
Maragkos et al., “Virtual Reality Assisted Robot Programming for Human Collaboration”, 29th International Conference on Flexible Automation and Intelligent Manufacturing, Procedia Manufacturing, vol. 38, 2019, pp. 1697-… [cited by applicant]
Marc Priggemeyer, Interactive calibration and visual programming of reconfigurable robotic workcells, 2018, pp. 1936-1401. https://ieeexplore.ieee.org/document/8452707 (Year: 2018). [cited by applicant]
Niki Kousi, Digital twin for adaption of robots' behavior in flexible robotic assembly lines, 2019, pp. 121-127. https://www.researchgate.net/publication/330669566_Digital_twin_for_adaptation_of_robots%27_behavior_in_fl… [cited by applicant]
Stefan Profanter, A Hardware-Agnostic OPC UA Skill Model for Robot Manipulators and Tools, 2019, pp. 1061-1068. https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8869205&isnumber=8868236 (Year: 2019). [cited by applicant]
Wikipedia, “Digital Twin”, Available Online at <https://en.wikipedia.org/w/index.php?t%20itle=Digital%20twin&oldid=924505803#Industry%20-level%20dynam,cs>, Nov. 4, 2019, 16 pages. [cited by applicant]
Hatem Fakhurldeen, CARA system Architecture—A Click and Assemble Robotic Assembly System, 2019, pp. 5830-5835. https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8794114 (Year: 2019). [cited by applicant]
Madimir Kuliaev, Towards Product Centric Manufacturing: From Digital Twins to Product Assembly, May 2019, pp. 164-170. https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8972137 (Year: 2019). [cited by applicant]
Garrett Thomas, Learning Robotic Assembly from CAD, 2018, pp. 1-8. https://arxiv.org/pdf/1803.07635.pdf (Year: 2018). [cited by applicant]
Alexander Perzylo, Intuitive Instruction of Industrial Robots: Semantic Process Descriptions for Small Lot Production, 2016, pp. 1-6. https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7759358 (Year: 2016). [cited by applicant]
Solhaug Linnerud, CAD-based system for programming of robotic assembly processes with human-in-the-loop, Jul. 2019, pp. 2303-2308. https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8781385 (Year: 2019). [cited by applicant]
Ming-Yu et al., “Fast object localization and pose estimation in heavy clutter for robotic bin picking”, The International Journal of Robotics Research, [Online] vol. 31, No. 8, Jul. 2012 (Jul. 2012), pp. 951-973, XP055… [cited by applicant]
Shure Loren: “Carving a Dinosaur”, Dec. 16, 2009 (Dec. 16, 2009), pp. 1-9, XP055975843. [cited by applicant]