IP Library Granted Patent US 12,440,987
Granted Patent B2
US 12,440,987 · App. 18/761,968 · Granted Oct 14, 2025

Processing systems and methods for providing processing of a variety of objects

Inventors: Thomas Wagner (Concord, MA); Kevin Ahearn (Nebo, NC); Benjamin Cohen (White Plains, NY); Michael Dawson-Haggerty (Pittsburgh, PA); Christopher Geyer (Arlington, MA); Thomas Koletschka (Somerville, MA); Kyle Maroney (Saunderstown, RI); Matthew T. Mason (Atlanta, GA); Gene Temple Price (Somerville, MA); Joseph Romano (Arlington, MA); Daniel Carlton Smith (Wexford, PA); Siddhartha Srinivasa (Seattle, WA); Prasanna Velagapudi (Pittsburgh, PA); Thomas Allen (Reading, MA)
Assignee: Berkshire Grey Operating Company, Inc.
B25J9/1669B07C3/18B07C5/36B25J9/0093B25J9/1612B25J9/1664B25J9/1687B25J9/1697B25J19/04G05B19/4183G05B2219/32037G05B2219/39106G05B2219/39295G05B2219/39476G05B2219/39484G05B2219/39504G05B2219/39548G05B2219/40053G05B2219/40078G05B2219/40116G05B2219/40538G05B2219/45045Y02P90/02
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,440,987
App. No.
18/761,968
Granted
Oct 14, 2025
Kind
B2
Abstract

A sortation system is disclosed that includes a programmable motion device including an end effector, a perception system for recognizing any of the identity, location, and orientation of an object presented in a plurality of objects, a grasp selection system for selecting a grasp location on the object, the grasp location being chosen to provide a secure grasp of the object by the end effector to permit the object to be moved from the plurality of objects to one of a plurality of destination locations, and a motion planning system for providing a motion path for the transport of the object when grasped by the end effector from the plurality of objects to the one of the plurality of destination locations, wherein the motion path is chosen to provide a path from the plurality of objects to the one of the plurality of destination locations.

Claims (38)

1. An object processing system comprising:

a programmable motion device including an end-effector;

a perception unit for capturing real-time image data of the plurality of objects at an input area;

an interactive display system including a touch screen input display for displaying the real-time image data and through which machine learning grasp input data regarding a plurality of objects is received; and

a control system accessing the machine learning grasp input data and for providing object grasp information regarding a grasp location for grasping the object responsive to the machine learning grasp input data regarding a plurality of objects.

2. The object processing system of claim 1 , wherein the machine learning grasp input data is received off-line not during object processing.

3. The object processing system of claim 1 , wherein the object grasp information is further responsive to information regarding at least one property of a suction cup of the end -effector.

4. The object processing system of claim 1 , wherein the object grasp information is further responsive to appearance-based features of the object.

5. The object processing system of claim 1 , wherein the machine learning grasp input data is received in real-time during object processing.

6. The object processing system of claim 5 , wherein the machine learning grasp input data is recorded with each grasp attempt.

7. The object processing system of claim 1 , wherein the object grasp information includes a grasp direction at the grasp location for grasping the object.

8. The object processing system of claim 7 , wherein the grasp direction is a direction that is normal to a surface of the object at the grasp location.

9. The object processing system of claim 8 , wherein the surface of the object at the grasp location is curved.

10. An object processing system comprising:

a programmable motion device including an end-effector;

a perception unit for capturing real-time image data of the plurality of objects at an input area;

an interactive display that includes a touch screen input display for displaying the real -time image data; and

a control system for providing object grasp information regarding a plurality of grasp locations for grasping the object with the end-effector, the plurality of object grasp locations being derived from a plurality of machine learning grasp input data regarding a plurality of objects, the machine learning grasp input data including data received via the interactive display system that includes the touch screen input display.

11. The object processing system of claim 10 , wherein the machine learning grasp input data is received off-line not during object processing.

12. The object processing system of claim 10 , wherein the object grasp information is further responsive to information regarding at least one property of a suction cup of the end -effector.

13. The object processing system of claim 10 , wherein the object grasp information is further responsive to appearance-based features of the object.

14. The object processing system of claim 10 , wherein the machine learning grasp input data is received in real-time during object processing.

15. The object processing system of claim 14 , wherein the machine learning grasp input data is recorded with each grasp attempt.

16. The object processing system of claim 10 , wherein the object grasp information includes a grasp direction at a selected grasp location of the plurality of grasp locations for grasping the object.

17. The object processing system of claim 16 , wherein the grasp direction is a direction that is normal to a surface of the object at the selected grasp location.

18. The object processing system of claim 17 , wherein the surface of the object at the selected grasp location is curved.

19. A method of processing objects received at an input area, said method comprising:

providing a programmable motion device with an end-effector. obtaining first grasp input information for a selected object of a plurality of objects in a container at an input area responsive to machine learning grasp input data;

using the end effector to move the selected object of the plurality of objects in the container at the input area without grasping the object; and

obtaining second grasp input information for the selected object of the plurality of objects in the container at the input area responsive to machine learning grasp input data.

20. The method of claim 19 , wherein the machine learning grasp input data is received in real-time during object processing.

21. The method of claim 19 , wherein the machine learning grasp input data is recorded with each grasp attempt.

22. The method of claim 19 , wherein the machine learning grasp input data is received off-line not during object processing.

23. The method of claim 19 , wherein the second grasp input information is further responsive to information regarding at least one property of a suction cup of the end-effector.

24. The method of claim 19 , wherein the second grasp input information is further responsive to appearance-based features of the object.

25. The method of claim 19 , wherein the second grasp input information includes a grasp direction at a selected grasp location of the plurality of grasp locations for grasping the object.

26. The method of claim 25 , wherein the grasp direction is a direction that is normal to a surface of the object at the selected grasp location.

27. The method of claim 26 , wherein the surface of the object at the selected grasp location is curved.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 25, 2024
From: WAGNER, THOMAS; AHEARN, KEVIN; COHEN, BENJAMIN; DAWSON-HAGGERTY, MICHAEL; GEYER, CHRISTOPHER; KOLETSCHKA, THOMAS; MARONEY, KYLE; MASON, MATTHEW T.; PRICE, GENE TEMPLE; ROMANO, JOSEPH; SMITH, DANIEL CARLTON; SRINIVASA, SIDDHARTHA; VELAGAPUDI, PRASANNA; ALLEN, THOMAS
To: BERKSHIRE GREY OPERATING COMPANY, INC.
Reel/Frame 068080/0621 →
Continuity (6)
Continuation 17843666 · Jun 17, 2022
Continuation 16775798 · Jan 29, 2020
Continuation 15348498 · Nov 10, 2016
Provisional Application 62277234 · Jan 11, 2016
Provisional Application 62255069 · Nov 13, 2015
Related Publication 20240359328A1 · Oct 31, 2024
References Cited (186)
US 2853333A · Littell · 1958 [cited by applicant]
US 3005652A · Helm · 1961 [cited by applicant]
US 3195941A · Morey · 1965 [cited by applicant]
US 3637249A · Kuhl et al. · 1972 [cited by applicant]
US 4389064A · Laverriere · 1983 [cited by applicant]
US 4557659A · Scaglia · 1985 [cited by applicant]
US 4786847A · Daggett et al. · 1988 [cited by applicant]
US 4828304A · No et al. · 1989 [cited by applicant]
US 4896357A · Hatano et al. · 1990 [cited by applicant]
US 5207465A · Rich · 1993 [cited by applicant]
US 5244338A · White · 1993 [cited by applicant]
US 5752729A · Crozier et al. · 1998 [cited by applicant]
US 5794789A · Payson et al. · 1998 [cited by applicant]
US 5865487A · Gore et al. · 1999 [cited by applicant]
US 5937143A · Watanabe et al. · 1999 [cited by applicant]
US 6015174A · Raes et al. · 2000 [cited by applicant]
US 6060677A · Ulrichsen et al. · 2000 [cited by applicant]
US 6088628A · Watanabe et al. · 2000 [cited by applicant]
US 6244640B1 · Le Bricquer et al. · 2001 [cited by applicant]
US 6721444B1 · Gu · 2004 [cited by examiner]
US 6817639B2 · Schmalz et al. · 2004 [cited by applicant]
US 7313464B1 · Perreault · 2007 [cited by examiner]
US 7474939B2 · Oda · 2009 [cited by examiner]
US 7677622B2 · Dunkmann et al. · 2010 [cited by applicant]
US 8070203B2 · Schaumberger · 2011 [cited by applicant]
US 8132835B2 · Ban et al. · 2012 [cited by applicant]
US 8267386B2 · Schaaf et al. · 2012 [cited by applicant]
US 8874270B2 · Ando · 2014 [cited by applicant]
US 8936291B2 · Yasuda et al. · 2015 [cited by applicant]
US 9102053B2 · Suzuki · 2015 [cited by applicant]
US 9102055B1 · Konolige et al. · 2015 [cited by applicant]
US 9227323B1 · Konolige · 2016 [cited by examiner]
US 9259844B2 · Xu · 2016 [cited by examiner]
US 9266237B2 · Nomura · 2016 [cited by applicant]
US 9283680B2 · Yasuda · 2016 [cited by examiner]
US 9486926B2 · Kawano · 2016 [cited by applicant]
US 9492923B2 · Wellman et al. · 2016 [cited by applicant]
US 9604363B2 · Ban · 2017 [cited by examiner]
US 9623570B1 · Krahn et al. · 2017 [cited by applicant]
US 9656813B2 · Dunkmann et al. · 2017 [cited by applicant]
US 9687982B1 · Jules · 2017 [cited by examiner]
US 9687983B1 · Prats · 2017 [cited by examiner]
US 10456915B1 · Diankov · 2019 [cited by applicant]
US 10625432B2 · Wagner · 2020 [cited by examiner]
US 11420329B2 · Wagner · 2022 [cited by examiner]
US 12059810B2 · Wagner · 2024 [cited by examiner]
US 20010056313A1 · Osborne, Jr. · 2001 [cited by applicant]
US 20020147568A1 · Wenzel et al. · 2002 [cited by applicant]
US 20030038491A1 · Schmalz et al. · 2003 [cited by applicant]
US 20040232716A1 · Reed et al. · 2004 [cited by applicant]
US 20050251290A1 · Skourup et al. · 2005 [cited by applicant]
US 20080181485A1 · Beis · 2008 [cited by examiner]
US 20100109360A1 · Meisho · 2010 [cited by applicant]
US 20100125361A1 · Mougin et al. · 2010 [cited by applicant]
US 20100175487A1 · Sato · 2010 [cited by applicant]
US 20100234857A1 · Itkowitz et al. · 2010 [cited by applicant]
US 20110320036A1 · Freudelsperger · 2011 [cited by applicant]
US 20130054029A1 · Huang · 2013 [cited by examiner]
US 20130110280A1 · Folk · 2013 [cited by applicant]
US 20130129464A1 · Regan et al. · 2013 [cited by applicant]
US 20130166061A1 · Yamamoto · 2013 [cited by applicant]
US 20130245824A1 · Barajas · 2013 [cited by applicant]
US 20130343640A1 · Buehler · 2013 [cited by examiner]
US 20130345872A1 · Brooks et al. · 2013 [cited by applicant]
US 20140005831A1 · Naderer et al. · 2014 [cited by applicant]
US 20140067121A1 · Brooks et al. · 2014 [cited by applicant]
US 20140067127A1 · Gatou · 2014 [cited by applicant]
US 20140195979A1 · Branton · 2014 [cited by examiner]
US 20140298231A1 · Saito · 2014 [cited by examiner]
US 20140305847A1 · Kudrus · 2014 [cited by examiner]
US 20150057793A1 · Kawano · 2015 [cited by applicant]
US 20150073589A1 · Khodl et al. · 2015 [cited by applicant]
US 20150081090A1 · Dong · 2015 [cited by applicant]
US 20150190925A1 · Hoffman · 2015 [cited by examiner]
US 20150224650A1 · Xu et al. · 2015 [cited by applicant]
US 20150306634A1 · Maeda et al. · 2015 [cited by applicant]
US 20150306770A1 · Mittal et al. · 2015 [cited by applicant]
US 20150328779A1 · Bowman et al. · 2015 [cited by applicant]
US 20150346708A1 · Mattern et al. · 2015 [cited by applicant]
US 20150352717A1 · Mundt et al. · 2015 [cited by applicant]
US 20150352721A1 · Wicks · 2015 [cited by examiner]
US 20150375398A1 · Penn et al. · 2015 [cited by applicant]
US 20150375401A1 · Dunkmann et al. · 2015 [cited by applicant]
US 20160075521A1 · Puchwein et al. · 2016 [cited by applicant]
US 20160101526A1 · Saito et al. · 2016 [cited by applicant]
US 20160136816A1 · Pistorino · 2016 [cited by applicant]
US 20160167227A1 · Wellman et al. · 2016 [cited by applicant]
US 20160221187A1 · Bradski et al. · 2016 [cited by applicant]
US 20160243704A1 · Vakanski et al. · 2016 [cited by applicant]
US 20160271805A1 · Kuolt et al. · 2016 [cited by applicant]
US 20160347545A1 · Lindbo et al. · 2016 [cited by applicant]
US 20160379076A1 · Nobuoka et al. · 2016 [cited by applicant]
US 20170024896A1 · Houghton et al. · 2017 [cited by applicant]
US 20170050315A1 · Sileane · 2017 [cited by applicant]
US 20170066597A1 · Hiroi · 2017 [cited by applicant]
US 20170080566A1 · Stubbs et al. · 2017 [cited by applicant]
US 20170106532A1 · Wellman · 2017 [cited by examiner]
US 20170136632A1 · Wagner et al. · 2017 [cited by applicant]
US 20170320210A1 · Ding et al. · 2017 [cited by applicant]
US 20180085922A1 · Ooba · 2018 [cited by examiner]
US 20180370038A1 · Isaac et al. · 2018 [cited by applicant]
US 20190358810A1 · Oddhner et al. · 2019 [cited by applicant]
US 20190361672A1 · Odhner et al. · 2019 [cited by applicant]
US 20200164505A1 · Boer et al. · 2020 [cited by applicant]
US 20200164531A1 · Wagner et al. · 2020 [cited by applicant]
US 20200189105A1 · Wen · 2020 [cited by examiner]
US 20200316782A1 · Chavez et al. · 2020 [cited by applicant]
US 20200319627A1 · Edwards et al. · 2020 [cited by applicant]
US 20200331144A1 · Huang · 2020 [cited by examiner]
US 20210081791A1 · Goodrich et al. · 2021 [cited by applicant]
US 20210122586A1 · Sun et al. · 2021 [cited by applicant]
US 20210260758A1 · Singh et al. · 2021 [cited by applicant]
US 20220203547A1 · Majumdar et al. · 2022 [cited by applicant]
US 20220281114A1 · Moreira Rodrigues et al. · 2022 [cited by applicant]
US 20220289501A1 · Sun et al. · 2022 [cited by applicant]
US 20220289502A1 · Sun et al. · 2022 [cited by applicant]
US 20220314447A1 · Wagner et al. · 2022 [cited by applicant]
US 20230260071A1 · Chavez et al. · 2023 [cited by applicant]
US 20230331416A1 · Matl et al. · 2023 [cited by applicant]
CA 2928645A1 · 2015 [cited by applicant]
CA 3004711C · 2022 [cited by applicant]
CA 3107257C · 2023 [cited by applicant]
CN 102448679A · 2012 [cited by applicant]
CN 102922521A · 2013 [cited by applicant]
CN 103157607A · 2013 [cited by applicant]
CN 103464383A · 2013 [cited by applicant]
CN 103645724A · 2014 [cited by applicant]
CN 103963058A · 2014 [cited by applicant]
CN 203955558U · 2014 [cited by applicant]
CN 108349083A · 2018 [cited by applicant]
CN 113731862A · 2021 [cited by applicant]
DE 3810989A1 · 1989 [cited by applicant]
DE 10121344A1 · 2002 [cited by applicant]
DE 10319253A1 · 2004 [cited by applicant]
DE 102007028680A1 · 2008 [cited by applicant]
DE 102007038834A1 · 2009 [cited by applicant]
DE 102010002317A1 · 2011 [cited by applicant]
DE 102012102333A1 · 2013 [cited by applicant]
EP 0317020A2 · 1989 [cited by applicant]
EP 1256421A1 · 2002 [cited by applicant]
EP 2233400A1 · 2010 [cited by applicant]
EP 2960024A2 · 2015 [cited by applicant]
EP 3374138B1 · 2022 [cited by applicant]
EP 4088889A1 · 2022 [cited by applicant]
JP 769470A · 1995 [cited by applicant]
JP 2010201536A · 2010 [cited by applicant]
JP 2015044274A · 2015 [cited by applicant]
JP 2007182286A · 2019 [cited by applicant]
WO 2010034044A2 · 2010 [cited by applicant]
WO 2010099873A1 · 2010 [cited by applicant]
WO 2014166650A1 · 2014 [cited by applicant]
WO 2015035300A1 · 2015 [cited by applicant]
WO 2015118171A1 · 2015 [cited by applicant]
WO 2015162390A1 · 2015 [cited by applicant]
WO 2017083574A1 · 2017 [cited by applicant]
Communication pursuant to Article 94(3) EPC issued by the European Patent Office in related European Patent Application No. 22172900.7 on Sep. 30, 2024, 7 pages. [cited by applicant]
Liu et al., Gesture-based NUI Application for Real-time Path Modification, 2013 IEEE 11th International Conference on Dependable, Autonomic and Secure Computing, pp. 446-449, retrieved on Jun. 25, 2014. [cited by applicant]
Lambrecht et al., Markerless Gesture-Based Motion Control and Programming of Industrial Robots, Emerging Technologies & Factory Automation (ETFA), 2011 IEEE 16th Conference On, IEEE, Sep. 5, 2011, 4 pages. [cited by applicant]
Bohg et al., “Data-Driven Grasp Synthesis-A Su,” IEEE Transactions On Robotics, IEEE Service Center, Piscataway, NJ, US, vol. 30, No. 2, Apr. 1, 2014, pp. 289-309. [cited by applicant]
Cipolla et al., “Visually guided grasping in unstructured environments,” Robotics and Autonomous Systems, Elsevier Science Publishers, Amsterdam, NL, vol. 19, No. 3-4, Mar. 1, 1997, pp. 337-346. [cited by applicant]
Communication pursuant to Article 94(3) EPC issued by the European Patent Office in related European Patent Application No. 16805614.1 on Oct. 14, 2020, 6 pages. [cited by applicant]
Communication pursuant to Rules 161(1) and 162 EPC issued by the European Patent Office in related European Patent Application No. 16805614.1 on Jun. 20, 2018, 3 pages. [cited by applicant]
EPO Form 1507N (Communication) and the European Search Report issued by the European Patent Office in related European Patent Application No. 22172900.7 on Oct. 19, 2022, 14 pages. [cited by applicant]
Examiner's Report issued by the Innovation, Science and Economic Development Canada (Canadian Intellectual Property Office) in related Canadian Patent Application No. 3,107,257 on Mar. 18, 2022, 6 pages. [cited by applicant]
Examiner's Report issued by the Innovation, Science and Economic Development Canada (Canadian Intellectual Property Office) in related Canadian Patent Application No. 3,192,023 on Jun. 13, 2024, 5 pages. [cited by applicant]
Final Office Action issued by the U.S. Patent and Trademark Office on Mar. 9, 2018 in related U.S. Appl. No. 15/348,498, 25 pages. [cited by applicant]
Final Office Action issued by the U.S. Patent and Trademark Office on Jun. 9, 2019 in related U.S. Appl. No. 15/348,498, 26 pages. [cited by applicant]
First Office Action, and its English translation, issued by the China National Intellectual Property Administration in related Chinese Patent Application No. 201680066557.2 on Jul. 28, 2020, 20 pages. [cited by applicant]
Hebert, et al., “A Robotic Gripper System for Limp Material Manipulation: Hardware and Software Development and Integration,” Proceedings of the 1997 IEEE International Conference on Robotics and Automation, Albuquerque… [cited by applicant]
International Preliminary Report on Patentability issued by the International Bureau of WIPO in related International Application No. PCT/US2016/061414 on May 15, 2018, 10 pages. [cited by applicant]
International Search Report and Written Opinion issued by the International Searching Authority, the European Patent Office, in related International Application No. PCT/US2016/061414 on Mar. 22, 2017, 14 pages. [cited by applicant]
Klingbeil et al., Grasping with application to an autonomous checkout robot, Robotics and Automation (ICRA), 2011 IEEE Inernational Conference On, IEEE, May 9, 2011, pp. 2837-2844. [cited by applicant]
Le et al., Learning to grasp objects with multiple contact points, 2010 IEEE International Conference on Robotics and Automation: ICRA 2010; Anchorage, Alaska, USA, May 3-8, 2010, IEEE, Piscataway, NJ, USA, May 3, 2010 … [cited by applicant]
Liu, et al., “Hand-Arm Coordination for a Tomato Harvesting Robot Based on Commercial Manipulator,” Proceeding of the IEEE, International Conference on Robotics and Biomimetics (ROBIO), Shenzhen, China, Dec. 2013, pp. 2… [cited by applicant]
Non-Final Office Action issued by the U.S. Patent and Trademark Office on Aug. 17, 2017 in related U.S. Appl. No. 15/348,498, 20 pages. [cited by applicant]
Non-Final Office Action issued by the U.S. Patent and Trademark Office on Sep. 25, 2018 in related U.S. Appl. No. 15/348,498, 28 pages. [cited by applicant]
Non-Final Office Action issued by the United States Patent and Trademark Office in related U.S. Appl. No. 16/775,798 on Nov. 26, 2021, 30 pages. [cited by applicant]
Non-Final Office Action issued by the United States Patent and Trademark Office in related U.S. Appl. No. 17/843,666 on Nov. 27, 2023, 14 pages. [cited by applicant]
Notice of Second Office Action and Second Office Action, along with its English translation, issued by the China National Intellectual Property Administration in related Chinese Patent Application No. 201680066557.2 on … [cited by applicant]
Notice on the First Office Action, along with its English translation, issued by the China National Intellectual Property Administration in related Chinese Patent Application No. 202111071608.X on Dec. 29, 2022, 11 page… [cited by applicant]
Notice on the Second Office Action, along with its English translation, issued by the China National Intellectual Property Administration in related Chinese Patent Application No. 202111071608.X on Jul. 6, 2023, 8 pages. [cited by applicant]
Notice on the Third Office Action issued by the China National Intellectual Property Administration in related Chinese Patent Application No. 202111071608.X on Nov. 24, 2023, 9 pages. [cited by applicant]
Office Action issued by the Innovation, Science and Economic Development Canada in related Canadian Patent Application No. 3,004,711 on Jun. 17, 2019, 8 pages. [cited by applicant]
Office Action issued by the Innovation, Science and Economic Development Canada in related Canadian Patent Application No. 3,004,711 on Mar. 10, 2020, 8 pages. [cited by applicant]
Office Action issued by the Innovation, Science and Economic Development Canada in related Canadian Patent Application No. 3,004,711 on Oct. 20, 2020, 5 pages. [cited by applicant]
Rembold et al., “Object turning for barcode search,” PROCEEDINGS, 2000 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2000), Oct. 31-Nov. 5, 2000, vol. 2, Oct. 31, 2000 (Oct. 31, 2000), p. 126… [cited by applicant]