Vision-based sensor system and control method for robot arms

1. A sensor system ( 100 ) for monitoring a status of a kinematic chain ( 1 ) having elements comprising links ( 15 ) and joints ( 11 ), the sensor system comprising:

a computing unit ( 5 ),

at least one imaging sensor ( 4 , 41 , 42 ) operably connected to the computing unit ( 5 ),

wherein the at least one imaging sensor ( 4 , 41 , 42 ) is adapted to be mounted to the kinematic chain ( 1 ) or to be mounted in the surroundings of the kinematic chain ( 1 ), and is further adapted for monitoring at least a part of the kinematic chain ( 1 ) and/or the surroundings of the kinematic chain ( 1 ), and

wherein the computing unit ( 5 ) is adapted to determine a pose and/or movement parameters of the kinematic chain ( 1 ) by analyzing characteristic features ( 3 , 31 , 32 , 33 , 38 , 39 ) in an output signal of the at least one imaging sensor ( 4 , 41 , 42 ),

wherein the characteristic features are provided on at least one link ( 15 ), at at least one joint ( 11 ), and/or in the surroundings of the kinematic chain,

wherein the computing unit ( 5 ) comprises

a pose estimator adapted to determine 6D poses of each of the characteristic features and

a joint state estimator to determine joint positions of the kinematic chain based on the 6D poses derived by the pose estimator, and

wherein at least one of the characteristic features is a rotational visual marker ( 34 ) mounted on a marked joint ( 11 ), the rotational visual marker ( 34 ) comprising

a first part rigidly connected to a first link connected by the marked joint and

a second part rigidly connected to a second link connected by the marked joint, and

wherein the computing unit ( 5 ) is configured to determine a relative orientation of the first part and the second part directly from the rotational visual marker ( 34 ).

2. The sensor system of claim 1 , wherein the at least one imaging sensor ( 4 , 41 , 42 ) is a 2D video camera.

3. The sensor system of claim 1 , wherein the computing unit ( 5 ) comprises:

stored information about properties of the kinematic chain ( 1 ), and

information about characteristic visual features ( 3 , 31 , 32 , 33 , 38 , 39 ) of at least one element of the kinematic chain ( 1 ), and/or information about the surroundings of the kinematic chain ( 1 ).

4. The sensor system of claim 1 , further comprising a control unit ( 6 ) operably coupled to the computing unit ( 5 ), wherein the computing unit ( 5 ) provides the determined relative orientation of the marked joint ( 11 ) to the control unit ( 6 ), and the control unit is adapted to control the relative orientation of the marked joint ( 11 ) based on the determined relative orientation.

5. The sensor system of claim 1 , wherein the computing unit ( 5 ) is adapted to determine the rotational joint position of at least three joints ( 11 ) on the basis of an analysis of the characteristic features ( 3 , 31 , 32 , 33 , 38 , 39 ) and of stored information on a structure of the kinematic chain ( 1 ).

6. The sensor system of claim 1 , wherein the computing unit ( 5 ) comprises:

stored information about properties of the kinematic chain ( 1 ), including geometrical data and data about the degrees of freedom of movement of the kinematic chain ( 1 ), and

information about characteristic visual features ( 3 , 31 , 32 , 33 , 38 , 39 ) of at least one element of the kinematic chain ( 1 ), and/or information about the surroundings of the kinematic chain ( 1 ).

7. A robot ( 105 ) including a kinematic chain ( 1 ) and the sensor system ( 100 ) of claim 1 .

8. A sensor system ( 100 ) for monitoring a status of a kinematic chain ( 1 ) having elements comprising links ( 15 ) and joints ( 11 ), the sensor system comprising:

a computing unit ( 5 ),

at least one imaging sensor ( 4 , 41 , 42 ) operably connected to the computing unit ( 5 ),

wherein the at least one imaging sensor ( 4 , 41 , 42 ) is adapted to be mounted to the kinematic chain ( 1 ) or to be mounted in the surroundings of the kinematic chain ( 1 ), and is further adapted for monitoring at least a part of the kinematic chain ( 1 ) and/or the surroundings of the kinematic chain ( 1 ), and

wherein the computing unit ( 5 ) is adapted to determine a pose and/or movement parameters of the kinematic chain ( 1 ) by analyzing characteristic features ( 3 , 31 , 32 , 33 , 38 , 39 ) in an output signal of the at least one imaging sensor ( 4 , 41 , 42 ),

wherein the characteristic features are provided on at least one link ( 15 ), at at least one joint ( 11 ), and/or in the surroundings of the kinematic chain,

wherein the computing unit ( 5 ) comprises

a pose estimator adapted to determine 6D poses of each of the characteristic features and

a joint state estimator to determine joint positions of the kinematic chain based on the 6D poses derived by the pose estimator, and

wherein the pose estimator includes processing multiple threads, including

a first thread that locates characteristic features whose pose is unknown, and

a second thread that tracks characteristic features for which a pose estimate is available.

9. The sensor system of claim 8 ,

wherein at least one of the characteristic features is a rotational visual marker ( 34 ) mounted on a marked joint ( 11 ), the rotational visual marker ( 34 ) comprising

a first part rigidly connected to a first link connected by the marked joint and

a second part rigidly connected to a second link connected by the marked joint, and

wherein the computing unit ( 5 ) is configured to determine a relative orientation of the first part and the second part directly from the rotational visual marker ( 34 ).

10. The sensor system of claim 9 ,

wherein at least one of the characteristic features ( 39 ) is a non-planar element that surrounds one of the links ( 15 ).

11. The sensor system of claim 9 ,

wherein the at least one imaging sensor ( 4 , 41 , 42 ) is adapted to be mounted on an element of the kinematic chain ( 1 ) and is adapted to monitor the surroundings of the kinematic chain ( 1 ), and

wherein the computing unit ( 5 ) is configured to determine the pose of the imaging sensor ( 4 , 41 , 42 ) from analyzing the output signal of the imaging sensor ( 4 , 41 , 42 ) and to determine a pose of at least one element in the kinematic chain ( 1 ) on the basis of the pose of the imaging sensor ( 4 , 41 , 42 ).

12. A method for determining a pose and/or movement parameters of at least one element of a kinematic chain ( 1 ) having elements comprising links ( 15 ) and joints ( 11 ), the method comprising:

visually monitoring at least one element of the kinematic chain ( 1 ) and/or the surroundings of the kinematic chain ( 1 ) with at least one imaging sensor ( 4 , 41 , 42 ),

identifying characteristic features ( 3 , 31 , 32 , 33 , 38 , 39 ) of the at least one element or of the surroundings in an output signal of the imaging sensor ( 4 , 41 , 42 ),

estimating 6D poses of the identified characteristic features ( 3 , 31 , 32 , 33 , 38 , 39 ), and

calculating the pose and/or the movement parameters of the at least one element on the basis of the estimated 6D poses, and determining a rotational joint position of at least one joint ( 11 ) by analyzing the characteristic features,

wherein from an output signal of an imaging sensor ( 4 , 41 , 42 ) mounted to an element of the kinematic chain ( 1 ), the rotational joint position of at least three joints ( 11 ) is determined by a computing unit ( 5 ) by analyzing characteristic features ( 3 , 31 , 32 , 33 , 38 , 39 ) in the surroundings of the kinematic chain ( 1 ) while taking into account information on the kinematic chain ( 1 ) stored in the computing unit ( 5 ).

13. The method of claim 12 , wherein the imaging sensor ( 4 , 41 , 42 ) is mounted to a moving part of the kinematic chain ( 1 ), and wherein the computing unit ( 5 ) determines the pose of the imaging sensor ( 4 , 41 , 42 ) by analyzing features of the surroundings, and wherein the configuration of joints ( 11 ) between a base link ( 16 ) and the imaging sensor ( 4 , 41 , 42 ) is indirectly determined from the pose of the imaging sensor ( 4 , 41 , 42 ) and on the basis of information on the kinematic chain ( 1 ) stored in the computing unit ( 5 ).

14. A method for determining a pose and/or movement parameters of at least one element of a kinematic chain ( 1 ) having elements comprising links ( 15 ) and joints ( 11 ), the method comprising:

visually monitoring at least one element of the kinematic chain ( 1 ) and/or the surroundings of the kinematic chain ( 1 ) with at least one imaging sensor ( 4 , 41 , 42 ),

identifying characteristic features ( 3 , 31 , 32 , 33 , 38 , 39 ) of the at least one element or of the surroundings in an output signal of the imaging sensor ( 4 , 41 , 42 ),

estimating 6D poses of the identified characteristic features ( 3 , 31 , 32 , 33 , 38 , 39 ), and

calculating the pose and/or the movement parameters of the at least one element on the basis of the estimated 6D poses, and determining a rotational joint position of at least one joint ( 11 ) by analyzing the characteristic features,

wherein estimating 6D poses of the identified characteristic features ( 3 , 31 , 32 , 33 , 38 , 39 ) includes

locating characteristic features whose pose is unknown in a first thread, and

tracking characteristic features for which a pose estimate is available in a second thread.

15. The method of claim 14 , wherein the characteristic features ( 3 , 31 , 32 , 33 , 38 , 39 ) comprise

a 3D shape surrounding a link of the kinematic chain ( 1 ).

16. The method of claim 14 , wherein from an output signal of an imaging sensor ( 4 , 41 , 42 ) mounted in the surroundings of the kinematic chain ( 1 ), the rotational joint position of at least one joint ( 11 ) is directly determined by a computing unit ( 5 ) by analyzing a rotational visual marker ( 34 ) mounted on the at least one joint ( 11 ).

17. The method of claim 14 , wherein an estimation of the position of at least one joint ( 11 ) is carried out by monitoring optical markers and/or textures provided at a movable part of an actuator ( 12 ) and/or at a movable part of a gear operably connected to an actuator ( 12 ), and by determining the movement of a link ( 15 ) and/or joint ( 11 ) actuated by the actuator ( 12 ) from analyzing the movement of the optical markers or textures.

18. The method of claim 14 , wherein from an output signal of an imaging sensor ( 4 , 41 , 42 ) mounted to an element of the kinematic chain ( 1 ), the rotational joint position of at least one joint ( 11 ) is determined by a computing unit ( 5 ) by analyzing characteristic features ( 3 , 31 , 32 , 33 , 38 , 39 ) in the surroundings of the kinematic chain ( 1 ) while taking into account information on the kinematic chain ( 1 ) stored in the computing unit ( 5 ).

19. The method of claim 14 , wherein from an output signal of an imaging sensor ( 4 , 41 , 42 ) mounted in the surroundings of the kinematic chain ( 1 ), the rotational joint position of at least three joints ( 11 ), is determined by a computing unit ( 5 ) by analyzing characteristic features ( 3 , 31 , 32 , 33 , 38 , 39 ) of the at least one joint ( 11 ) and/or of at least one link ( 15 ) connected to it.

20. The method of claim 14 , wherein from an output signal of an imaging sensor ( 4 , 41 , 42 ) mounted to an element of the kinematic chain ( 1 ), the rotational joint position of at least three joints ( 11 ) is determined by a computing unit ( 5 ) by analyzing characteristic features ( 3 , 31 , 32 , 33 , 38 , 39 ) in the surroundings of the kinematic chain ( 1 ) while taking into account information on the kinematic chain ( 1 ) stored in the computing unit ( 5 ).