Granted Patent B2

US 12,573,086 · App. 18/205,065 · Granted Mar 10, 2026

Inventors: Hiroto Mizohana (Kanagawa, JP); Ryoji Tsukamoto (Kanagawa, JP)

Assignee: Canon Kabushiki Kaisha

G06T7/74

Abstract

Claims (75)

1 . A control method for controlling an apparatus including a movable portion by a visual servo, the control method comprising:

obtaining a goal image corresponding to a goal positional relationship between the movable portion and a target object;

extracting goal feature points from the goal image by image processing;

obtaining a current image corresponding to a current positional relationship between the movable portion and the target object;

extracting candidate feature points from the current image by image processing;

extracting current feature points associated with the goal feature points from the candidate feature points by performing matching processing between the candidate feature points and the goal feature points;

correcting the current feature points by using the current feature points obtained by the matching processing and a priority order set for the goal feature points; and

generating, by using matching information based on matching between the corrected current feature points and the goal feature points, a control signal for moving the movable portion.

2 . The control method according to claim 1 , further comprising:

classifying the goal feature points into a plurality of goal feature point clusters by first clustering processing;

obtaining goal feature point cluster representative information for each of the plurality of goal feature point clusters;

classifying the current feature points into a plurality of current feature point clusters by second clustering processing; and

obtaining current feature point cluster representative information for each of the plurality of current feature point clusters,

wherein the matching information includes the current feature point cluster representative information and the goal feature point cluster representative information.

3 . The control method according to claim 2 , wherein a priority order is set for the goal feature point clusters into which the goal feature points have been classified, and

wherein, in the matching processing, a current feature point cluster associated with a goal feature point cluster placed higher in the priority order is obtained with a higher priority.

4 . The control method according to claim 2 , wherein the goal feature points include a first goal feature point, the current feature points include a first current feature point associated with the first goal feature point in the matching processing, the goal feature point clusters include a first goal feature point cluster including the first goal feature point, and the current feature point clusters include a first current feature point cluster corresponding to the first goal feature point cluster, and

wherein, in the second clustering processing, the first current feature point is classified into the first current feature point cluster.

5 . The control method according to claim 2 , wherein the first clustering processing and/or the second clustering processing corresponds to at least one of first processing, second processing, or third processing below,

wherein, in the first processing:

the goal feature points are classified into the plurality of goal feature point clusters in the first clustering processing on a basis of a distance in the goal image, and/or

the current feature points are classified into the plurality of current feature point clusters in the second clustering processing on a basis of a distance in the current image,

wherein, in the second processing:

the goal feature points are classified into the plurality of goal feature point clusters in the first clustering processing on a basis of a gradation level in the goal image, and/or

the current feature points are classified into the plurality of current feature point clusters in the second clustering processing on a basis of a gradation level in the current image, and

wherein, in the third processing:

the goal feature points are classified into the plurality of goal feature point clusters in the first clustering processing on a basis of a shape model of the target object and/or a shape model of the movable portion, and/or

the current feature points are classified into the plurality of current feature point clusters in the second clustering processing on a basis of the shape model of the target object and/or the shape model of the movable portion.

6 . The control method according to claim 2 , wherein a position of a center of gravity of the goal feature points included in each of the plurality of goal feature point clusters is obtained as the goal feature point cluster representative information, and

wherein a position of a center of gravity of the current feature points included in each of the plurality of current feature point clusters is obtained as the current feature point cluster representative information.

7 . The control method according to claim 2 , wherein a position of a center of gravity of the goal feature points included in each of the plurality of goal feature point clusters, and a magnification ratio and/or a rotation angle are obtained as the goal feature point cluster representative information, and

wherein a position of a center of gravity of the current feature points included in each of the plurality of current feature point clusters, and the magnification ratio and/or the rotation angle are obtained as the current feature point cluster representative information.

8 . The control method according to claim 1 , wherein the priority order is set by using at least one of:

detection frequency of the goal feature points;

contribution of the goal feature points to visual servo control performance;

detection precision of the goal feature points; or

a group of images captured from a plurality of positions.

9 . The control method according to claim 1 , wherein the corrected current feature points are generated by using at least one of:

one or more of the current feature points placed low in the priority order;

geometric transformation from one or more of the current feature points; and

a physical model and a motion history.

10 . The control method according to claim 1 , wherein the image processing is processing utilizing an image feature value that is not changed by any of rotation, enlargement, and size reduction.

11 . The control method according to claim 1 , wherein the goal image and the current image are captured by an image pickup apparatus attached to the movable portion.

12 . The control method according to claim 1 , wherein the goal image and the current image are captured by an image pickup apparatus disposed at a position where the image pickup apparatus is capable of imaging the movable portion and the target object.

13 . The control method according to claim 1 , wherein the apparatus including the movable portion is a robot.

14 . The control method according to claim 1 , wherein an information of the extracting goal feature points has a degree of freedom of y,

wherein in the generating, the control signal for moving the movable portion is generated at a degree of freedom of n, and

wherein the number of N of the goal feature points extracted from the goal image satisfies N≥1+ (n+1)/ψ.

15 . The control method according to claim 14 , wherein the number N satisfies N≥1+ (n+1)/ψ.

16 . The control method according to claim 14 , wherein the number N satisfies N≥L+1, where L represents a minimum integer M satisfying M>(n+1)/ψ.

17 . The control method according to claim 14 , wherein the number N satisfies N≥2×L, where L represents a minimum integer M satisfying M>(n+1)/ψ.

18 . The control method according to claim 14 , wherein the number N of the goal feature points extracted from the goal image satisfies N≥3×L, where L represents a minimum integer M satisfying M> (n+1)/ψ.

19 . The control method according to claim 14 , wherein the number N satisfies N≤L+40.

20 . The control method according to claim 14 , wherein the number N satisfies N≤10×L.

21 . The control method according to claim 14 , wherein n is 6 or 7, and ψ is 2 or 3.

22 . A non-transitory computer-readable recording medium storing a control program for causing a control portion to execute the control method according to claim 1 .

23 . A method for manufacturing a product, the method comprising manufacturing the product by controlling the apparatus by the control method according to claim 1 , wherein the apparatus is an apparatus used for manufacturing the product.

24 . A system comprising:

an apparatus including a movable portion; and

a control portion,

wherein the control portion executes the control method according to claim 6 .

25 . The system according to claim 24 , wherein the control portion is configured to:

classify the goal feature points into a plurality of goal feature point clusters by first clustering processing;

obtain goal feature point cluster representative information for each of the plurality of goal feature point clusters;

classify the current feature points into a plurality of current feature point clusters by second clustering processing; and

obtain current feature point cluster representative information for each of the plurality of current feature point clusters, and

wherein the matching information includes the current feature point cluster representative information and the goal feature point cluster representative information.

26 . The system according to claim 25 , wherein a priority order is set for the goal feature point clusters into which the goal feature points have been classified, and

wherein in the matching processing, a current feature point cluster associated with a goal feature point cluster that is higher in the priority order is obtained with a higher priority.

27 . The system according to claim 24 , wherein a priority order is set for the extracted goal feature points, and

wherein in the matching processing, a current feature point associated with a goal feature point placed higher in the priority order is extracted with a higher priority.

28 . The system according to claim 24 , wherein the apparatus including the movable portion is a robot.

29 . The system according to claim 24 , wherein the control portion is configured to display, on a display screen, part or all of an image captured by an image pickup apparatus, the goal image, the candidate feature points, the goal feature points, the current image, the current feature points, and the matching information.

30 . The system according to claim 29 , wherein the control portion is configured to display, on the display screen, an instruction input portion related to an operation of the image pickup apparatus, and/or an instruction input portion related to an operation of registering information in a storage portion.

31 . The system according to claim 24 , wherein the control portion is configured to display, on a display screen, part or all of an image captured by an image pickup apparatus, the goal image, the candidate feature points, the goal feature points, the current image, the current feature points, the matching information, the goal feature point clusters, the goal feature point cluster representative information, the current feature point clusters, and the current feature point cluster representative information.

Assignments (1)

Priority Claims (2)

Continuity (1)

References Cited (37)