Recording medium, image processing device, image processing method, and in-vehicle interface device
Provided are a recording medium, an image processing device, an image processing method, and an in-vehicle interface device capable of realizing a non-contact user interface matching a subjective opinion of a user. A computer readable non-transitory recording medium recording a computer program causes a computer to execute processes of acquiring an image in which a plurality of indicators is captured, and setting a pointer designating a position represented by ends of a first indicator and a second indicator at a position between the end of the first indicator and the end of the second indicator specified based on the acquired image.
1 . A computer readable non-transitory recording medium recording a computer program causing a computer to execute processes of:
acquiring an image in which a plurality of indicators is captured;
setting a pointer designating a position represented by ends of a first indicator and a second indicator at a position between the end of the first indicator and the end of the second indicator specified based on the acquired image;
storing a history of the position of the set pointer in association with a frame;
adjusting a number of frames for calculating a moving average according to a moving speed of an end of the indicator;
calculating a moving average of the position of the pointer in each of the adjusted number of frames upon receipt of a current frame; and
setting the position of the pointer in the current frame based on the calculated moving average.
2 . The computer readable non-transitory recording medium recording the computer program according to claim 1 , wherein:
the image in which the plurality of indicators is captured is an image of a hand or finger,
the end of the first indicator and the end of the second indicator are a fingertip of a first finger and a fingertip of a second finger, and
the computer program causes the computer to execute a process of setting the pointer at a middle position between the fingertip of the first finger and the fingertip of the second finger.
3 . The computer readable non-transitory recording medium recording the computer program according to claim 2 , wherein:
one of the first finger and the second finger is a thumb, and
the computer program causes the computer to execute a process of setting the pointer at a pad of the thumb.
4 . The computer readable non-transitory recording medium recording the computer program according to claim 2 , wherein the computer program causes the computer to execute processes of:
detecting a tip of a fingertip by performing a search along a search axis in a predetermined direction on the acquired image of the hand or finger;
setting a perpendicular axis separated from the tip by a predetermined distance and perpendicular to the search axis;
detecting an outer boundary of each of the first finger and the second finger by performing a search along the set perpendicular axis;
specifying a midpoint of the detected outer boundary; and
setting the pointer based on the specified midpoint.
5 . The computer readable non-transitory recording medium recording the computer program according to claim 4 , wherein the computer program causes the computer to execute a process of setting the pointer at a position offset from the midpoint by a predetermined distance in a direction parallel to the search axis.
6 . The computer readable non-transitory recording medium recording the computer program according to claim 1 , wherein the computer program causes the computer to execute a process of accepting an operation of setting a movement amount for moving the position of the set pointer to a desired position.
7 . The computer readable non-transitory recording medium recording the computer program according to claim 1 , wherein the computer program causes the computer to execute processes of:
displaying a plurality of positions of the pointer as selectable candidate positions; and
accepting an operation of selecting a position of the pointer from among the displayed candidate positions.
8 . The computer readable non-transitory recording medium recording the computer program according to claim 2 , wherein the computer program causes the computer to execute processes of:
inputting the image of the hand or finger to a learning model generated to output classification information including a first shape in which the first finger and the second finger are in contact or a second shape in which the first finger and the second finger are separated from each other when the image of the hand or finger is input; and
switching a display mode of a pointer located between a tip of the first finger and a tip of the second finger depending on the classification information output by the learning model.
9 . The computer readable non-transitory recording medium recording the computer program according to claim 8 , wherein the computer program causes the computer to execute processes of:
specifying a distance to the hand or finger;
adjusting a size of an image input to the learning model based on the specified distance; and
inputting an image after adjustment to the learning model.
10 . The computer readable non-transitory recording medium recording the computer program according to claim 1 , wherein the computer program causes the computer to execute a process of composing an acquired image or an image obtained by processing the image with a symbol representing the pointer and displaying the image on a display screen.
11 . The computer readable non-transitory recording medium recording the computer program according to claim 2 , wherein the computer program causes the computer to execute processes of:
displaying an image of the hand or finger in a state in which the first finger and the second finger are separated from each other on a display screen;
displaying a pointer between the fingertip of the first finger and the fingertip of the second finger; and
executing an operation on the display screen indicated by the pointer when accepting an action of putting the fingertip of the first finger and the fingertip of the second finger into contact.
12 . An image processing device comprising
a controller,
wherein the controller is configured to:
acquire an image in which a plurality of indicators is captured;
set a pointer designating a position represented by ends of a first indicator and a second indicator at a position between the end of the first indicator and the end of the second indicator specified based on the acquired image;
store a history of the position of the set pointer in association with a frame;
adjust a number of frames for calculating a moving average according to a moving speed of an end of the indicator;
calculate a moving average of the position of the pointer in each of the adjusted number of frames upon receipt of a current frame; and
set the position of the pointer in the current frame based on the calculated moving average.
13 . The image processing device according to claim 12 , wherein the controller acquires an image in which the plurality of indicators is captured by a TOF (Time Of Flight) camera.
14 . An image processing method comprising:
acquiring an image in which a plurality of indicators is captured;
setting a pointer designating a position represented by ends of a first indicator and a second indicator at a position between the end of the first indicator and the end of the second indicator specified based on the acquired image;
storing a history of the position of the set pointer in association with a frame;
adjusting a number of frames for calculating a moving average according to a moving speed of an end of the indicator;
calculating a moving average of the position of the pointer in each of the adjusted number of frames upon receipt of a current frame; and
setting the position of the pointer in the current frame based on the calculated moving average.
15 . An in-vehicle interface device comprising
a controller,
wherein the controller is configured to:
acquire an image in which a plurality of indicators is captured;
set a pointer designating a position represented by ends of a first indicator and a second indicator at a position between the end of the first indicator and the end of the second indicator specified based on the acquired image;
store a history of the position of the set pointer in association with a frame;
adjust a number of frames for calculating a moving average according to a moving speed of an end of the indicator;
calculate a moving average of the position of the pointer in each of the adjusted number of frames upon receipt of a current frame;
set the position of the pointer in the current frame based on the calculated moving average; and
accept an operation by the pointer displayed on an in-vehicle device operation screen.
16 . A computer readable non-transitory recording medium recording a computer program causing a computer to execute processes of:
acquiring an image in which a plurality of indicators is captured;
setting a pointer designating a position represented by ends of a first indicator and a second indicator at a position between the end of the first indicator and the end of the second indicator specified based on the acquired image, wherein:
the image in which the plurality of indicators is captured is an image of a hand or finger,
the end of the first indicator and the end of the second indicator are a fingertip of a first finger and a fingertip of a second finger,
the computer program causes the computer to execute a process of setting the pointer at a middle position between the fingertip of the first finger and the fingertip of the second finger;
detecting a tip of a fingertip in a current frame by performing a search along a search axis in a predetermined direction on the acquired image of the hand or finger;
setting a perpendicular axis separated from the tip by a predetermined distance and perpendicular to the search axis;
detecting an outer boundary of each of the first finger and the second finger by performing a search along the set perpendicular axis;
specifying a midpoint of the detected outer boundary; and
setting the position of the pointer in the current frame based on the specified midpoint.
17 . A computer readable non-transitory recording medium recording a computer program causing a computer to execute processes of:
acquiring an image in which a plurality of indicators is captured;
setting a pointer in a non-selected state and designating a position represented by ends of a first indicator and a second indicator at a position between the end of the first indicator and the end of the second indicator specified based on the acquired image, wherein:
the image in which the plurality of indicators is captured is an image of a hand or finger,
the end of the first indicator and the end of the second indicator are a fingertip of a first finger and a fingertip of a second finger,
the computer program causes the computer to execute a process of setting the pointer at a middle position between the fingertip of the first finger and the fingertip of the second finger;
inputting the image of the hand or finger to a learning model generated to output classification information including a first shape indicating a selected state in which the first finger and the second finger are in contact or a second shape indicating a non-selected state in which the first finger and the second finger are separated from each other when the image of the hand or finger is input; and
switching a display mode of a pointer located between a tip of the first finger and a tip of the second finger depending on the classification information output by the learning model, wherein the classification indicates a selected state.
18 . A computer readable non-transitory recording medium recording a computer program causing a computer to execute processes of:
acquiring an image in which a plurality of indicators is captured;
setting a pointer designating a position represented by ends of a first indicator and a second indicator at a position between the end of the first indicator and the end of the second indicator specified based on the acquired image, wherein:
the image in which the plurality of indicators is captured is an image of a hand or finger,
the end of the first indicator and the end of the second indicator are a fingertip of a first finger and a fingertip of a second finger;
detecting a tip of a fingertip by performing a search along a search axis in a predetermined direction on the acquired image of the hand or finger;
setting a perpendicular axis separated from the tip by a predetermined distance and perpendicular to the search axis;
detecting a first outer boundary point and a second outer boundary point of the first finger and the second finger, respectively, by performing a search along the set perpendicular axis;
correcting the direction in a current frame of the search axis based on a first tangent line tangent to a boundary of the first finger at the detected first outer boundary point and a second tangent line tangent to a boundary of the second finger at the detected second outer boundary point; and
setting the position of the pointer in the current frame based on the corrected search axis.
19 . A computer readable non-transitory recording medium recording a computer program causing a computer to execute processes of:
acquiring an image in which a plurality of indicators is captured;
setting a pointer designating a position represented by ends of a first indicator and a second indicator at a position between the end of the first indicator and the end of the second indicator specified based on the acquired image, wherein:
the image in which the plurality of indicators is captured is an image of a hand or finger,
the end of the first indicator and the end of the second indicator are a fingertip of a first finger and a fingertip of a second finger;
storing a position of a fingertip specified when the position of the pointer is set and a direction of a search axis in association with a frame; and
setting the position of the pointer by detecting a tip of a fingertip by performing a search along a search axis associated with a current frame based on a position of the fingertip associated with the current frame when the position of the fingertip and a direction of the search axis associated with the current frame are stored.
20 . A computer readable non-transitory recording medium recording a computer program causing a computer to execute processes of:
acquiring an image in which a plurality of indicators is captured;
setting a pointer designating a position represented by ends of a first indicator and a second indicator at a position between the end of the first indicator and the end of the second indicator specified based on the acquired image, wherein:
the image in which the plurality of indicators is captured is an image of a hand or finger,
the end of the first indicator and the end of the second indicator are a fingertip of a first finger and a fingertip of a second finger;
specifying a contour of the hand or finger based on the acquired image of the hand or finger;
associating each point on the specified contour with a distance to a search axis in a predetermined direction;
specifying a point on the contour in a current frame corresponding to a local minimum point of the distance as a fingertip end; and
setting the position of the pointer in the current frame based on the local minimum point.
21 . An image processing device comprising
a controller,
wherein the controller is configured to:
acquire an image in which a plurality of indicators is captured;
set a pointer designating a position represented by ends of a first indicator and a second indicator at a position between the end of the first indicator and the end of the second indicator specified based on the acquired image, wherein:
the image in which the plurality of indicators is captured is an image of a hand or finger,
the end of the first indicator and the end of the second indicator are a fingertip of a first finger and a fingertip of a second finger;
setting the pointer at a middle position between the fingertip of the first finger and the fingertip of the second finger;
detecting a tip of a fingertip in a current frame by performing a search along a search axis in a predetermined direction on the acquired image of the hand or finger;
setting a perpendicular axis separated from the tip by a predetermined distance and perpendicular to the search axis;
detecting an outer boundary of each of the first finger and the second finger by performing a search along the set perpendicular axis;
specifying a midpoint of the detected outer boundary; and
setting the position of the pointer in the current frame based on the specified midpoint.
22 . An image processing device comprising
a controller,
wherein the controller is configured to:
acquire an image in which a plurality of indicators is captured;
set a pointer designating a position represented by ends of a first indicator and a second indicator at a position between the end of the first indicator and the end of the second indicator specified based on the acquired image, wherein:
the image in which the plurality of indicators is captured is an image of a hand or finger,
the end of the first indicator and the end of the second indicator are a fingertip of a first finger and a fingertip of a second finger;
setting the pointer at a middle position between the fingertip of the first finger and the fingertip of the second finger;
inputting the image of the hand or finger to a learning model generated to output classification information including a first shape indicating a selected state in which the first finger and the second finger are in contact or a second shape indicating a non-selected state in which the first finger and the second finger are separated from each other when the image of the hand or finger is input; and
switching a display mode of a pointer located between a tip of the first finger and a tip of the second finger depending on the classification information output by the learning model, wherein the classification indicates a selected state.
23 . An image processing device comprising
a controller,
wherein the controller is configured to:
acquire an image in which a plurality of indicators is captured;
set a pointer designating a position represented by ends of a first indicator and a second indicator at a position between the end of the first indicator and the end of the second indicator specified based on the acquired image, wherein:
the image in which the plurality of indicators is captured is an image of a hand or finger,
the end of the first indicator and the end of the second indicator are a fingertip of a first finger and a fingertip of a second finger;
setting the pointer at a middle position between the fingertip of the first finger and the fingertip of the second finger;
detecting a tip of a fingertip by performing a search along a search axis in a predetermined direction on the acquired image of the hand or finger;
setting a perpendicular axis separated from the tip by a predetermined distance and perpendicular to the search axis;
detecting a first outer boundary point and a second outer boundary point of the first finger and the second finger, respectively, by performing a search along the set perpendicular axis;
correcting the direction in a current frame of the search axis based on a first tangent line tangent to a boundary of the first finger at the detected first outer boundary point and a second tangent line tangent to a boundary of the second finger at the detected second outer boundary point; and
setting the position of the pointer in the current frame based on the corrected search axis.
24 . An image processing device comprising
a controller,
wherein the controller is configured to:
acquire an image in which a plurality of indicators is captured;
set a pointer designating a position represented by ends of a first indicator and a second indicator at a position between the end of the first indicator and the end of the second indicator specified based on the acquired image, wherein:
the image in which the plurality of indicators is captured is an image of a hand or finger,
the end of the first indicator and the end of the second indicator are a fingertip of a first finger and a fingertip of a second finger;
setting the pointer at a middle position between the fingertip of the first finger and the fingertip of the second finger;
storing a position of a fingertip specified when the position of the pointer is set and a direction of a search axis in association with a frame; and
setting the position of the pointer by detecting a tip of a fingertip by performing a search along a search axis associated with a current frame based on a position of the fingertip associated with the current frame when the position of the fingertip and a direction of the search axis associated with the current frame are stored.
25 . An image processing device comprising
a controller,
wherein the controller is configured to:
acquire an image in which a plurality of indicators is captured;
set a pointer designating a position represented by ends of a first indicator and a second indicator at a position between the end of the first indicator and the end of the second indicator specified based on the acquired image, wherein:
the image in which the plurality of indicators is captured is an image of a hand or finger,
the end of the first indicator and the end of the second indicator are a fingertip of a first finger and a fingertip of a second finger;
setting the pointer at a middle position between the fingertip of the first finger and the fingertip of the second finger;
specifying a contour of the hand or finger based on the acquired image of the hand or finger;
associating each point on the specified contour with a distance to a search axis in a predetermined direction;
specifying a point on the contour in a current frame corresponding to a local minimum point of the distance as a fingertip end; and
setting the position of the pointer in the current frame based on the local minimum point.
26 . An image processing method comprising:
acquiring an image in which a plurality of indicators is captured;
setting a pointer designating a position represented by ends of a first indicator and a second indicator at a position between the end of the first indicator and the end of the second indicator specified based on the acquired image, wherein:
the image in which the plurality of indicators is captured is an image of a hand or finger,
the end of the first indicator and the end of the second indicator are a fingertip of a first finger and a fingertip of a second finger;
setting the pointer at a middle position between the fingertip of the first finger and the fingertip of the second finger;
detecting a tip of a fingertip in a current frame by performing a search along a search axis in a predetermined direction on the acquired image of the hand or finger;
setting a perpendicular axis separated from the tip by a predetermined distance and perpendicular to the search axis;
detecting an outer boundary of each of the first finger and the second finger by performing a search along the set perpendicular axis;
specifying a midpoint of the detected outer boundary; and
setting the position of the pointer in the current frame based on the specified midpoint.
27 . An image processing method comprising:
acquiring an image in which a plurality of indicators is captured;
setting a pointer designating a position represented by ends of a first indicator and a second indicator at a position between the end of the first indicator and the end of the second indicator specified based on the acquired image, wherein:
the image in which the plurality of indicators is captured is an image of a hand or finger,
the end of the first indicator and the end of the second indicator are a fingertip of a first finger and a fingertip of a second finger;
setting the pointer at a middle position between the fingertip of the first finger and the fingertip of the second finger;
inputting the image of the hand or finger to a learning model generated to output classification information including a first shape indicating a selected state in which the first finger and the second finger are in contact or a second shape indicating a non-selected state in which the first finger and the second finger are separated from each other when the image of the hand or finger is input; and
switching a display mode of a pointer located between a tip of the first finger and a tip of the second finger depending on the classification information output by the learning model, wherein the classification indicates a selected state.
28 . An image processing method comprising:
acquiring an image in which a plurality of indicators is captured;
setting a pointer designating a position represented by ends of a first indicator and a second indicator at a position between the end of the first indicator and the end of the second indicator specified based on the acquired image, wherein:
the image in which the plurality of indicators is captured is an image of a hand or finger,
the end of the first indicator and the end of the second indicator are a fingertip of a first finger and a fingertip of a second finger;
setting the pointer at a middle position between the fingertip of the first finger and the fingertip of the second finger;
detecting a tip of a fingertip by performing a search along a search axis in a predetermined direction on the acquired image of the hand or finger;
setting a perpendicular axis separated from the tip by a predetermined distance and perpendicular to the search axis;
detecting a first outer boundary point and a second outer boundary point of the first finger and the second finger, respectively, by performing a search along the set perpendicular axis; and
correcting the direction in a current frame of the search axis based on a first tangent line tangent to a boundary of the first finger at the detected first outer boundary point and a second tangent line tangent to a boundary of the second finger at the detected second outer boundary point; and
setting the position of the pointer in the current frame based on the corrected search axis.
29 . An image processing method comprising:
acquiring an image in which a plurality of indicators is captured;
setting a pointer designating a position represented by ends of a first indicator and a second indicator at a position between the end of the first indicator and the end of the second indicator specified based on the acquired image, wherein:
the image in which the plurality of indicators is captured is an image of a hand or finger,
the end of the first indicator and the end of the second indicator are a fingertip of a first finger and a fingertip of a second finger;
setting the pointer at a middle position between the fingertip of the first finger and the fingertip of the second finger;
storing a position of a fingertip specified when the position of the pointer is set and a direction of a search axis in association with a frame; and
setting the position of the pointer by detecting a tip of a fingertip by performing a search along a search axis associated with a current frame based on a position of the fingertip associated with the current frame when the position of the fingertip and a direction of the search axis associated with the current frame are stored.
30 . An image processing method comprising:
acquiring an image in which a plurality of indicators is captured;
setting a pointer designating a position represented by ends of a first indicator and a second indicator at a position between the end of the first indicator and the end of the second indicator specified based on the acquired image, wherein:
the image in which the plurality of indicators is captured is an image of a hand or finger,
the end of the first indicator and the end of the second indicator are a fingertip of a first finger and a fingertip of a second finger;
setting the pointer at a middle position between the fingertip of the first finger and the fingertip of the second finger;
specifying a contour of the hand or finger based on the acquired image of the hand or finger;
associating each point on the specified contour with a distance to a search axis in a predetermined direction;
specifying a point on the contour in a current frame corresponding to a local minimum point of the distance as a fingertip end; and
setting the position of the pointer in the current frame based on the local minimum point.
31 . An in-vehicle interface device comprising
a controller,
wherein the controller is configured to:
acquire an image in which a plurality of indicators is captured;
set a pointer designating a position represented by ends of a first indicator and a second indicator at a position between the end of the first indicator and the end of the second indicator specified based on the acquired image;
accept an operation by the pointer displayed on an in-vehicle device operation screen, wherein:
the image in which the plurality of indicators is captured is an image of a hand or finger,
the end of the first indicator and the end of the second indicator are a fingertip of a first finger and a fingertip of a second finger;
setting the pointer at a middle position between the fingertip of the first finger and the fingertip of the second finger;
detecting a tip of a fingertip in a current frame by performing a search along a search axis in a predetermined direction on the acquired image of the hand or finger;
setting a perpendicular axis separated from the tip by a predetermined distance and perpendicular to the search axis;
detecting an outer boundary of each of the first finger and the second finger by performing a search along the set perpendicular axis;
specifying a midpoint of the detected outer boundary; and
setting the position of the pointer in the current frame based on the specified midpoint.
32 . An in-vehicle interface device comprising
a controller,
wherein the controller is configured to:
acquire an image in which a plurality of indicators is captured;
set a pointer designating a position represented by ends of a first indicator and a second indicator at a position between the end of the first indicator and the end of the second indicator specified based on the acquired image;
accept an operation by the pointer displayed on an in-vehicle device operation screen, wherein:
the image in which the plurality of indicators is captured is an image of a hand or finger,
the end of the first indicator and the end of the second indicator are a fingertip of a first finger and a fingertip of a second finger;
setting the pointer at a middle position between the fingertip of the first finger and the fingertip of the second finger;
inputting the image of the hand or finger to a learning model generated to output classification information including a first shape indicating a selected state in which the first finger and the second finger are in contact or a second shape indicating a non-selected state in which the first finger and the second finger are separated from each other when the image of the hand or finger is input; and
switching a display mode of a pointer located between a tip of the first finger and a tip of the second finger depending on the classification information output by the learning model, wherein the classification indicates a selected state.
33 . An in-vehicle interface device comprising
a controller,
wherein the controller is configured to:
acquire an image in which a plurality of indicators is captured;
set a pointer designating a position represented by ends of a first indicator and a second indicator at a position between the end of the first indicator and the end of the second indicator specified based on the acquired image;
accept an operation by the pointer displayed on an in-vehicle device operation screen, wherein:
the image in which the plurality of indicators is captured is an image of a hand or finger,
the end of the first indicator and the end of the second indicator are a fingertip of a first finger and a fingertip of a second finger;
setting the pointer at a middle position between the fingertip of the first finger and the fingertip of the second finger;
detecting a tip of a fingertip by performing a search along a search axis in a predetermined direction on the acquired image of the hand or finger;
setting a perpendicular axis separated from the tip by a predetermined distance and perpendicular to the search axis;
detecting a first outer boundary point and a second outer boundary point of the first finger and the second finger, respectively, by performing a search along the set perpendicular axis;
correcting the direction in a current frame of the search axis based on a first tangent line tangent to a boundary of the first finger at the detected first outer boundary point and a second tangent line tangent to a boundary of the second finger at the detected second outer boundary point; and
setting the position of the pointer in the current frame based on the corrected search axis.
34 . An in-vehicle interface device comprising
a controller,
wherein the controller is configured to:
acquire an image in which a plurality of indicators is captured;
set a pointer designating a position represented by ends of a first indicator and a second indicator at a position between the end of the first indicator and the end of the second indicator specified based on the acquired image;
accept an operation by the pointer displayed on an in-vehicle device operation screen, wherein:
the image in which the plurality of indicators is captured is an image of a hand or finger,
the end of the first indicator and the end of the second indicator are a fingertip of a first finger and a fingertip of a second finger;
setting the pointer at a middle position between the fingertip of the first finger and the fingertip of the second finger;
storing a position of a fingertip specified when the position of the pointer is set and a direction of a search axis in association with a frame; and
setting the position of the pointer by detecting a tip of a fingertip by performing a search along a search axis associated with a current frame based on a position of the fingertip associated with the current frame when the position of the fingertip and a direction of the search axis associated with the current frame are stored.
35 . An in-vehicle interface device comprising
a controller,
wherein the controller is configured to:
acquire an image in which a plurality of indicators is captured;
set a pointer designating a position represented by ends of a first indicator and a second indicator at a position between the end of the first indicator and the end of the second indicator specified based on the acquired image;
accept an operation by the pointer displayed on an in-vehicle device operation screen, wherein:
the image in which the plurality of indicators is captured is an image of a hand or finger,
the end of the first indicator and the end of the second indicator are a fingertip of a first finger and a fingertip of a second finger;
setting the pointer at a middle position between the fingertip of the first finger and the fingertip of the second finger;
specifying a contour of the hand or finger based on the acquired image of the hand or finger;
associating each point on the specified contour with a distance to a search axis in a predetermined direction;
specifying a point on the contour in a current frame corresponding to a local minimum point of the distance as a fingertip end; and
setting the position of the pointer in the current frame based on the local minimum point.