Non-transitory computer-readable recording medium, information processing method, and information processing apparatus for detecting fraud at accounting machine
An information processing apparatus acquires video image data on a person who is scanning a code of a commodity product to an accounting machine, and specifies, from the acquired video image data by analyzing the acquired video image data, a region of a hand of the person and a region of the commodity product that is being gripped in the hand of the person. The information processing apparatus tracks either a movement of the hand of the person that is gripping the commodity product, or, a movement of the gripped commodity product, and generates, based on a change in the tracked movement of the hand or a change in the tracked movement of the commodity product, an alert connected to an abnormality of a behavior of registering the commodity product to the accounting machine.
1 . A non-transitory computer-readable recording medium having stored therein an information processing program that causes a computer to execute a process comprising:
acquiring video image data on a person who is scanning a code of a commodity product to an accounting machine;
specifying, from the acquired video image data by analyzing the acquired video image data, a first region that includes a hand of the person and a second region that includes the commodity product that is being gripped in the hand of the person;
tracking either a movement of the hand of the person that is gripping the commodity product, or a movement of the gripped commodity product; and
generating, based on a change in the tracked movement of the hand or a change in the tracked movement of the commodity product, an alert connected to an abnormality of a behavior of registering the commodity product to the accounting machine, wherein
the specifying includes specifying, from the acquired video image data by inputting the acquired video image data to a machine learning model, the first region and the second region,
the tracking includes tracking, when a number of commodity products that are being gripped in the hand of the person is plural, either the movement of the hand of the person that is gripping the plurality of commodity products, or, the movement of the plurality of commodity products,
the generating the alert includes generating, based on the change in the tracked movement of the hand or a change in the tracked movement of the plurality of commodity products, the alert connected to an abnormality of a behavior of registering the commodity products to the accounting machine,
the machine learning model outputs, in accordance with an input of the video image data, a bounding box that indicates the first region, a bounding box of the commodity product that indicates the second region, and skeleton information on the hand of the person,
the specifying includes generating, by inputting the acquired video image data to the machine learning model, the bounding box that indicates the first region of the hand, the bounding box of the commodity product that indicates the second region of the commodity product, and the skeleton information on the hand,
the tracking includes determining, when a number of the generated bounding boxes of the commodity product is plural, whether or not positions of the plurality of bounding boxes are overlapped, and
the generating includes
identifying, when it is determined that the overlapped positions are present, a pattern of the movement of the hand of the person based on the skeleton information on the hand of the person, and
generating, by comparing the identified pattern of the movement of the hand with a rule that is set in advance, the alert related to the abnormality of the behavior of registering the commodity products to the accounting machine.
2 . The non-transitory computer-readable recording medium according to claim 1 , wherein
the generating includes
determining, when the movement of the commodity products has been tracked, whether or not a change in the movement of the plurality of commodity products included in the second region of the commodity product corresponds to a pattern that is assigned in advance, and
generating, when the change in the movement of the plurality of commodity products corresponds to the pattern, the alert connected to the abnormality of the behavior of registering the commodity products to the accounting machine.
3 . The non-transitory computer-readable recording medium according to claim 1 , wherein the generating includes generating, when the movement of the hand has been tracked, and in a case where the movement of the hand corresponds to a motion of piling up the plurality of commodity products, the alert connected to the abnormality of the behavior of registering the commodity products to the accounting machine.
4 . The non-transitory computer-readable recording medium according to claim 1 , wherein
the machine learning model specifies, in accordance with an input of the video image data, the first region, the second region, and a relationship between the first region and the second region,
the specifying includes specifying, based on the first region, the second region, and the relationship that have been specified by inputting the acquired video image data to the machine learning model, a first motion of gripping a first commodity product in a right hand of the person and a second motion of gripping a second commodity product in a left hand,
the tracking includes tracking the first motion and the second motion, and
the generating includes detecting, based on the tracked first motion and the tracked second motion, that the code of the commodity product of either the first commodity product or the second commodity product has not been scanned to the accounting machine.
5 . The non-transitory computer-readable recording medium according to claim 4 , wherein, when it is determined that the code of the commodity product of each of the first commodity product and the second commodity product has not been scanned to the accounting machine based on a positional relationship between a set of the tracked first motion and the tracked second motion and an area that is set in order to scan the commodity products to the accounting machine, the generating the alert includes generating the alert indicating the abnormality of the behavior of registering the commodity products to the accounting machine.
6 . The non-transitory computer-readable recording medium according to claim 1 , wherein the generating includes generating, as the alert connected to the abnormality of the behavior of registering the commodity product to the accounting machine, an alert indicating that there is a commodity product that has not yet been registered to the accounting machine by the person, or, indicating that the commodity product that has been registered to the accounting machine by the person is abnormal.
7 . The non-transitory computer-readable recording medium according to claim 1 , wherein the process further includes notifying, when the alert connected to the abnormality of the behavior of registering the commodity product to the accounting machine is generated, a terminal used by a store clerk of identification information on the accounting machine and the generated alert in an associated manner.
8 . The non-transitory computer-readable recording medium according to claim 1 , wherein the generating includes outputting, in a case where the alert connected to the abnormality of the behavior of registering the commodity product to the accounting machine has been generated, voice or a screen that makes the person located at the accounting machine aware of a registration omission of the commodity product from the accounting machine.
9 . The non-transitory computer-readable recording medium according to claim 1 , wherein the process further includes:
capturing, when the alert connected to the abnormality of the behavior of registering the commodity product to the accounting machine has been generated, the person by a camera included in the accounting machine; and
storing image data on the captured person and the alert in an associated manner in a storage.
10 . The non-transitory computer-readable recording medium according to claim 1 , wherein
the accounting machine is a self-service checkout terminal,
the specifying includes specifying, based on the acquired video image data, the first region from the acquired video image data,
the tracking includes tracking, when a number of commodity products that are being gripped in the hand of the person is plural, the movement of the hand of the person that is gripping the plurality of commodity products, and
the generating includes generating, based on the change in the tracked movement of the hand, the alert connected to the abnormality of the behavior of registering the commodity products to the self-service checkout terminal.
11 . An information processing method comprising:
acquiring video image data on a person who is scanning a code of a commodity product to an accounting machine;
specifying, from the acquired video image data by analyzing the acquired video image data, a first region that includes a hand of the person and a second region that includes the commodity product that is being gripped in the hand of the person;
tracking either a movement of the hand of the person that is gripping the commodity product, or, a movement of the gripped commodity product; and
generating, based on a change in the tracked movement of the hand or a change in the tracked movement of the commodity product, an alert connected to an abnormality of a behavior of registering the commodity product to the accounting machine, using a processor, wherein
the specifying includes specifying, from the acquired video image data by inputting the acquired video image data to a machine learning model, the first region and the second region,
the tracking includes tracking, when a number of commodity products that are being gripped in the hand of the person is plural, either the movement of the hand of the person that is gripping the plurality of commodity products, or, the movement of the plurality of commodity products,
the generating the alert includes generating, based on the change in the tracked movement of the hand or a change in the tracked movement of the plurality of commodity products, the alert connected to an abnormality of a behavior of registering the commodity products to the accounting machine,
the machine learning model outputs, in accordance with an input of the video image data, a bounding box that indicates the first region, a bounding box of the commodity product that indicates the second region, and skeleton information on the hand of the person,
the specifying includes generating, by inputting the acquired video image data to the machine learning model, the bounding box that indicates the first region of the hand, the bounding box of the commodity product that indicates the second region of the commodity product, and the skeleton information on the hand,
the tracking includes determining, when a number of the generated bounding boxes of the commodity product is plural, whether or not positions of the plurality of bounding boxes are overlapped, and
the generating includes
identifying, when it is determined that the overlapped positions are present, a pattern of the movement of the hand of the person based on the skeleton information on the hand of the person, and
generating, by comparing the identified pattern of the movement of the hand with a rule that is set in advance, the alert related to the abnormality of the behavior of registering the commodity products to the accounting machine.
12 . An information processing apparatus comprising:
a memory; and
a processor coupled to the memory and configured to:
acquire video image data on a person who is scanning a code of a commodity product to an accounting machine;
specify, from the acquired video image data by analyzing the acquired video image data, a first region that includes a hand of the person and a second region that includes the commodity product that is being gripped in the hand of the person;
track either a movement of the hand of the person that is gripping the commodity product, or, a movement of the gripped commodity product; and
generate, based on a change in the tracked movement of the hand or a change in the tracked movement of the commodity product, an alert connected to an abnormality of a behavior of registering the commodity product to the accounting machine, wherein the processor further configures to:
specify, from the acquired video image data by inputting the acquired video image data to a machine learning model, the first region and the second region,
track, when a number of commodity products that are being gripped in the hand of the person is plural, either the movement of the hand of the person that is gripping the plurality of commodity products, or, the movement of the plurality of commodity products,
generate, based on the change in the tracked movement of the hand or a change in the tracked movement of the plurality of commodity products, the alert connected to an abnormality of a behavior of registering the commodity products to the accounting machine, wherein
the machine learning model outputs, in accordance with an input of the video image data, a bounding box that indicates the first region, a bounding box of the commodity product that indicates the second region, and skeleton information on the hand of the person, wherein the processor further configures to:
generate, by inputting the acquired video image data to the machine learning model, the bounding box that indicates the first region of the hand, the bounding box of the commodity product that indicates the second region of the commodity product, and the skeleton information on the hand,
determine, when a number of the generated bounding boxes of the commodity product is plural, whether or not positions of the plurality of bounding boxes are overlapped,
identify, when it is determined that the overlapped positions are present, a pattern of the movement of the hand of the person based on the skeleton information on the hand of the person, and
generate, by comparing the identified pattern of the movement of the hand with a rule that is set in advance, the alert related to the abnormality of the behavior of registering the commodity products to the accounting machine.