Information processing program, information processing method, and information processing device

1 . A non-transitory computer-readable recording medium storing an information processing program for causing a computer to execute a process comprising:

obtaining video obtained by imaging a plurality of product arrangement locations in a store;

analyzing the obtained video to specify, from the video, a first area that includes a product arranged in the store, a second area that includes a customer to purchase the product, and a relationship in which interaction between the product and the customer is identified;

specifying, for each product arrangement location of the plurality of product arrangement locations, an action of the customer grasping the product based on the specified relationship and a position of the customer when the specified relationship is indicated;

counting, for each product arrangement location of the plurality of product arrangement locations, a quantity of the action of the customer grasping the product;

tracking the customer included in the video to identify an accounting machine used by the tracked customer;

specifying, from registration information obtained from the identified accounting machine, a purchase quantity of the product that was subject to the customer grasping action at a product arrangement location where the customer indicated the grasping action;

generating, for each product arrangement location of the plurality of product arrangement locations, attention information that indicates a degree of attention in the each product arrangement location, the attention information including, as the degree of attention, a conversion rate for the each product arrangement location based on the counted quantity of the customer grasping action and the specified purchase quantity; and

causing a display device to output, for each product arrangement location of the plurality of product arrangement locations, the conversion rate indicated in the generated attention information in association with the each product arrangement location.

2 . The non-transitory computer-readable recording medium according to claim 1 , the program causing the computer to execute the process further comprising:

specifying a type of the relationship indicated by the customer at a predetermined location where the product is arranged in the store based on the specified relationship and a position of the customer when the specified relationship is indicated;

counting an actual quantity for each type of the relationship indicated by the customer at the predetermined location; and

causing the display device to output the attention information in which the counted actual quantity is associated with the predetermined location.

3 . The non-transitory computer-readable recording medium according to claim 1 , wherein the causing includes

causing the display device to output, for each product arrangement location of the plurality of product arrangement locations, the counted quantity of the customer grasping action in association with the each product arrangement location.

4 . The non-transitory computer-readable recording medium according to claim 1 , wherein the causing includes

causing the display device to output, for each product arrangement location of the plurality of product arrangement locations, the specified purchase quantity in association with the each product arrangement location.

5 . The non-transitory computer-readable recording medium according to claim 1 , wherein

the specifying the first area, the second area, and the relationship includes:

specifying the first area, the second area, and the relationship by inputting the obtained video to a machine learning model, and

the machine learning model includes a model for human object interaction detection (HOID) trained to identify information related to the first area that indicates a first class that indicates an object that includes the product and an area where the object appears, information related to the second area that indicates a second class that indicates a person who purchases the product and an area where the person appears, and interaction between the first class and the second class.

6 . An information processing method comprising:

obtaining video obtained by imaging a plurality of product arrangement locations in a store;

analyzing the obtained video to specify, from the video, a first area that includes a product arranged in the store, a second area that includes a customer to purchase the product, and a relationship in which interaction between the product and the customer is identified;

specifying, for each product arrangement location of the plurality of product arrangement locations, an action of the customer grasping the product based on the specified relationship and a position of the customer when the specified relationship is indicated;

counting, for each product arrangement location of the plurality of product arrangement locations, a quantity of the action of the customer grasping the product;

tracking the customer included in the video to identify an accounting machine used by the tracked customer;

specifying, from registration information obtained from the identified accounting machine, a purchase quantity of the product that was subject to the customer grasping action at a product arrangement location where the customer indicated the grasping action;

generating, for each product arrangement location of the plurality of product arrangement locations, attention information that indicates a degree of attention in the each product arrangement location, the attention information including, as the degree of attention, a conversion rate for the each product arrangement location based on the counted quantity of the customer grasping action and the specified purchase quantity; and

causing a display device to output, for each product arrangement location of the plurality of product arrangement locations, the conversion rate indicated in the generated attention information in association with the each product arrangement location.

7 . The information processing method according to claim 6 , further comprising:

specifying a type of the relationship indicated by the customer at a predetermined location where the product is arranged in the store based on the specified relationship and a position of the customer when the specified relationship is indicated;

counting an actual quantity for each type of the relationship indicated by the customer at the predetermined location; and

causing the display device to output the attention information in which the counted actual quantity is associated with the predetermined location.

8 . The information processing method according to claim 6 , wherein the causing includes

causing the display device to output, for each product arrangement location of the plurality of product arrangement locations, the counted quantity of the customer grasping action in association with the each product arrangement location.

9 . The information processing method according to claim 6 , wherein the causing includes

causing the display device to output, for each product arrangement location of the plurality of product arrangement locations, the specified purchase quantity in association with the each product arrangement location.

10 . The information processing method according to claim 6 , wherein

the specifying the first area, the second area, and the relationship includes:

specifying the first area, the second area, and the relationship by inputting the obtained video to a machine learning model, and

the machine learning model includes a model for human object interaction detection (HOID) trained to identify information related to the first area that indicates a first class that indicates an object that includes the product and an area where the object appears, information related to the second area that indicates a second class that indicates a person who purchases the product and an area where the person appears, and interaction between the first class and the second class.

11 . An information processing device comprising:

a memory; and

a processor coupled to the memory and configured to perform processing including:

obtaining video obtained by imaging a plurality of product arrangement locations in a store;

analyzing the obtained video to specify, from the video, a first area that includes a product arranged in the store, a second area that includes a customer to purchase the product, and a relationship in which interaction between the product and the customer is identified;

specifying, for each product arrangement location of the plurality of product arrangement locations, an action of the customer grasping the product based on the specified relationship and a position of the customer when the specified relationship is indicated;

counting, for each product arrangement location of the plurality of product arrangement locations, a quantity of the action of the customer grasping the product;

tracking the customer included in the video to identify an accounting machine used by the tracked customer;

specifying, from registration information obtained from the identified accounting machine, a purchase quantity of the product that was subject to the customer grasping action at a product arrangement location where the customer indicated the grasping action;

generating, for each product arrangement location of the plurality of product arrangement locations, attention information that indicates a degree of attention in the each product arrangement location, the attention information including, as the degree of attention, a conversion rate for the each product arrangement location based on the counted quantity of the customer grasping action and the specified purchase quantity; and

causing a display device to output, for each product arrangement location of the plurality of product arrangement locations, the conversion rate indicated in the generated attention information in association with the each product arrangement location.

12 . The information processing device according to claim 11 , wherein the processor is configured to:

specify a type of the relationship indicated by the customer at a predetermined location where the product is arranged in the store based on the specified relationship and a position of the customer when the specified relationship is indicated;

count an actual quantity for each type of the relationship indicated by the customer at the predetermined location; and

cause the display device to output the attention information in which the counted actual quantity is associated with the predetermined location.

13 . The information processing device according to claim 11 , wherein the causing includes

causing the display device to output, for each product arrangement location of the plurality of product arrangement locations, the counted quantity of the customer grasping action in association with the each product arrangement location.

14 . The information processing device according to claim 11 , wherein the causing includes

causing the display device to output, for each product arrangement location of the plurality of product arrangement locations, the specified purchase quantity in association with the each product arrangement location.

15 . The information processing device according to claim 11 , wherein

the specifying the first area, the second area, and the relationship includes:

specifying the first area, the second area, and the relationship by inputting the obtained video to a machine learning model, and

the machine learning model includes a model for human object interaction detection (HOID) trained to identify information related to the first area that indicates a first class that indicates an object that includes the product and an area where the object appears, information related to the second area that indicates a second class that indicates a person who purchases the product and an area where the person appears, and interaction between the first class and the second class.