Storage medium, alert generation method, and information processing device
A non-transitory computer-readable storage medium storing an alert generation program that causes at least one computer to execute a process, the process includes acquiring a video of a person who holds a product to be registered in an accounting machine; specifying, by inputting the acquired video to a machine learning model, a product candidate that corresponds to the product included in the video from a plurality of product candidates; acquiring an item of the product input by the person from a plurality of product candidates output by the accounting machine; and generating an alert that indicates an abnormality of the product registered in the accounting machine based on the acquired item of the product and the specified product candidate.
1 . A non-transitory computer-readable storage medium storing an alert generation program that causes at least one computer to execute a process, the process comprising: acquiring a video of a person who holds a product to be registered in an accounting machine; specifying, by inputting the acquired video to a machine learning model, a product candidate that corresponds to the product included in the video from a plurality of product candidates; acquiring an item of the product input by the person from a plurality of product candidates output by the accounting machine; and generating an alert that indicates an abnormality of the product registered in the accounting machine based on the acquired item of the product and the specified product candidate
wherein the specifying includes: inputting the video to an image encoder included in the machine learning model, inputting a plurality of texts that corresponds to the plurality of product candidates to a text encoder included in the machine learning model, and specifying a product candidate that corresponds to the product included in the video among the plurality of product candidates based on similarity between a vector of the video output from the image encoder and vectors of the texts output from the text encoder,
wherein the machine learning model refers to reference source data in which attributes of products are associated with each of a plurality of hierarchies, and wherein the specifying includes: specifying the product candidate by inputting the video to the image encoder, inputting texts for respective attributes of products of a first hierarchy to the text encoder, narrowing down attributes that correspond to the product included in the video among the attributes of the products of the first hierarchy based on similarity between a vector of the video output from the image encoder and vectors of the texts output from the text encoder, inputting the video to the image encoder, inputting texts for respective attributes of products of a second hierarchy obtained by narrowing down from the attributes of the products of the first hierarchy to the text encoder, and specifying an attribute that corresponds to the product included in the video among the attributes of the products of the second hierarchy based on similarity between the vector of the video output from the image encoder and vectors of the texts output from the text encoder.
2 . The non-transitory computer-readable storage medium according to claim 1 , wherein the accounting machine registers an item of a product selected by the person from a list of products output in a display of the accounting machine, and the acquiring includes acquiring the item of the product registered in the accounting machine.
3 . The non-transitory computer-readable storage medium according to claim 1 , wherein the generating includes generating an alert that warns a mismatch between an item of the specified product candidate and an item of the product acquired from the accounting machine.
4 . The non-transitory computer-readable storage medium according to claim 1 , wherein the generating includes generating an alert that includes one of a difference in a purchase amount between an item of the specified product candidate and an item of the product acquired from the accounting machine, and identification information regarding the accounting machine.
5 . An alert generation method for a computer to execute a process comprising: acquiring a video of a person who holds a product to be registered in an accounting machine; specifying, by inputting the acquired video to a machine learning model, a product candidate that corresponds to the product included in the video from a plurality of product candidates; acquiring an item of the product input by the person from a plurality of product candidates output by the accounting machine; and generating an alert that indicates an abnormality of the product registered in the accounting machine based on the acquired item of the product and the specified product candidate, wherein the specifying includes: inputting the video to an image encoder included in the machine learning model, inputting a plurality of texts that corresponds to the plurality of product candidates to a text encoder included in the machine learning model, and specifying a product candidate that corresponds to the product included in the video among the plurality of product candidates based on similarity between a vector of the video output from the image encoder and vectors of the texts output from the text encoder,
wherein the machine learning model refers to reference source data in which attributes of products are associated with each of a plurality of hierarchies, and wherein the specifying includes: specifying the product candidate by inputting the video to the image encoder, inputting texts for respective attributes of products of a first hierarchy to the text encoder, narrowing down attributes that correspond to the product included in the video among the attributes of the products of the first hierarchy based on similarity between a vector of the video output from the image encoder and vectors of the texts output from the text encoder, inputting the video to the image encoder, inputting texts for respective attributes of products of a second hierarchy obtained by narrowing down from the attributes of the products of the first hierarchy to the text encoder, and specifying an attribute that corresponds to the product included in the video among the attributes of the products of the second hierarchy based on similarity between the vector of the video output from the image encoder and vectors of the texts output from the text encoder.
6 . The alert generation method according to claim 5 , wherein the accounting machine registers an item of a product selected by the person from a list of products output in a display of the accounting machine, and the acquiring includes acquiring the item of the product registered in the accounting machine.
7 . The alert generation method according to claim 5 , wherein the generating includes generating an alert that warns a mismatch between an item of the specified product candidate and an item of the product acquired from the accounting machine.
8 . The alert generation method according to claim 5 , wherein the generating includes generating an alert that includes one of a difference in a purchase amount between an item of the specified product candidate and an item of the product acquired from the accounting machine, and identification information regarding the accounting machine.
9 . An information processing device comprising: one or more memories; and one or more processors coupled to the one or more memories and the one or more processors configured to: acquire a video of a person who holds a product to be registered in an accounting machine, specify, by inputting the acquired video to a machine learning model, a product candidate that corresponds to the product included in the video from a plurality of product candidates, acquire an item of the product input by the person from a plurality of product candidates output by the accounting machine, and generate an alert that indicates an abnormality of the product registered in the accounting machine based on the acquired item of the product and the specified product candidate
wherein the specifying includes: inputting the video to an image encoder included in the machine learning model, inputting a plurality of texts that corresponds to the plurality of product candidates to a text encoder included in the machine learning model, and specifying a product candidate that corresponds to the product included in the video among the plurality of product candidates based on similarity between a vector of the video output from the image encoder and vectors of the texts output from the text encoder,
wherein the machine learning model refers to reference source data in which attributes of products are associated with each of a plurality of hierarchies, and wherein the specifying includes: specifying the product candidate by inputting the video to the image encoder, inputting texts for respective attributes of products of a first hierarchy to the text encoder, narrowing down attributes that correspond to the product included in the video among the attributes of the products of the first hierarchy based on similarity between a vector of the video output from the image encoder and vectors of the texts output from the text encoder, inputting the video to the image encoder, inputting texts for respective attributes of products of a second hierarchy obtained by narrowing down from the attributes of the products of the first hierarchy to the text encoder, and specifying an attribute that corresponds to the product included in the video among the attributes of the products of the second hierarchy based on similarity between the vector of the video output from the image encoder and vectors of the texts output from the text encoder.