School food and beverage recognition and processing system
A school food and beverage recognition and processing system includes a self-checkout system that attaches to a school meal line. The self-checkout system uses multiple cameras to obtain images of a student for identification and images of a meal, identify contents of the meal, and determine whether the meal qualifies for reimbursement through a health program or agency. In addition to determining meal reimbursement qualifications, the self-checkout system tracks the meals and items sold for inventory and purchasing purposes. The self-checkout system sends reports to the school or administration agency including reimbursement information, inventory information, and customer information. This information is used to update a student's prepaid school meal account and track transactions.
1 . A self-checkout system for school meals in a reimbursable school meal program, comprising:
a frame configured to mount to a school meal line, the frame extending a width from a first side of the self-checkout system to a second side of the self-checkout system, the frame extending a frame height from a base surface of the frame to an upper surface of the frame;
a support arm attached to the upper surface of the frame and extending an arm height from the upper surface of the frame in a direction away from the base surface of the frame;
a head unit attached to the support arm, the head unit comprising:
a forward image sensor arranged at least partially inside the head unit; and
a downward image sensor arranged at least partially inside the head unit, wherein the forward image sensor and the downward image sensor are each disposed on a front side of the head unit, wherein the forward image sensor comprises a first field of view comprising a lower height limit and an upper height limit measured from a floor, wherein the downward image sensor comprises a second field of view that encompasses a tray viewing area arranged below the lower height limit, and wherein the tray viewing area is arranged adjacent the base surface of the frame;
a tray detection sensor comprising a sensing area within the tray viewing area;
a processor; and
a memory coupled with and readable by the processor and storing therein instructions that, when executed by the processor, cause the processor to:
detect when a tray enters the sensing area within the tray viewing area;
in response to detecting when the tray enters the sensing area;
activate the forward image sensor to capture an image of a user associated with the tray within the first field of view; and
activate the downward image sensor to capture an image of the tray within the tray viewing area;
identify, based on the image of the user, the user associated with the tray;
retrieve menu information for items available to the user on the date and time at which the image of the tray was captured;
identify, based on the image of the tray and the menu information, items present on the tray as among the items available to the user on the date and time at which the image of the tray was captured; and
determine whether the items identified as being present on the tray constitute a school meal eligible for reimbursement according to meal qualification rules of the reimbursable school meal program.
2 . The self-checkout system of claim 1 , further comprising:
a visual indicator arranged at least partially inside the head unit and comprising at least one light emitting diode; and
an audio output device arranged at least partially inside the head unit and comprising at least one speaker.
3 . The self-checkout system of claim 2 , wherein the instructions, when executed by the processor, further cause the processor to:
determine, based on the image of the tray within the tray viewing area, that at least a portion of the tray is outside of the tray viewing area; and
activate at least one of the visual indicator and the audio output device to emit an instruction to the user to arrange the tray entirely within the tray viewing area.
4 . The self-checkout system of claim 1 , wherein the instructions, when executed by the processor, further cause the processor to send, across a communication network, information about the user and the tray to a server for further processing.
5 . The self-checkout system of claim 1 , further comprising a head cover attached to the head unit and covering an outer surface of the head unit, wherein the head cover includes a decoration applied to an outer surface of the head cover.
6 . The self-checkout system of claim 5 , wherein the decoration includes features of an illustrated face, and wherein at least one of the features of the illustrated face align with at least one of the forward image sensor and the downward image sensor.
7 . The self-checkout system of claim 5 , wherein the head cover is made from a neoprene material, and wherein the neoprene material comprises a wall thickness surrounding an outer circumferential surface of the head unit.
8 . A self-checkout method for school meals in a reimbursable school meal program, comprising:
identifying a user associated with a tray based on an image of the user captured contemporaneously with an image of the tray;
receiving menu information for menu items available to the user on the date and time at which the image of the tray was captured;
identifying, based on the image of the tray and the menu information, items present on the tray as among the items available to the user on the date and time at which the image of the tray was captured;
determining whether the items identified as being present on the tray constitute a school meal eligible for reimbursement according to meal qualification rules of the reimbursable school meal program; and
sending information about the identified items to a recipient.
9 . The self-checkout method of claim 8 , further comprising:
storing information associated with the image of the user and the image of the tray in a memory accessible by a processing system; and
transferring the stored information to the processing system for one or both of:
identifying the user associated with the tray; and
determining whether the items identified as being present on the tray constitute the school meal eligible for reimbursement.
10 . The self-checkout method of claim 8 , wherein identifying the items present on the tray comprises comparing, using an image recognition algorithm, the items present on the tray with images of the items available to the user on the date and time at which the image of the tray was captured.
11 . The self-checkout method of claim 10 , wherein comparing the items present on the tray with images of the items comprises:
selecting one or more images of the tray from among a plurality of contemporaneously captured images of the tray for use in identifying the items present on the tray; and
comparing the selected one or more images of the tray with the images of the items available to the user on the date and time at which the image of the tray was captured.
12 . The self-checkout method of claim 8 , wherein sending the ifnormaiton about the identified items to the recipient comprises sending a determination of whether the identified items are reimbursable.
13 . The self-checkout method of claim 8 , wherein identifying the user associated with the tray comprises applying a facial recognition algorithm to the image of the user.
14 . The self-checkout method of claim 8 , further comprising providing alerts based on an outcome of determining whether the items identified as being present on the tray constitute the school meal eligible for reimbursement.
15 . The self-checkout method of claim 14 , wherein determining whether the items identified as being present on the tray constitute the school meal eligible for reimbursement comprises determining that the items present on the tray include a restricted item, an improper portion, or an item that the user is allergic to, and wherein the providing alerts based on an outcome of determining whether the items identified as being present on the tray constitute the school meal eligible for reimbursemnt comprises:
providing a first alert when the outcome of the determining step is that the items present on the tray do not constitute a reimbursable school meal;
providing a second alert when the outcome of the determining step is that the items present on the tray include a restricted item;
providing a third alert when the outcome of the determining step is that the items present on the tray include an improper portion; and
providing a fourth alert when the outcome of the determining step is that the items present on the tray include an item that the user is allergic to.
16 . The method of claim 8 , wherein identifying the items present on the tray as among the items available to the user on the date and time at which the image of the tray was captured comprises providing the image of the tray to an artificial intelligence (AI) object recognition model trained to identify the items present on the tray using a library of images labeled in accordance with pre-identified food or beverage items depicted in the images of the library.
17 . The method of claim 16 , wherein determining whether the items present on the tray constitute the school meal eligible for reimbursement comprises:
providing, to the AI object recognition model, labeled images for each item of the items available to the user on the date and time at which the image of the tray was captured, wherein the AI object recognition model is further trained to identify the items present on the tray using only a subset of images of the library having labels corresponding to the items available to the user on the date and time at which the image of the tray was captured;
receiving, from the AI object recognition model, an identity of each item present on the tray; and
determining whether data representative of a listing of identities of the items present on the tray received from the AI object recognition model includes a sufficient number of different items to constitute the school meal eligible for reimbursement.
18 . The method of claim 17 , wherein the AI object recognition model is further trained to assign a confidence score to the identification of each item present on the tray, wherein receiving the identity of each present on the tray comprises receiving the confidence score associated with the identification by the AI object recognition model of each item on the tray, and wherein the method further comprises
identifying any of the items identified by the AI object recognition model having a confidence score not meeting a threshold; and
routing the image of the tray for human review for manual identification of those items present on the tray having confidences scores not meeting the threshold.
19 . The method of claim 18 , further comprising training the AI object recognition model using results of the manual identification.
20 . A system comprising one or more circuits to:
identify a user associated with a tray based on an image of the user captured contemporaneously with an image of the tray;
receive menu information for menu items available to the user on the date and time at which the image of the tray was captured;
identify, based on the image of the tray and the menu information, items present on the tray as among the items available to the user on the date and time at which the image of the tray was captured;
determine whether the items identified as being present on the tray constitute a school meal eligible for reimbursement according to meal qualification rules of a reimbursable school meal program; and
send information about the identified items to a recipient.